🎢 RL Course 02 - CartPole
Last time you learned the words. Today you meet a real environment — a pole balanced on a cart — and train a policy with your own hands, right in the browser. No library, no GPU: just four numbers and some luck.
I
🎮 The Environment
Four numbers in, two actions out, +1 for every step the pole stays up. You play it first.
II
🎲 Baseline & Policy
Random guessing sets the bar. A policy with just 4 weights can beat it by a mile.
III
🧪 Search
Random search and hill climbing: generate, score, keep the best — and see why one score can lie.
IV
🧱 The Wall
Why guessing works for 4 weights, fails for a humanoid, and what the loop's gate must respect.
🔑 Big insight: training is just propose → test → keep the best. Everything later (evolution, PPO, humanoids) is a smarter way to do those three words.
🎢
Lesson 2 · Part A · Foundations
📖
Source: Sutton & Barto, Example 3.4 (Pole-Balancing)
📚 Based on "Reinforcement Learning: An Introduction" (Sutton & Barto), Ch.3 Example 3.4, with Gymnasium's CartPole-v1 physics
🖤 Blackboard: CartPole
Slide 1 / 11
Where We Are — Lesson 2 of 9
🖼️ The marker moved up one step: CART.
- Lesson 1 gave you the words and the loop (collect → train → test vs champion → deploy → monitor)
- Gymnasium is a free Python library of ready-made practice worlds ("environments") for reinforcement learning — a gym for robot brains (中文：强化学习的"标准练习场")
- CartPole-v1 is one of its environments: a cart (小车) that slides left/right on a track, with a pole (杆子) balanced on top. "v1" = version 1 of the rules, so everyone's scores can be compared (中文：第 1 版规则，成绩可以互相比较)
- Three words from lesson 1, one line each: reward (奖励) = the score the world hands back after each step · policy (策略) = the rule that picks an action from what the robot sees · value (价值) = how much total reward you can expect from here on if you keep playing well. Today is mostly about reward and policy; value comes back in later lessons
- Today we make these words real on the smallest useful problem — and build a miniature of the loop by hand
- Source: Sutton & Barto Ch.3 Example 3.4, Pole-Balancing. The Python in the code panel is a faithful copy of Gymnasium's CartPole-v1, and the widgets on these slides run the same physics in JavaScript
CartPole — Rules First, Then Be the Policy
🖼️ Four gauges tell the state. Two arrows are the only actions.
- State (4 numbers) = what the robot can see each step: cart position, cart speed, pole angle, pole angular speed (中文：状态)
- Action (2 choices) = what it can do each step: push left or push right, a fixed-size shove (中文：动作)
- Step = one tick of the game, 0.02 s, so 50 steps per second
- Episode = one try, from the start until it ends (中文：回合). It ends when the pole tilts past the limit angle, the cart leaves the track (±2.4), or 500 steps pass. Official CartPole-v1 uses a tight 12° limit; the widget above lets you widen it so a human can play
- Policy = the rule that turns a state into an action. Just now you were the policy: your eyes read the state, your hand picked the action
- Sutton & Barto, Example 3.4: "A failure is said to occur if the pole falls past a given angle from vertical or if the cart runs off the track. The pole is reset to vertical after each failure."
- The leaderboard compares your best tries with three simple bots. In the next slides you will build bots better than all of them
Reward +1 — and Return = Steps Survived
🖼️ The +1 counter runs while the pole is up and stops when it falls.
- This continues slide 2. There your score was how many steps you lasted. Here we say exactly why that number is the right score
- A step is one tick of the game (0.02 s): the robot looks at the 4 gauges, picks push-left or push-right, and the physics moves forward one tick
- Every step in which the pole is still up pays +1. Nothing else is ever rewarded: not angle, not distance
- So the return of an episode = the number of steps survived. Perfect = 500
- Sutton & Barto, Example 3.4: "The reward in this case could be +1 for every time step on which failure did not occur, so that the return at each time would be the number of steps until failure"
- The trick: in lesson 1 a reward for speed alone made the robot dive and slide on its face, so we had to add "stay upright" terms. Here we get "stay upright" for free: falling ends the episode, and a policy that falls early collects fewer +1s. A simple reward plus an ending rule teaches the whole skill
Baseline — What Does Pure Guessing Score?
🖼️ Coin-flip control earns a tiny bar.
- A random policy ignores the pole completely. Every step it flips a coin: heads → push left, tails → push right. The "random" is that coin
- With the official 12° limit it survives only ~22 steps — about half a second. (The widget replays one episode in slow motion with a wider limit so you can watch it; the 100-episode score also shows the official number)
- Baseline = the score of the simplest possible policy, pure guessing. Think of the score you would get by guessing every answer on a test: it is the starting grade. Any policy we train must beat it, otherwise it has learned nothing
- Averaging over 100 episodes matters: single episodes vary a lot (min and max in the widget). Remember that — it comes back on slide 7
A Policy With Just Four Numbers
🖼️ Four knobs (weights) → add up → right or left.
- Linear policy: multiply each state number by its weight, add them up. Positive total → push right; otherwise push left. The weights are 4 dials on the robot's decision rule — not the pole's weight
- The four weights are the policy (the robot's "brain" is just this one rule). Training = finding good weights, meaning the weights whose policy keeps the pole up for the most steps
- Angle only (0, 0, 1, 0): "push toward the lean" → ~43 steps. Better than random, still bad
- Angle + pole speed (0, 0, 1, 1): ~480 steps, almost perfect, from one extra weight
- Why 500? That is the game's time limit (10 s), so 500 is the best possible score. A policy that reaches it never lost the pole: it would balance forever if the clock let it. That is the goal of the whole lesson: find weights that reach 500, then check it is not luck (slide 7)
- Why speed helps: angle says where the pole is; speed says where it is heading. Sutton & Barto §9.5 make this point: "high angular velocity can be either good or bad depending on the angle"
Training = Propose, Test, Keep the Best
🖼️ Many cards tossed, one golden card kept.
- Random search: draw random weights, score each on a few episodes, keep the best so far
- Hill climbing: instead of drawing from scratch, nudge the current best weights a little and keep the nudge only if the score improves
- This is the whole training loop in miniature: propose → test → keep the best
- Run each method several times. Random search usually finds a good policy within about 20 candidates. Hill climbing sometimes gets stuck — slide 8 explains why
One Score Can Lie
🖼️ The same weights: a tall score from 5 lucky tests, a shorter one from 100 fresh tests.
- Each episode starts from a slightly different random position, so scores are noisy
- If you pick the best of 60 noisy scores, you partly pick the luckiest. Our best weights scored 404 during the search and 363 on 100 fresh episodes
- The fix: re-test the winner on many fresh episodes before believing it
- This is exactly why lesson 1's champion gate compares policies on many fresh test games, not one lucky score
Hill Climbing Can Get Stuck
🖼️ On a flat plain every nudge looks the same; a big jump lands on the mountain.
- Hill climbing needs small changes to give a hint: better → keep, worse → drop
- Start near weights that score ~10 and every small nudge also scores ~10. No hint which way is "uphill" — it stalls
- Random search jumps: each candidate lands somewhere new, so it keeps finding better regions. This is exploration vs exploitation (lesson 1) in disguise
- Neither method is "best": random search wins on flat plains, hill climbing wins when good weights have good neighbors. Smarter methods (next lessons) combine both ideas
The Wall — 4 Weights Are Easy, a Humanoid Is Not
🖼️ Four dials can be guessed. A wall of dials cannot.
- If each weight has 10 possible values, there are 10 × 10 × 10 × 10 = 10,000 policies for CartPole — and luck found a good one in ~17 tries
- With 100 weights: a 1 followed by 100 zeros. Guessing cannot cover that
- A humanoid's neural-network policy has hundreds of thousands of weights
- So bigger robots need methods that use feedback to move in a good direction: evolution strategies (rl-03) and policy gradients / PPO (rl-04, Sutton & Barto Ch.13)
The Loop in Miniature — And a Gate That Respects Luck
🖼️ The gate's margin must be wider than the shaded luck band.
- What you did today is lesson 1's loop, tiny: collect (play episodes) → train (random search) → test (fresh episodes) → keep the champion
- Test the same champion twice and you may get 179.6, then 195.8. That 16.2-point gap is pure luck
- A candidate scoring 192.4 sits inside the luck band, so the data cannot say it is better. A real gate asks for a margin bigger than the noise, or many more test episodes
- Keep this: in rl-09 the gate will use exactly this idea on a walking robot
Real Time — and What You Built
🖼️ 500 steps × 0.02 s = 10 seconds of balancing, computed in a blink.
- Each step is 0.02 s, so 500 steps = 10 seconds and the policy makes 50 decisions per second
- A computer can simulate those 10 seconds in a few thousandths of a second, which is why training can try thousands of episodes (lesson 1: simulation)
- You now have: an environment, a baseline, a 4-weight policy, a trainer, a fair test — and the reasons each one is needed
- Next (rl-03): a crowd of 2D walkers evolving in your browser, where the weights number in the hundreds and guessing no longer works
- Now the quiz: 12 questions, with the real Python behind every number
🎯 Select World Difficulty (选择世界难度)
Choose your challenge level for this course.
💻 Current Python Code (当前Python代码):
Output:
⚠️ Critical Hit!
Your player's health is low! (您的玩家生命值很低!)
🎉 Course Complete!
🏆
Cart Apprentice!
⏱️
✅
✅ You finished RL Lesson 2 — CartPole, the random baseline, linear policies, random search, noisy scores, and the limits of guessing.