State
What the robot knows right now. Here: which square it's on.
Engineering · Robot Autonomy · Machine Learning
How do you teach a robot without telling it every step? You give it points! Points for good things, and “oops” for bad things. The robot tries again and again until it learns the best way to its charger.
Nobody programs every step of a robot dog's gait anymore. Instead, engineers write down what counts as good and bad, and let the robot practice — millions of times, usually in simulation. That's reinforcement learning. Train a small robot to find its charger without falling down the stairs, and watch its "brain" fill in as it learns.
Explorer · Story time
When you teach a puppy to sit, you don’t explain it with words. When the puppy sits, it gets a treat. Soon it learns: sitting is good!
I learn the same way. Nobody tells me the path to my charger. When I reach it, I get +1 point. If I roll down the stairs, I get −1. Every extra step costs a tiny bit, so I learn to hurry.
At first I bump around. But each time, I remember which squares led to good things. Green squares mean “this way is good!” Striped red squares mean “careful, trouble nearby!”
The idea
In motion planning a person writes the algorithm. In reinforcement learning (RL), a person writes only the reward: +1 for reaching the charger, −1 for the stairs, a small cost for every step. The robot has to figure out the rest by trying.
What the robot knows right now. Here: which square it's on.
What it can do: move up, right, down, or left.
The score for what just happened. The robot tries to maximize the total reward, not just the next one.
The robot's strategy: "in this state, do that." Learning = improving the policy. The arrows in the sim are the policy.
Sim · Q-learning
Press Train and watch the robot practice. Green squares are good places to be. Red striped squares are near trouble. The little arrows show which way the robot has learned to go. When the arrows look good, press Test the policy to see if it found the charger!
Press Train. At first the robot stumbles randomly. Every move, it updates a number called a Q-value: "how good is taking this action from this square?" Green squares have learned they lead to the charger; red ones have learned trouble is near. The arrow is the best action the robot knows so far.
Keyboard: Tab into the grid, then ←↑→↓ move between squares and read what the robot has learned there. Enter places the selected tool.
Train with the speed slider low. Where do green squares show up first: near the robot, or near the charger?
Pick Charger and tap a new square. Press Test the policy. Does the robot know about the new spot yet?
Pick Wall and build a maze. Press Erase the brain, then Train. Can your robot solve it?
Train at low speed for 30 seconds. Where does green appear first — near the start or near the charger? Why?
Erase the brain, set ε to 0, and train. Then try ε = 0.2. Why does a robot that never explores often get stuck with a bad route?
After training, move the charger. Test the policy. The robot still goes to the old spot. How long does relearning take?
Add wheel slip of 0.2 and retrain. Does the learned route still hug the stairs, or does it keep a safer distance?
The update rule
Q(s, a) ← Q(s, a) + α · [ reward + γ · max Q(s′, ·) − Q(s, a) ]
In words: "My old guess for this move, nudged toward what actually happened: the reward I just got, plus the best I think I can do from where I landed." α sets the size of the nudge. γ sets how much future reward matters. Good news slowly flows backward from the charger, one square at a time. That's why green spreads outward from the goal.
With probability ε the robot tries a random move instead of its best one. Too little and it never finds better routes. Too much and it never uses what it knows.
Robots do exactly what the reward says, not what you meant. Remove the step cost and a robot may wander forever, since nothing tells it to hurry.
Engineer deep dive
Q*(s, a) = E[ r + γ · maxa′ Q*(s′, a′) ]
Q-learning is a stochastic approximation of this fixed point. It converges to Q* if every state–action pair is visited infinitely often and the learning rate decays so that Σα = ∞ and Σα² < ∞ (Watkins & Dayan, 1992). The sim's fixed α trades guaranteed convergence for faster adaptation.
A reward k steps away is worth γk now. The rough planning horizon is 1 / (1 − γ): at γ = 0.95 that is 20 steps; at 0.99, 100 steps. Set γ too low and the charger's reward fades before it reaches the start square.
Q-learning updates toward max Q(s′, ·), the greedy action, even while it explores. That is why it learns the risky edge-of-the-stairs route under slip. SARSA updates toward the action it actually takes next, so it learns a safer route that accounts for its own random moves. Try predicting SARSA's route on this map.
DQN (Mnih et al., 2015) replaces the table with a neural network Q(s, a; θ) and adds two stabilizers: an experience replay buffer that breaks correlation between samples, and a slowly updated target network for the max Q term. Robot locomotion typically uses policy-gradient methods such as PPO instead.
From grid to real robots
This robot has 96 squares × 4 moves = 384 numbers to learn, so a table works. A real walking robot's state is dozens of joint angles, speeds, and camera pixels, which is far too many for any table. Deep RL swaps the table for a neural network, built from the same neurons as the perceptron, which estimates Q-values or picks actions directly.
Robots train in physics simulators, running thousands of copies faster than real time. The policy then moves onto the real robot. The simulator randomly changes friction, weight, and lighting so the policy can handle the messy real world.
Legged-robot walking and recovery, dexterous robot hands, drone racing, and warehouse grasping. DeepMind's AlphaGo used RL too — see the Go AI lesson.
RL needs huge amounts of practice, and practice on real hardware breaks things. That's why most robots still combine learned skills with the classic methods in this series.
Where you’ll see it
Click any photo to enlarge it.
Robot autonomy series
Quick check
1. What is a reward?
2. Why does the robot get better after many tries?
3. Why should the robot sometimes try a random move?
1. In reinforcement learning, what does the human designer mainly provide?
2. Why do learned values spread outward from the charger?
3. What does ε (epsilon) control?
4. Why does deep RL use a neural network instead of a Q-table?
5. With ε = 0.2 exploration, which learns a route that stays further from the stairs?