Engineering · Robot Autonomy · Machine Learning

Learning by Trial & Error

How do you teach a robot without telling it every step? You give it points! Points for good things, and “oops” for bad things. The robot tries again and again until it learns the best way to its charger.

Nobody programs every step of a robot dog's gait anymore. Instead, engineers write down what counts as good and bad, and let the robot practice — millions of times, usually in simulation. That's reinforcement learning. Train a small robot to find its charger without falling down the stairs, and watch its "brain" fill in as it learns.

Ages 6–18 · 3 levels45–60 minutesLive Q-learningEditable world
← Back to Engineering
A grid where arrows point along a learned route from the robot to a lightning-bolt charger, avoiding red stair cells ⚡

Explorer · Story time

Learning like a puppy

When you teach a puppy to sit, you don’t explain it with words. When the puppy sits, it gets a treat. Soon it learns: sitting is good!

I learn the same way. Nobody tells me the path to my charger. When I reach it, I get +1 point. If I roll down the stairs, I get −1. Every extra step costs a tiny bit, so I learn to hurry.

At first I bump around. But each time, I remember which squares led to good things. Green squares mean “this way is good!” Striped red squares mean “careful, trouble nearby!”

  • Reward · points for doing well
  • Try again · practice makes better
  • Explore · try something new
  • Learn · remember what worked

Play it as a class

  1. Hot and cold. Hide an object. One student searches while the class says only “warmer” or “colder.” Those words are rewards! How many tries until they find it?
  2. Floor grid. Tape a 4 × 4 grid on the floor. Put a “charger” paper in one square and “stairs” in another. The robot student steps one square at a time, and the class gives a thumbs up or down. Write a sticky-note arrow in each square once the robot learns the best way.

The idea

Rewards instead of instructions

In motion planning a person writes the algorithm. In reinforcement learning (RL), a person writes only the reward: +1 for reaching the charger, −1 for the stairs, a small cost for every step. The robot has to figure out the rest by trying.

State

What the robot knows right now. Here: which square it's on.

Action

What it can do: move up, right, down, or left.

Reward

The score for what just happened. The robot tries to maximize the total reward, not just the next one.

Policy

The robot's strategy: "in this state, do that." Learning = improving the policy. The arrows in the sim are the policy.

Sim · Q-learning

Train the robot

Press Train and watch the robot practice. Green squares are good places to be. Red striped squares are near trouble. The little arrows show which way the robot has learned to go. When the arrows look good, press Test the policy to see if it found the charger!

Press Train. At first the robot stumbles randomly. Every move, it updates a number called a Q-value: "how good is taking this action from this square?" Green squares have learned they lead to the charger; red ones have learned trouble is near. The arrow is the best action the robot knows so far.

  • Learned: good
  • Learned: trouble
  • Wall
  • Stairs (hazard stripe)
  • Charger

Keyboard: Tab into the grid, then ←↑→↓ move between squares and read what the robot has learned there. Enter places the selected tool.

Learning
Edit the world (click squares)

1 · Watch

Train with the speed slider low. Where do green squares show up first: near the robot, or near the charger?

2 · Move the charger

Pick Charger and tap a new square. Press Test the policy. Does the robot know about the new spot yet?

3 · Build a maze

Pick Wall and build a maze. Press Erase the brain, then Train. Can your robot solve it?

1 · Watch it learn

Train at low speed for 30 seconds. Where does green appear first — near the start or near the charger? Why?

2 · Curiosity

Erase the brain, set ε to 0, and train. Then try ε = 0.2. Why does a robot that never explores often get stuck with a bad route?

3 · Change the world

After training, move the charger. Test the policy. The robot still goes to the old spot. How long does relearning take?

4 · Risk

Add wheel slip of 0.2 and retrain. Does the learned route still hug the stairs, or does it keep a safer distance?

The update rule

One line of learning

Q(s, a) ← Q(s, a) + α · [ reward + γ · max Q(s′, ·) − Q(s, a) ]

In words: "My old guess for this move, nudged toward what actually happened: the reward I just got, plus the best I think I can do from where I landed." α sets the size of the nudge. γ sets how much future reward matters. Good news slowly flows backward from the charger, one square at a time. That's why green spreads outward from the goal.

Explore vs. exploit

With probability ε the robot tries a random move instead of its best one. Too little and it never finds better routes. Too much and it never uses what it knows.

Reward design

Robots do exactly what the reward says, not what you meant. Remove the step cost and a robot may wander forever, since nothing tells it to hurry.

Engineer deep dive

Bellman, discounting, and convergenceAges 15+

The Bellman optimality equation

Q*(s, a) = E[ r + γ · maxa′ Q*(s′, a′) ]

Q-learning is a stochastic approximation of this fixed point. It converges to Q* if every state–action pair is visited infinitely often and the learning rate decays so that Σα = ∞ and Σα² < ∞ (Watkins & Dayan, 1992). The sim's fixed α trades guaranteed convergence for faster adaptation.

Effective horizon

A reward k steps away is worth γk now. The rough planning horizon is 1 / (1 − γ): at γ = 0.95 that is 20 steps; at 0.99, 100 steps. Set γ too low and the charger's reward fades before it reaches the start square.

Off-policy vs. on-policy

Q-learning updates toward max Q(s′, ·), the greedy action, even while it explores. That is why it learns the risky edge-of-the-stairs route under slip. SARSA updates toward the action it actually takes next, so it learns a safer route that accounts for its own random moves. Try predicting SARSA's route on this map.

Deep Q-Networks

DQN (Mnih et al., 2015) replaces the table with a neural network Q(s, a; θ) and adds two stabilizers: an experience replay buffer that breaks correlation between samples, and a slowly updated target network for the max Q term. Robot locomotion typically uses policy-gradient methods such as PPO instead.

From grid to real robots

Deep reinforcement learning

This robot has 96 squares × 4 moves = 384 numbers to learn, so a table works. A real walking robot's state is dozens of joint angles, speeds, and camera pixels, which is far too many for any table. Deep RL swaps the table for a neural network, built from the same neurons as the perceptron, which estimates Q-values or picks actions directly.

Sim-to-real

Robots train in physics simulators, running thousands of copies faster than real time. The policy then moves onto the real robot. The simulator randomly changes friction, weight, and lighting so the policy can handle the messy real world.

Where it works

Legged-robot walking and recovery, dexterous robot hands, drone racing, and warehouse grasping. DeepMind's AlphaGo used RL too — see the Go AI lesson.

Where it doesn't (yet)

RL needs huge amounts of practice, and practice on real hardware breaks things. That's why most robots still combine learned skills with the classic methods in this series.

Where you’ll see it

Robots that learned by practicing

Click any photo to enlarge it.

A yellow Boston Dynamics Spot four-legged robot walking beside airmen on a runway
Legged robots Four-legged robots like Spot must balance on ice, stairs, and gravel. Engineers now train walking controllers with reinforcement learning in simulation, then move them to the real robot.Photo: Senior Airman John Ennis · Public domain · Wikimedia Commons
NASA’s Valkyrie humanoid robot standing against a dark background
NASA’s Valkyrie A 1.8 m humanoid built for the DARPA Robotics Challenge. Humanoids have dozens of joints, far too many states for a Q-table, which is why deep RL uses neural networks.Photo: NASA/Bill Stafford, James Blair, Regan Geeseman · Public domain · Wikimedia Commons
A diagram of a Go board showing the numbered moves of game 4 between Lee Sedol and AlphaGo
Lee Sedol vs. AlphaGo, 2016 AlphaGo learned partly by playing millions of games against itself, using the same reward idea as our robot. Lee Sedol won this game (Game 4), his only win of the match.Photo: Wesalius · CC BY-SA 4.0 · Wikimedia Commons

Robot autonomy series

Sense · Think · Act

Quick check

Lock it in

1. What is a reward?

2. Why does the robot get better after many tries?

3. Why should the robot sometimes try a random move?

1. In reinforcement learning, what does the human designer mainly provide?

2. Why do learned values spread outward from the charger?

3. What does ε (epsilon) control?

4. Why does deep RL use a neural network instead of a Q-table?

5. With ε = 0.2 exploration, which learns a route that stays further from the stairs?