From Experience to Action

How Robots Learn

Yunsung Lee

WoRV / MaumAI

2026-10-23

From robot data to robot behavior

W06 · Action as a New Modality

What experience do robots learn from?

Today: How does experience change their actions?

The semester in one picture

Semester timeline, week 09 highlighted AUG SEP OCT NOV DEC Sep 4 Sep 11 holiday Oct 2 holiday report today Oct 23 Nov 6 Nov 20 Nov 27 Dec 4 essay Lecture Guest lecture Keynote Leadership talk holiday / report / essay week

How this course runs

  • Attendance: every session
  • Assignments and deadlines: the LMS
  • Questions: Wooclap throughout the lecture

Official notices, assignments, deadlines and grades live on the DGIST LMS.

Ask anytime

QR code for the question wall
Wooclap code ZNQKOGFapp.wooclap.com/ZNQKOGF

Questions now. Answers shared after the lecture.

Making coffee

Coffee tool dropPhysical Intelligence · Nov 2025 · autonomy not stated · source playback speed

Today’s route

01Robot tasks
02Learning signals
03Action sequences
04Requests + actions
05Predicted outcomes
06Plans + execution

From a robot’s task to the decisions behind its actions.

Three action domains

Several abilities in one robot

Helix 02: body and handsFigure · Jan 2026 · autonomy claimed · source playback speed

Observe, choose, act

Teaching adaptation of UC Berkeley CS188, 2026

Policy: a rule or model for choosing actions.

Feedback and learning

During execution

New observation → Next action

During learning

Experience → Updated policy

Why manipulation, and why now?

Precise actions, sustained over many steps

Long-horizon tasks require adaptation and recovery.

Robot foundation models are accelerating progress.

Generalization and reliability remain open challenges.

Earlier long-horizon manipulation (2021) · π0.5 (2025) · π*0.6 (2025)

From one task to reusable skills

Hu et al., 2024 · Bai et al., 2026 · Teaching synthesis

Three directions we will connect

These directions can work together in one robot.

Selected, compatible directions · Teaching synthesis of Bai et al., 2026 and Hou et al., 2026

How does experience improve a policy?

02 · Learning signals

01Robot tasks
02Learning signals
03Action sequences
04Requests + actions
05Predicted outcomes
06Plans + execution

What tells the model how to improve?

Self-supervised learning

Masked input

Prediction

Original image

He et al., 2022

The data supplies the comparison.

Behavior cloning

Imitation learning · Berkeley CS185/285

Observation, action, outcome

Role Coffee example
Observation The gripper is open
Action Command the gripper to close
Later outcome The tool is grasped

Illustrative record, not a recovered control log.

An action can create an unfamiliar scene

Next observation differs from the demonstrations

Physical Intelligence, 2025 · Ross et al., 2011

DAgger: Dataset Aggregation

Ross et al., 2011

The expert labels the visited situation

The expert labels the visited situationSchematic: a photo of the coffee tool tilted in a situation the demonstrations never showed is sent to the expert, who supplies an action; the situation and action form a new training pair that is added to the data before retraining; the last state asks what happens if only a score is given instead of an action. Situation the robot visited The tool is tilted ask the expert Action supplied by the expert Adjust the grip before lifting situation + action → new training pair Added to the data, then retrain Imitation: the expert supplies the action to copy What if no one gives the action, only a score for how it went?

Physical Intelligence, 2025 · Illustrative correction. Queries can cover successful states too.

Reinforcement learning (RL)

MIT 6.S191, 2025 · A conceptual reward example

The consequence can arrive later

The consequence can arrive laterSchematic: three steps of the coffee task happen in order (grip the handle, move the tool, fit it into the machine) and the reward arrives only after the third; the last state shows that the grip chosen at step 1 decided the outcome at step 3, so learning must look at the rewards that follow. Reward arrives here Grip the handle Move the tool Fit it into the machine 1 2 3 earlier later No reward yet while the first two choices are madeThe score comes only when the tool fits, or jams The grip chosen at step 1 decided the outcome at step 3Learn from the rewards that follow, not only the next moment

Collect experience, then learn from it

Collect experience, then learn from itSchematic: each attempt is one record of observation, action, outcome and reward, with a failed attempt kept alongside successes; the pile of records is used to improve the policy, and many attempts can come from practice in simulation. The three example rows are illustrative, not measured data. ObservationActionOutcomeReward Tool at the machinePush inFitted✓ +1 Tool tiltedLiftDropped✗ −1 Cup under the spoutPress brewCoffee flows✓ +1 One attempt per row; the failed row is kept too Improve the policy from every record Need many attempts? practice in simulation next

Practice in simulation

Learning to walk: simulationETH Zurich / NVIDIA · Oct 2021 · autonomous · source playback speed

Execute on a physical robot

Learning to walk: physical robotETH Zurich / NVIDIA · Oct 2021 · autonomous · source playback speed

Different signals can work together

Approach Learning signal Where it falls short
Self-supervision The data itself No action to copy
Imitation Expert actions Unfamiliar scenes
Reinforcement learning Rewards Reward design, many attempts

03 · Action sequences

01Robot tasks
02Learning signals
03Action sequences
04Requests + actions
05Predicted outcomes
06Plans + execution

How does a policy produce movement?

ALOHA: collecting demonstrations

ALOHA: demonstration collectionZhao et al. · Apr 2023 · teleoperated · 10x source speed

A Low-cost Open-source Hardware System for Bimanual Teleoperation

Action Chunking with Transformers (ACT)

ACT predicts a chunk of future joint targets, and three reasons whySchematic: eight blank cells stand for a chunk of k = 100 joint targets; they carry no values and are not drawn to time scale, and temporal ensembling is not shown. Images + robot state ACT One prediction: a chunk of k joint targets k = 100 targets = 2 s at 50 Hz (one every 20 ms) Why predict a chunk instead of one step? Fewer decisions k targets per prediction, so small errors pile up k times more slowly k = 1: 1% → k = 100: 44% Fewer decisions k targets per prediction, so small errors pile up k times more slowly k = 1: 1% → k = 100: 44% Keeps human rhythm Pauses and linked motions are learned together, not per step Fewer decisions k targets per prediction, so small errors pile up k times more slowly k = 1: 1% → k = 100: 44% Keeps human rhythm Pauses and linked motions are learned together, not per step One answer, k steps A large network answers slowly, but motors need a target every 20 ms

Zhao et al., 2023 · Success rates from the paper’s simulated ablation without temporal ensembling

A chunk is a timed sequence of targets

A prepared eight-action chunk sends one target every20ms; the low-level motor controller tracks each target.Target times after the chunk is ready020406080100120140msBuffered chunkA0A1A2A3A4A5A6A7ControllerA0Eight targets are ready togetherBuffered chunkA0A1A2A3A4A5A6A7ControllerA1One target is sent every 20 msBuffered chunkA0A1A2A3A4A5A6A7ControllerA3The motor controller tracks each targetnow

Zhao et al., 2023 · Illustrative timeline; animation slowed for explanation

ACT combines predictions for the same moment

ACT predicts overlapping chunks and combines predictions for the same physical execution time.0 ms20 ms40 ms60 ms80 ms100 ms120 msObserve 0 msA0A1A2A3A4Predict several future targetsObserve 0 msA0A1A2A3A4Observe 20 msB0B1B2B3B4A new observation produces another chunkObserve 0 msA0A1A2A3A4Observe 20 msB0B1B2B3B4Observe 40 msC0C1C2C3C4Predictions overlap in physical timeObserve 0 msA0A1A2A3A4Observe 20 msB0B1B2B3B4Observe 40 msC0C1C2C3C4Combine A2, B1 and C0 for 40 ms

Zhao et al., 2023 · Inference time omitted; overlapping target times

ACT opens a small lid

ACT: opening a lidZhao et al. · Apr 2023 · autonomous · real-time source speed

More than one good way to move

Illustrative example. Diffusion Policy · Chi et al., 2023

Diffusion models generate by repeated denoising

Random noise → repeated learned updates → image

Ho et al., 2020 · Original DDPM process figure, unchanged

Learn a denoising update, then reuse it

Diffusion training predicts known injected noise; generation iteratively updates noisy data with a trained model.Training: add known random noiseImage+ noiseNoisy image→→We know exactly which noise was addedTraining: predict the added noiseNoisy imageModelNoise estimate→→Compare the estimate with that known noiseThe model also receives the noise levelGeneration: repeat learned updatesRandom noiseModel updatesNew image→→Reuse the trained model; do not train againNoise level changes with each update

Ho et al., 2020 · Teaching schematic

Diffusion Policy denoises an action chunk

Diffusion Policy refines a whole action chunk: the same eight future commands shown as numbers and as planned pusher positions on the push-T table Schematic: the eight one-dimensional values and the dot paths on a stylised top view of the push-T table are illustrative, not model output or a recorded trajectory. Future commands, one number each (schematic) Action 0 Action 7 Every update uses the same observation Each cell is where the pusher will be at that moment Columns are future robot time, not computation steps Top view of the push-T table Dots = planned pusher positions 0.72 -0.83 0.55 -0.63 0.85 -0.48 0.92 -0.35 Random action candidate 0.10 0.52 -0.13 0.55 0.12 0.66 0.23 0.79 Denoising update 1 0.13 0.28 0.45 0.60 0.50 0.30 0.27 0.22 Denoising update 2 0.15 0.32 0.49 0.55 0.48 0.33 0.23 0.25 Ready action chunk pusher

Chi et al., 2023 · Schematic action values, not model output

ACT vs. Diffusion Policy: how the chunk is made

ACT and Diffusion Policy both turn the current observation into an action chunk, but generate it differently Schematic block diagram: ACT produces the chunk in one pass through its network, while Diffusion Policy starts from noise and repeats a denoising update K times; box sizes and spacing do not represent network size, compute or timing. ACT Diffusion Policy Observation Action chunk Observation Action chunk One pass through the network Noise Denoising update × K Both: a chunk of future commands, execute part of it, observe again one answer per observation K updates before the first command; many paths possible

Zhao et al., 2023 · Chi et al., 2023

Predicted actions and actual movement

Diffusion Policy: action candidatesChi et al. · Mar 2023 · autonomous · source playback speed

Diffusion Policy manipulates a mug

Diffusion Policy: mug orientationChi et al. · Mar 2023 · autonomous · 4x source speed

Predict farther than you execute

Receding horizon executes a short prefix then updates from new observations; numbers are an illustrative zero-latency schematic.Predict eight future actionsOld predictionA0A1A2A3A4A5A6A7Prediction horizon = 8Remaining old predictions can be replacedExecute the first threeOld predictionA0A1A2A3A4A5A6A7Execution horizon = 3Remaining old predictions can be replacedObserve again and shift the predictionOld predictionA0A1A2A3A4A5A6A7New predictionB0B1B2B3B4…Remaining old predictions can be replaced

Chi et al., 2023 · Illustrative horizons; inference delay omitted here

Observation and action are separated by delay

Example20ms command period and60ms inference. Real observation-to-action delay includes sensing, inference, communication and scheduling.Capture a sceneCameraModelNew command0 ms20 ms40 ms60 ms80 ms100 msOAn empty action buffer means waitingCompute the action chunkCameraModelNew command0 ms20 ms40 ms60 ms80 ms100 msO60 ms inferenceAn empty action buffer means waitingThe world has moved during inferenceCameraModelNew command0 ms20 ms40 ms60 ms80 ms100 msO60 ms inferenceNewImage age + inference + communication + scheduling

Black et al., 2025 · Illustrative 20 ms commands and 60 ms inference

Generate the next chunk while the robot moves

Asynchronous example: observe at60ms, compute until120ms while old actions execute, then consume new index3 aligned to120ms.020406080100120140160180200Time / msExecute ACompute BExecute BA0A1A2A3A4A5A6A7Keep a prepared chunk in the action bufferExecute ACompute BExecute BA0A1A2A3A4A5A6A760 to 120 msGenerate B while A3, A4 and A5 executeExecute ACompute BExecute BA0A1A2A3A4A5A6A760 to 120 msB0B1B2B3B4B5B6B7B0 to B2 refer to times that have already passedExecute ACompute BExecute BA0A1A2A3A4A5A6A760 to 120 msB0B1B2B3B4B5B6B7At 120 ms, switch to B3; do not restart at B0

Black et al., 2025 · Illustrative asynchronous timeline

A smooth handoff also needs compatible actions

RTC guidance uses the committed prefix, decaying guidance for later overlap and freely generated tail; it is not simple output averaging.A new chunk must agree with committed actionsA360B0A480B1A5100B2~A6120B3~A7140B4new160B5new180B6new200B7Time / msNew chunkExecuted by ACan changeNew futureB0, B1 and B2 are fixed to A3, A4 and A5Asynchronous scheduling alone does not ensure continuityKeep later overlap close, while allowing correctionsA360B0A480B1A5100B2~A6120B3~A7140B4new160B5new180B6new200B7Time / msNew chunkExecuted by ACan changeNew futureA6 and A7 guide the handoff, but may be revisedAsynchronous scheduling alone does not ensure continuityGenerate the remaining futureA360B0A480B1A5100B2~A6120B3~A7140B4new160B5new180B6new200B7Time / msNew chunkExecuted by ACan changeNew futureB5, B6 and B7 extend beyond the old chunkAsynchronous scheduling alone does not ensure continuity

RTC · Black et al., 2025 · Schematic guidance regions

Keep the short future, update with feedback

Question What we just saw
What teaches it? Demonstrations
What does it output? A short action chunk
How is it generated? ACT or Diffusion Policy

Prepare ahead. Observe again. Update what comes next.

The examples here learn from demonstrations; these are compatible design choices.

04 · Requests + actions

01Robot tasks
02Learning signals
03Action sequences
04Requests + actions
05Predicted outcomes
06Plans + execution

How can one model handle different requests?

“Pick up the red marker with the left hand”

pi0.7: follow an instructionPhysical Intelligence · Apr 2026 · autonomous · 1x / 2x source speed

Vision-Language Model (VLM)

A VLM answers in text; an action head turns it into a VLASchematic: the photo is a real frame from a robot demonstration video, but the question, the text reply and the four action cells are illustrative, not recorded model output. Which object is the red marker? VLM the model behind ChatGPT, Claude text out The red marker, left of the calculator. Image + text in, text out: a VLM Action head actions out VLM + action head = Vision-Language-Action (VLA)

Physical Intelligence, 2026 · Desk frame from the π0.7 demo; the reply text is illustrative

Connecting robot experience

OpenVLA · Kim et al., 2024 · Conceptual training inputs

These choices can live in one model

VLA: observation + instruction → action

Design question Compatible choices
How does it learn? Imitation, RL
What does it output? Individual actions, chunks
How does it generate them? Token prediction, diffusion

Teaching synthesis · Bai et al., 2026 · OpenVLA · π0

One foundation, many tasks

pi0.7: serving coffeePhysical Intelligence · Apr 2026 · autonomous · 5x source speed

Robot Foundation Model (RFM)

Gemini Robotics On-Device 2: six tasks, three robot typesGoogle DeepMind · Jul 2026 · autonomy claimed · 1x to 4x source speed

Google DeepMind, 2026 · One on-device VLA, adapted to each robot type with a few hours of data (publisher description)

What changed in the test?

  • New scene: cup color or position
  • New task: a different request
  • New robot: a different arm or gripper

What extra experience did the model receive?

Can experience improve reliability?

Coffee tool dropPhysical Intelligence · Nov 2025 · autonomy not stated · source playback speed

RECAP: experience and corrections

Layers of experienceSchematic of three layers of experience, from broad pretraining through coffee demonstrations to the robot's own attempts with expert corrections, with the last two tagged by the learning signals from chapter 2. Broad pretraining Many robots, many tasks Coffee demonstrations A person shows the task The robot's own attempts Successes, failures, expert corrections Imitation (ch. 2) Outcomes → RL (ch. 2) Corrections → DAgger idea New here: the robot learns from what it did itself

RECAP · Physical Intelligence, 2025
RL with Experience and Corrections via Advantage-conditioned Policies

Learning to evaluate progress

Repeated dropsProgress resumes
Original RECAP coffee value chart with negative value axis and time axis
Learned value ↑Time →

Physical Intelligence, 2025 · Original figure, only empty margins outside the viewport

Using evaluation to improve actions

Better or worse than expectedSchematic only: two recorded attempts, one action tagged better than the expected score and one tagged worse, with both tags used to train the next policy; the marker positions are illustrative, not measured. Attempt A Tool fitted on the first try expected better ↑ Attempt B Tool dropped, then retried expected worse ↓ Compare each recorded action with the score expected there The tag is the learning signal, not a replay of successes Train the next policy Both tags are used; at run time, ask for ‘better’

RECAP · Physical Intelligence, 2025 · Illustrative attempts

Collect, learn, deploy again

RECAP · Physical Intelligence, 2025

Successful espressos per hour

Unchanged original RECAP espresso throughput chart with four bars

Task demonstrations

On the pretrained model

+ RECAP experience

Attempts and corrections

Physical Intelligence, 2025 · Original result. SFT: Supervised Fine-Tuning. Throughput: successes per hour.

Experience can teach actions and consequences

Original Stanford diagram: robot interaction data supports both foundation policies and foundation world models, with arrows connecting the two

Policy
Choose an action

World model
Predict its consequences

Yunzhu Li, Stanford CS231n, 2025, slide 96 · Original diagram; surrounding title and footer cropped

05 · Predicted outcomes

01Robot tasks
02Learning signals
03Action sequences
04Requests + actions
05Predicted outcomes
06Plans + execution

What might happen after an action?

The action changes the predicted outcome

Closed gripper

Open gripper

Both rows show model predictions.

V-JEPA 2 · Assran et al., 2025 · Video Joint Embedding Predictive Architecture

Compare predicted outcomes, then act

Model predictive control · Stanford CS231n, 2025 · Illustrative plan

World Action Model (WAM)

A conceptual family of designs. DreamZero, 2026

DreamZero: future video and robot actions

DreamZero, 2026 · Simplified inference; coffee icon is illustrative

Generated future

DreamZero: generated future ⚠DreamZero / NVIDIA · Feb 2026 · autonomy not stated · source playback speed

⚠ Example flagged by the authors. This video is generated, not recorded.

Physical execution

DreamZero: physical execution ⚠DreamZero / NVIDIA · Feb 2026 · autonomous · 3x source speed

⚠ Example flagged by the authors. This is the observed robot execution.

Prediction returns to observation

Predict → Act → Observe

What should happen next in the whole task?

06 · Plans + execution

01Robot tasks
02Learning signals
03Action sequences
04Requests + actions
05Predicted outcomes
06Plans + execution

Who chooses the next task?

Two modes of human thinking

System 1

Fast, automatic

Recognize your familiar mug.

System 2

Deliberate, effortful

Plan around a broken coffee machine.

A way to describe thinking, not two separate brain modules.

Stanovich & West (2000) · Kahneman, Thinking, Fast and Slow (2011)

Two systems, connected by feedback

System 2: goals and plans. System 1: separate action execution.

The result can change the next subgoal

Hi Robot: VLM and VLA

Original Hi Robot architecture

VLM: choose the next task

↓ Instruction

VLA: execute the task

Hi Robot, 2025

“Get me something sweet that’s not in the basket”

Hi Robot: a new requestHi Robot authors · Feb 2025 · autonomous · 6x source speed

Three routes to robot action

RoboDojo-Sim model Score Success
PhysicalRSI 36.27 31.38%
GPT-6 Astra 28.97 22.48%
VPP2 39.26 32.26%

Different systems. Three selected successful rollouts.

RoboDojo leaderboard · Average, including Agent entries · v2026.10.9, checked Oct 12, 2026

PhysicalRSI: planning and separate execution

PhysicalRSI 1.0 · Skills can include a VLA or code policies

PhysicalRSI: align and insert a key

PhysicalRSI: key insertion (simulation)HKU MMLab & KAI / RoboDojo · Sep 2026 · autonomous · source playback speed

GPT-6 Astra: from images to movement targets

No separate learned motor policy in this evaluation.

RoboDojo Astra evaluation · Evaluation paper

Astra: read the counts, then press

Astra: count and press (simulation)RoboDojo Team · Sep 2026 · autonomous · source playback speed

VPP2: video prediction supports action

VPP2: Video Prediction Policy 2 (2026) · Separate VLM planner evaluated elsewhere

VPP2: place bread in both toaster slots

VPP2: bread placement (simulation)RobotEra / RoboDojo · Oct 2026 · autonomous · source playback speed

Reasoning, skills and feedback

Model What it produces What executes it
PhysicalRSI Plans and skills Skills / code
GPT-6 Astra Movement targets Control code
VPP2 Action sequences Robot controller

All three use feedback from execution.

Back to the dropped coffee tool

Coffee tool dropPhysical Intelligence · Nov 2025 · autonomy not stated · source playback speed

Recover now. Learn for the next attempt.

What did it learn, and how does it act?

QR code for the question wall
Wooclap code ZNQKOGFapp.wooclap.com/ZNQKOGF

Leave a question. Answers follow after the lecture.

+

Further detail

Reference material after the lecture

Locomotion on a changing surface

RMA: locomotion on soft groundKumar et al. · Jul 2021 · autonomous · source playback speed

Navigation among obstacles

X-NavDP: obstacle navigationX-NavDP authors · Jul 2026 · autonomy claimed · source playback speed

ACT: architecture and temporal ensembling

Original ACT architecture panel
Original action chunking and temporal ensembling panel

Zhao et al., 2023 · Enlarged original panels · Full original figure

Diffusion Policy: prediction and execution

Chi et al., 2023

DreamZero: original architecture

DreamZero, 2026

World prediction at training and execution

Train with future video. Execute with actions.

InternW0-Delta, 2026

Learning and action modeling: the survey view

Bai et al., 2026, Fig. 5 · Unchanged original · CC BY 4.0

Selected references

Surveys and learning resources