Action as a new modality and the scaling of robot learning dataInstructor-authored schematic. complete. Reference mapping is in the accompanying Quarto source. Action as a New Modalityand the Scaling of Robot Learning DataYunsung Lee · WoRV, MaumAIDGIST · October 2, 2026

Seminar for Comprehensive Competency Cultivation · HSS118
AI: Artificial Intelligence

Connecting our previous lectures

Reconnect embodied action and video prediction to the question of learning experienceInstructor-authored schematic. complete. Reference mapping is in the accompanying Quarto source. ActPredictLearn fromexperienceW02W03Today

W02: The Paradigm Shift Toward Embodied AI. W03: Hyeongmin Lee, DGIST (2026), slide 31.

The semester in one picture

Semester timeline, week 06 highlighted AUG SEP OCT NOV DEC Sep 4 Sep 11 holiday today Oct 2 holiday report Oct 23 Nov 6 Nov 20 Nov 27 Dec 4 essay Lecture Guest lecture Keynote Leadership talk holiday / report / essay week

Course series: DGIST HSS118, Fall 2026

How this course runs

  • Attendance: every session
  • Assignments: the Learning Management System (LMS)
  • Questions: Wooclap throughout the lecture

Official notices, assignments, deadlines and grades live on the DGIST LMS.

Ask anytime

QR code for the question wall
Wooclap code ZNQKOGFapp.wooclap.com/ZNQKOGF

Leave a question at any time.

https://app.wooclap.com/ZNQKOGF

What experience would teach this?

Open the drawer and put an empty cup insideInstructor-authored schematic. 1. Reference mapping is in the accompanying Quarto source. FindOpenPut inside
Open the drawer and put an empty cup insideInstructor-authored schematic. 2. Reference mapping is in the accompanying Quarto source. FindOpenPut insideWhat experience would teach this?

Own teaching example; record structure informed by ETH Real-World Robotics (2025), p6. © Robert Katzschmann.

What experience teaches a robot?

Lecture route: 1 of 5

1

Learn from
text

2

Output
actions

3

Collect
examples

4

Scale with
people

5

More kinds
of video

Our task: open a drawer and put an empty cup inside.

First, how do models learn from text?

1. What did language models learn from?

Predicting the next piece of text Instructor-authored reconstruction. Wei slide 4; recording 00:35-04:23. Stage: candidates. Schematic charts contain no measured values. Studentslike to ...Languagemodelstudyplaybanana Large Language Model (LLM)
Predicting the next piece of text Instructor-authored reconstruction. Wei slide 4; recording 00:35-04:23. Stage: training feedback. Schematic charts contain no measured values. Studentslike to ...LanguagemodelstudyplaybananaTeaching example: studyAdjust the model's internal numbers.

Adapted from Jason Wei, Stanford CS25 (2024), slide 4. Illustrative scores.

One task, many skills

One prediction rule can teach different skills Instructor-authored reconstruction. Wei slide 6; recording 04:23-08:57. Stage: Grammar question. Schematic charts contain no measured values. In my free time, I like to ...codebanana
One prediction rule can teach different skills Instructor-authored reconstruction. Wei slide 6; recording 04:23-08:57. Stage: Grammar answer. Schematic charts contain no measured values. In my free time, I like to ...codebananaGrammar
One prediction rule can teach different skills Instructor-authored reconstruction. Wei slide 6; recording 04:23-08:57. Stage: Knowledge question. Schematic charts contain no measured values. The capital of South Korea is ...SeoulLondon
One prediction rule can teach different skills Instructor-authored reconstruction. Wei slide 6; recording 04:23-08:57. Stage: Knowledge answer. Schematic charts contain no measured values. The capital of South Korea is ...SeoulLondonKnowledge
One prediction rule can teach different skills Instructor-authored reconstruction. Wei slide 6; recording 04:23-08:57. Stage: Arithmetic question. Schematic charts contain no measured values. 3 + 8 + 4 = ...1511
One prediction rule can teach different skills Instructor-authored reconstruction. Wei slide 6; recording 04:23-08:57. Stage: Arithmetic answer. Schematic charts contain no measured values. 3 + 8 + 4 = ...1511Arithmetic

Adapted from Wei, Stanford CS25 (2024), slide 6.

One passage, many next-token tasks

One passage supplies the next training target: Seoul Instructor-authored reconstruction of the input-prefix and next-target relation in Jason Wei's Stanford CS25 slide 7. The underline moves through Seoul, the period, and It. Current input: South Korea's capital is. Current target: Seoul. Training uses the original text; generation appends model outputs. Token boundaries are illustrative. Original text South Korea's capital is Seoul. It ... Input: everything before the underline Next target South Korea's capital is Seoul Training: the original text supplies the target.
One passage supplies the next training target: . Instructor-authored reconstruction of the input-prefix and next-target relation in Jason Wei's Stanford CS25 slide 7. The underline moves through Seoul, the period, and It. Current input: South Korea's capital is Seoul. Current target: .. Training uses the original text; generation appends model outputs. Token boundaries are illustrative. Original text South Korea's capital is Seoul. It ... Input: everything before the underline Next target South Korea's capital is Seoul . Training: the original text supplies the target.
One passage supplies the next training target: It Instructor-authored reconstruction of the input-prefix and next-target relation in Jason Wei's Stanford CS25 slide 7. The underline moves through Seoul, the period, and It. Current input: South Korea's capital is Seoul.. Current target: It. Training uses the original text; generation appends model outputs. Token boundaries are illustrative. Original text South Korea's capital is Seoul. It ... Input: everything before the underline Next target South Korea's capital is Seoul. It Training: the original text supplies the target. Generation: append the output, then predict again.

Adapted from Wei, Stanford CS25 (2024), slide 7. Teaching passage; illustrative token boundaries.

More training, lower prediction error

More training compute can improve prediction Instructor-authored reconstruction. Wei slide 9 and live chalkboard near 10:00-11:30. Stage: axes. Schematic charts contain no measured values. Prediction errorTraining compute SchematicLower is better ↓
More training compute can improve prediction Instructor-authored reconstruction. Wei slide 9 and live chalkboard near 10:00-11:30. Stage: trend. Schematic charts contain no measured values. Prediction errorTraining compute SchematicLower is better ↓

Schematic reconstruction of Wei’s chalkboard explanation, after Kaplan et al. (2020).

One average, different abilities

One average, different abilitiesUnchanged conceptual curves from Jason Wei, Stanford CS25 slide 13. Instructor labels identify the white average curve, blue grammar curve and green harder-task curve. Down means lower prediction error; right means more training compute.Prediction errorTraining compute →AverageGrammarHarder tasksLower erroris better ↓

Original Jason Wei, Stanford CS25 (2024), slide 13. Original conceptual curves with teaching cues; not measured data.

More compute becomes affordable

Chung original historical compute chart

More computation becomes feasible with a similar budget Instructor-authored reconstruction. Chung slide 7; recording 37:49 onward. Stage: qualitative comparison. Schematic charts contain no measured values. The same budgetEarlierLater$$More computation becomes affordable.

Original Hyung Won Chung, Stanford CS25 (2024), slide 7. Historical hardware cost/performance; qualitative takeaway follows.
GPU: Graphics Processing Unit, a processor for parallel computation.

Same task, different model designs

Two ways to organize the same translation task Instructor-authored simplified design example based on Chung's reading and writing discussion. The blue design has separate reading and writing parts. The teal design shares one model for both. Both learn from data. These examples explain predefined roles; the following curves are conceptual, not measured comparisons of these architectures. One design choice: split or share rolesSeparate rolesEnglishReadWriteGermanShared modelEnglishRead + writeGermanBoth models learn from data.
More and less predefined structure can scale differentlyUnchanged original two-curve conceptual plot from Hyung Won Chung, Stanford CS25 slide 10. The crop and uniform scale match the following three-curve budget comparison. Large instructor cues reset performance upward and training compute rightward.Performance ↑Training compute →Less computeBlue leads here
More and less predefined structure can scale differentlyUnchanged original two-curve conceptual plot from Hyung Won Chung, Stanford CS25 slide 10. The crop and uniform scale match the following three-curve budget comparison. Large instructor cues reset performance upward and training compute rightward.Performance ↑Training compute →More computeTeal can pull ahead

Hyung Won Chung, Stanford CS25 (2024), slides 10 and 37. Teaching diagram and original conceptual curves; not measured architecture comparisons.

The best choice depends on resources

Three designs can scale differentlyOriginal conceptual three-curve plot from Hyung Won Chung, Stanford CS25. Both states use the identical source-coordinate viewport and uniform scale. The budget state adds an instructor label and arrow below the original dashed budget line.Performance ↑Training compute →
Available compute changes the best current choiceOriginal conceptual three-curve plot from Hyung Won Chung, Stanford CS25. Both states use the identical source-coordinate viewport and uniform scale. The budget state adds an instructor label and arrow below the original dashed budget line.Performance ↑Training compute →Available compute

Original Hyung Won Chung, Stanford CS25 (2024), slides 11-12. Original conceptual curves with teaching cues.

One model can read and write

Input and output can be handled with a shared sequence model Instructor-authored reconstruction. Chung slides 22-37, especially 37; simplified architecture example. Stage: separate processing. Schematic charts contain no measured values. Separate rolesInput wordsReadWriteOutput words
Input and output can be handled with a shared sequence model Instructor-authored reconstruction. Chung slides 22-37, especially 37; simplified architecture example. Stage: shared processing. Schematic charts contain no measured values. Separate rolesInput wordsReadWriteOutput wordsShared modelInput + output wordsContinue thesequenceOne model for both

Adapted from Chung, Stanford CS25 (2024), slides 22-37, 54-58, 64-66.

What experience could teach action?

Extend the data question from language to action Instructor-authored reconstruction. Instructor synthesis from both talks and the W06 action-data topic. Stage: bridge. Schematic charts contain no measured values. TextNext wordExperienceNext action?What experience should we collect?

Instructor synthesis after Wei and Chung, Stanford CS25 (2024).

2. Answering with actions

Lecture route: 2 of 5

1

Learn from
text

2

Output
actions

3

Collect
examples

4

Scale with
people

5

More kinds
of video

Text provides examples to learn from.

What would an example of action contain?

Add vision: from LLM to multimodal

Adding vision to a text-based LLM, stage 1 Instructor-authored teaching schematic. Text, vision and audio are examples of information modalities. This LLM-based example takes text and an image, and outputs text. The image contains only a cup and a drawer. The example is an MLLM and a vision-language model; the names are not asserted to be universally equivalent. Audio is a modality example, not an input to the displayed model. Modality = a type of informationTextLLMTextExamples: Text · Vision · AudioStart with one input modality: text.
Adding vision to a text-based LLM, stage 2 Instructor-authored teaching schematic. Text, vision and audio are examples of information modalities. This LLM-based example takes text and an image, and outputs text. The image contains only a cup and a drawer. The example is an MLLM and a vision-language model; the names are not asserted to be universally equivalent. Audio is a modality example, not an input to the displayed model. Text inputWhat's here?Vision inputMultimodalmodelTextMultimodal = more than one modalityText + vision are two modalities.
Adding vision to a text-based LLM, stage 3 Instructor-authored teaching schematic. Text, vision and audio are examples of information modalities. This LLM-based example takes text and an image, and outputs text. The image contains only a cup and a drawer. The example is an MLLM and a vision-language model; the names are not asserted to be universally equivalent. Audio is a modality example, not an input to the displayed model. Text inputWhat's here?Vision inputMLLMLLM + visionA cup anda drawer.Text outputMLLM: Multimodal Large Language ModelVLM: Vision-Language Model (text + vision)

Teaching schematic. ETH Robot Learning (2026); Qwen2.5-VL (2025); Qwen2.5-Omni (2025).

Make the answer a movement

Now act on the sceneInstructor-authored teaching schematic. The output is executable movement rather than a sentence. Illustrative commands, not recorded robot control values. Same imageOpen the drawer. Put the cup inside.VLAMovementVision-Language-Action
The task consists of many small movementsInstructor-authored teaching schematic. Illustrative task sequence; the three labels are not a controller command vector. Illustrative commands, not recorded robot control values. OpenLiftPlaceMany small movements make the task.

Vision-Language-Action (VLA). Teaching schematic after Mees, ETH (2026), 50:15-52:30.

What does the robot need to know?

Inputs for choosing a movementInstructor-authored teaching schematic. 1 input roles visible; a common example, not a universal model requirement. Illustrative commands, not recorded robot control values. What I seeChoose movement
Inputs for choosing a movementInstructor-authored teaching schematic. 2 input roles visible; a common example, not a universal model requirement. Illustrative commands, not recorded robot control values. What I seeWhat to doOpen the drawer.Put the cup inside.Choose movement
Inputs for choosing a movementInstructor-authored teaching schematic. 3 input roles visible; a common example, not a universal model requirement. Illustrative commands, not recorded robot control values. What I seeWhat to doOpen the drawer.Put the cup inside.My bodyHand positionChoose movement

Inputs vary by system. Based on MIT Robotic Manipulation, Fall 2025, Lecture 20, p2.

What is an action?

Actions can be represented as valuesInstructor-authored teaching schematic. 1 command types visible; illustrative directions and units. Illustrative commands, not recorded robot control values. Move+2 cm
Actions can be represented as valuesInstructor-authored teaching schematic. 2 command types visible; illustrative directions and units. Illustrative commands, not recorded robot control values. Move+2 cmTurnRotate the wrist
Actions can be represented as valuesInstructor-authored teaching schematic. 3 command types visible; illustrative directions and units. Illustrative commands, not recorded robot control values. Move+2 cmTurnRotate the wristGripClose the fingers

Illustrative command types and units. After Mees, ETH (2026); Levine, Berkeley CS285, Spring 2026, p12.

Look, move, look again

Look, move, and look againInstructor-authored teaching schematic. First observation and command. Illustrative commands, not recorded robot control values. ObserveChooseMove
Look, move, and look againInstructor-authored teaching schematic. New observation after movement. Illustrative commands, not recorded robot control values. ObserveChooseMoveLook again
Look, move, and look againInstructor-authored teaching schematic. Partial opening prompts another observation and decision. Illustrative commands, not recorded robot control values. ObserveChooseMoveLook againWhat next?Only partly open
A second attempt at the cupOpenVLA authors · Jun 2024 · autonomy claimed · 1.5x speed

Diagram after Stanford CS231n (2025), p29. Video: OpenVLA (2024), publisher’s 1.5x playback.

Connect knowledge to movement

Connect knowledge to physical movementInstructor-authored teaching schematic. Prior language and vision knowledge. Illustrative commands, not recorded robot control values. Language + imagesRecognize the cupUnderstand the goal
Connect knowledge to physical movementInstructor-authored teaching schematic. Add paired observation and command experience. Illustrative commands, not recorded robot control values. Language + imagesRecognize the cupUnderstand the goalAction experienceWhat I saw → what I did

Teaching schematic after Mees, ETH Robot Learning (2026), 52:30-54:20.

Agents act on their environment

An agent observes, acts, and changes its environmentOwn teaching reconstruction of the agent-environment framework. Observation returns from environment to agent. Actions change the drawer and cup. In the final state reward is an illustrative numerical goal score, not a claim about robot feelings or a required demonstration label. Agentuses a policyPerson or robotEnvironmentDrawer closedObservationWhat does the agent see now?
An agent observes, acts, and changes its environmentOwn teaching reconstruction of the agent-environment framework. Observation returns from environment to agent. Actions change the drawer and cup. In the final state reward is an illustrative numerical goal score, not a claim about robot feelings or a required demonstration label. Agentuses a policyPerson or robotEnvironmentDrawer openedObservationAction: pull the handle
An agent observes, acts, and changes its environmentOwn teaching reconstruction of the agent-environment framework. Observation returns from environment to agent. Actions change the drawer and cup. In the final state reward is an illustrative numerical goal score, not a claim about robot feelings or a required demonstration label. Agentuses a policyPerson or robotEnvironmentCup insideObservationAction: place the cupReward+1: goal reached

Own agent-environment schematic after Stanford CS224R, Introduction (2025). Reward is an illustrative goal score.

3. Collecting experience

Lecture route: 3 of 5

1

Learn from
text

2

Output
actions

3

Collect
examples

4

Scale with
people

5

More kinds
of video

The robot needs observations paired with actions.

Who will make those examples?

3. Who creates a robot’s experience?

A person controls ALOHAZhao et al., ALOHA · Apr 2023 · teleoperated · 1x speed

ALOHA operator holding leader devices

ALOHA: A Low-cost Open-source Hardware System for Bimanual Teleoperation.
Original Zhao et al., ALOHA (2023). Teleoperated · Real-time excerpt.

Hand motion becomes robot motion

Leader and follower move togetherALOHA 2 Team · Feb 2024 · teleoperated · 1x speed
Leader motion is mapped into robot commands, rather than identical body jointsInstructor-authored schematic. 2. Reference mapping is in the accompanying Quarto source. Leader deviceCommandRobotSee the resultTeleoperation: a person guides the robot

Original ALOHA 2 (2024), teleoperated at 1x. Command/feedback diagram after ETH Real-World Robotics (2025).

Aligned records become training pairs

Camera and measured robot state, paired with human actionAt each timestamp, camera and current measured robot state form the observation. The human command at that same timestamp is stored as the action target; changed state is recorded at a later moment. Moment 1Moment 2Moment 3CameraMeasuredrobot stategripperopengripperclosedgrippermoved backHumanactionClose gripperPullMove to cupAt each timestamp: observation + demonstrated action
One aligned record becomes an observation-action training pairCurrent camera frame and measured robot state form the observation. The demonstrator command at that time is the supervised action target. The next robot state is not substituted as the current observation. At moment 2ObservationState:gripperclosedCamera frameAction to learnPullHuman commandTrain a policy: observation → demonstrated action

Adapted from ETH Real-World Robotics (2025), p6. © Robert Katzschmann.

Collection hardware has a cost

Teleoperation requires a robot and control equipment during collectionOwn hardware schematic. Recording a teleoperated demonstration occupies a robot and needs operator hardware and setup. Scene reset and recording checks add further work. No specific monetary price is claimed. During teleoperation data collectionPersonControl deviceRobotRobot hardware + teleoperation equipmentSet up, calibrate, reset, check recordings.
A handheld camera and gripper reduce collection hardware requirementsOriginal UMI hardware image retained unchanged inside a native viewport. The robot is absent during human capture, not unnecessary for deployment. Qualitative low-cost comparison follows the UMI authors; no price is invented. Universal Manipulation Interface (UMI)Handheld captureCamera + gripperNo robot during captureA lower-cost way to collect demonstrations

Collection workflow based on ETH Real-World Robotics (2025), p17, and Universal Manipulation Interface (UMI, 2024), section VII. © Robert Katzschmann.

UMI: Universal Manipulation Interface

The camera and square gripper markers on a human-held UMI toolFaithful viewport of the original UMI photograph. Camera and marker callouts are instructor annotations. Both jaws, the egg between them, the gripper gap, and the human hand remain visible; the photograph is not retouched.CameraMarkersBuilt-in IMUInertialMeasurementUnit

Original Chi et al., UMI (2024), Fig2 and project demo. Photo enlarged with teaching callouts. Human-held tool video · Source playback speed.

Learn from the recording

Human-held tool

Matched camera views from a human-held UMI tool and a robot gripper

Robot

Matched camera views from a human-held UMI tool and a robot gripper

Human-held gripper data trains a policy; the robot uses fresh observations at executionInstructor-authored schematic. 1. Reference mapping is in the accompanying Quarto source. Human-held toolView + movementTrain apolicy
Human-held gripper data trains a policy; the robot uses fresh observations at executionInstructor-authored schematic. 2. Reference mapping is in the accompanying Quarto source. After learningNew camera viewUse thepolicyRobotNext movement

Sunday: Skill Capture Glove → Memo

Views: UMI (2024), Fig2. Video: Sunday Robotics (2025), two official excerpts. Human capture, then Memo; robot segment originally 5x.

More people, more places

Could people record without the gripper?

Original UMI project montage (2024). Human collection in three settings · Source playback speed.

4. Scaling with human video

Lecture route: 4 of 5

1

Learn from
text

2

Output
actions

3

Collect
examples

4

Scale with
people

5

More kinds
of video

We can record demonstrations with robots and tools.

Could human activity give us much more data?

Egocentric video = first-person view

Lecture route: 4 of 5

1

Learn from
text

2

Output
actions

3

Collect
examples

4

Scale with
people

5

More kinds
of video

Human activity is already being recorded.

How much experience could human video provide?

Video by Sergi Coffee (YouTube). CC BY 4.0, creator-linked master; 8-second muted excerpt.

4. Learning from human activity

EgoVerse original collection setup figure

Original Punamiya et al., EgoVerse (2026), capture setup. Also discussed by Danfei Xu, Stanford ENGR319 (2026).

What a human record contains

Human video with estimated hand and camera motion plus task annotations

Video · Movement estimates · Task annotation

Original EgoVerse camera and hand tracks, focus view

Original EgoVerse task annotation, focus view

Original EgoVerse annotation demo (2026). Human demonstration · Source playback speed.

How much experience is 100,000 hours?

One hundred thousand hours equals about 11.4 years of continuous playbackInstructor-authored schematic. 1. Reference mapping is in the accompanying Quarto source. 100,000 hoursHow long is that?
100000hours is approximately11.4years of continuous playbackEleven full year blocks and0.415525of a year. Footage duration, not collection time.100,000 hours≈ 11.4 yearsEach full block = 1 yearPlayback: 24 hours/day · 365 days/year

Arithmetic: 100,000 ÷ 24 ÷ 365. Continuous playback, not collection time.

A public data program at this scale

100000hour public program and10000hour first announced batchOne of ten equal blocks is filled. Announcement dated2026-08-26; card access checked2026-09-28.EgoSuite-Open100K100,000 h full program10,000 h first announced batchAnnouncement: August 26, 2026Dataset access requires a request

EgoSuite-Open100K announcement, August 26, 2026. Human recordings at source playback speed. Staged access; current total not verified.

Generalist: a growing data foundation

Generalist data hours are attributed to GEN-1, with GEN-1.5 shown as a later developmentInstructor-authored schematic. 2. Reference mapping is in the accompanying Quarto source. GEN-1April 2, 2026500,000+ hdata reportedGEN-1.5August 19, 2026A later modelNew-task demosA data foundation, then continued development

Human demonstration

Robot execution

Original Generalist GEN-1.5 demos (2026). 500,000+h belongs to GEN-1, Apr 2. Source playback speed; robot autonomy claimed.

Dyna-2: a million hours of human video

Large human-video pretraining is connected to robot control through robot-data trainingInstructor-authored schematic. 2. Reference mapping is in the accompanying Quarto source. 1,000,000+ hoursDyna-2: mainly first-person human videoVideo learningRobot-datatrainingRobotReported in August 2026

Original Dyna-2 (2026), Figure 4: human pretraining footage, source playback speed. Robot-data training follows.

Many people collect at the same time

1000collectors for8productive hours each per day produce100000hours in12.5ideal daysIllustrative arithmetic, not the staffing or collection time of a named dataset. Excludes setup, quality checks, processing and losses.Hypothetical collection example1,000people×8 hper person / day=8,000 hper day100,000 h ÷ 8,000 h/day12.5 ideal days

Illustrative arithmetic only: 1,000 people × 8 h/day × 12.5 days = 100,000 h.

Which experiences fill those hours?

Repeatable tasks and broader activitiesSelected unmodified photo regions from the EgoVerse composition figure. The left shows two common tasks across two recordings. The right shows packing, cooking, cleaning and gardening examples from the source open-ended activities. This is a qualitative selection, not a quantitative comparison of dataset proportions.Repeatable tasksBroader activities
Two questions to ask after viewing actual dataset samplesPaired with the original EgoVerse coverage figure; equal emphasis on activity selection and camera viewpoint.Which experiences fill these hours?ActivitiesViewpointsWhat was filmed?Where was the camera?

Selected original examples from EgoVerse (2026), dataset composition. Teaching labels and questions; these excerpts do not show dataset proportions.

5. Learning from more kinds of video

Lecture route: 5 of 5

1

Learn from
text

2

Output
actions

3

Collect
examples

4

Scale with
people

5

More kinds
of video

We have learned from our own recorded actions.

Could we also learn by watching other people?

A view tied to the body

Ego-video supplies a viewpoint connected to the actorUnaltered frame from Sergi Coffee parkour master. Body-centered visual cues are useful but do not equal proprioceptive sensory measurements.Your viewpointYour moving bodyYour interaction
Proprioception: body position sense without lookingOwn illustrative nose-touch gesture for a classroom exercise, not a clinical assessment. Spoken explanation distinguishes it from egocentric RGB video.ProprioceptionWhere is my body?Eyes closed.Touch your nose.A body sense, beyond what the camera sees.

Frame: Sergi Coffee, CC BY 4.0. Body-position sense: Proske & Gandevia (2012).

First-person video is one slice

First-person collection selects only part of human visual experienceSchematic selection biases; no claim that all collections are laboratory-only or any quantitative proportion of online video.Purpose-built captureViewpointTaskSetupMany hours can repeat a chosen setup.Daily life is broader than the collection plan.
First-person collection selects only part of human visual experienceSchematic selection biases; no claim that all collections are laboratory-only or any quantitative proportion of online video.A wider range of videoSportCookingEveryday lifeLearning beyond the collection setupExocentric video = a view from outside the actor

Own selection schematic. Diverse collection: Ego4D; broader video pretraining: V-JEPA 2 (2025). No quantitative share of web video is implied.

We also learn by watching others

Learning by watching another actorUnaltered AFC photograph of Australia v Korea Republic, February 2024. It is an action photograph, not a reconstructed free-kick trajectory. The spoken free-kick and golf examples describe observational learning.Watch an actionFrom outsidethe actor’s body
Learning by watching another actorUnaltered AFC photograph of Australia v Korea Republic, February 2024. It is an action photograph, not a reconstructed free-kick trajectory. The spoken free-kick and golf examples describe observational learning.WatchPredictTryObservation + practice

Original photograph: AFC, Australia v Korea Republic (2024). Observation and practice: D’Innocenzo et al. (2016).

From an action to a prediction

A kick provides clues about the following ball trajectoryOwn schematic of a curved ball trajectory, not measured footage or a deterministic physics calculation. The point is prediction from observed action and outcome.The kickThe ball’s pathWhere next?
A kick provides clues about the following ball trajectoryOwn schematic of a curved ball trajectory, not measured footage or a deterministic physics calculation. The point is prediction from observed action and outcome.The kickThe ball’s pathWatch the action. Learn to anticipate the change.

Own illustrative trajectory; not a measured match replay. Observational learning: D’Innocenzo et al. (2016).

A model of what happens next

Predict what an action will changeOwn large-type reconstruction after Hyeongmin Lee, DGIST 2026 slide31, using the established drawer example. Predicted scene is illustrative, not empirical model output.NowWorld modelPredicted nextPredict apossible future+ action: kickBall curvesWhat I see + what I do → what happens next.
Predict what an action will changeOwn large-type reconstruction after Hyeongmin Lee, DGIST 2026 slide31, using the established drawer example. Predicted scene is illustrative, not empirical model output.NowWorld modelPredicted nextPredict apossible future+ action: pullDrawer opensCompare possible actions before acting.

Reconstructed after Hyeongmin Lee, DGIST (2026), slide 31. Original diagram.

Two ways to predict a world

Compare visible-frame and representation predictionIllustrative teaching comparison of output targets. Generative models may also use latent computations internally. Feature glyphs are not hand-designed semantic state labels.What should the model predict?Visible framesLearned featuresImages we can watchValues the model can useTwo useful approaches to learning a predictive model
Compare visible-frame and representation predictionIllustrative teaching comparison of output targets. Generative models may also use latent computations internally. Feature glyphs are not hand-designed semantic state labels.What should the model predict?Visible framesLearned featuresDreamDojo · Genie 3V-JEPA · LeWorldModelV-JEPA: Video Joint-Embedding Predictive Architecture

Examples: DreamDojo, Genie 3, V-JEPA, LeWorldModel. The comparison concerns prediction targets.

Predict images you can watch

A generative world model predicts visible future framesOwn illustrative filmstrip. Frame-by-frame depiction is an introductory abstraction; real systems can generate chunks and work in compressed latent space.Current sceneNext frameThen…?PullPlace cupLearn from video by predicting its next images.
A generative world model predicts visible future framesOwn illustrative filmstrip. Frame-by-frame depiction is an introductory abstraction; real systems can generate chunks and work in compressed latent space.Current sceneNext frameThen…PullPlace cupContinue from the frames already generated.

Own teaching filmstrip after DreamDojo (2026) and Genie 3 (2025).

Generated futures can respond to actions

DreamDojo: generated robot futures

Genie 3: a generated world responds to keys

Official generated-video examples: NVIDIA GEAR, DreamDojo (2026) · Google DeepMind, Genie 3 (2025). Source playback speed.

JEPA predicts representations

Predict features of hidden video information Original masked dog-video frames from the V-JEPA diagram. Native arrows and illustrative purple blocks explain learned feature prediction without incidental architecture labels. Visible videoPredictPredicted featuresPredict information hidden from view
Compare predicted features with a full-video reference The original masked and full dog-video frame groups are shown intact in separate viewports. Native arrows and illustrative feature blocks show the prediction/reference comparison; they are not model measurements. Visible videoPredictPredicted featuresFull videoReference featuresCompare

JEPA: Joint-Embedding Predictive Architecture. Original video panels: CMU (2025), p35, after Meta V-JEPA. Feature bars are illustrative.

Use the prediction to choose an action

V-JEPA 2: planning, then real robot execution

LeWorldModel: planning in feature space

Simulated rolloutGoal image

Meta, V-JEPA 2 (2025): real robot. Maes et al., LeWorldModel (2026): simulated Push-T; original animation shown four times.

More videos, more experience to learn from

From broader video to predictive knowledge and grounded actionOwn synthesis: video offers predictive learning opportunities; reliable physical prediction and embodiment grounding remain necessary.More of the world’s videoOther peopleOther actionsOther settingsLearn what may happen. Connect it to a robot.

Lecture synthesis. V-JEPA 2 · DreamDojo · LeWorldModel.

Three ideas to keep

Record action, scale human experience, and learn from broader videoInstructor-authored schematic. 1. Reference mapping is in the accompanying Quarto source. 1Record actionWhat happened?
Record action, scale human experience, and learn from broader videoInstructor-authored schematic. 2. Reference mapping is in the accompanying Quarto source. 1Record actionWhat happened?2Scale experienceWho can contribute?
Record action, scale human experience, and learn from broader videoInstructor-authored schematic. 3. Reference mapping is in the accompanying Quarto source. 1Record actionWhat happened?2Scale experienceWho can contribute?3Learn from videoWhat can we infer?

Lecture synthesis. Collection: ETH Real-World Robotics (2025); learning loop: Stanford CS231n (2025), p69.

What would you teach a robot?

Choose an everyday behavior and the experience needed to teach itInstructor-authored schematic. complete. Reference mapping is in the accompanying Quarto source. One everyday taskWhat would you show?What would you record?Think for 20 seconds.

How does experience become a skill?

The next lecture focuses on methods that turn experience into skillInstructor-authored schematic. complete. Reference mapping is in the accompanying Quarto source. ExperienceLearningActionNext lecture: October 23How does experience become a skill?

Next session from the course series; learning-loop framing: Stanford CS231n (2025), p69.

Your questions

QR code for the question wall
Wooclap code ZNQKOGFapp.wooclap.com/ZNQKOGF

Sources