DreamDojo: Foundation World Model for Dexterous Robotics

Gao, Ye, Yang, Zheng, Jang et al. (NVIDIA, 2026)

Yunsung Lee

WoRV / MaumAI

Roadmap

1. Motivation — why robot data is not enough
2. Latent Action Model — self-supervised actions
3. Architecture & Training — building the world model
4. Distillation — making it real-time
5. Results — benchmarks and applications
6. Code Insights — under the hood
7. Discussion — takeaways and landscape

DreamZero Recap: WAMs for Zero-shot Policies

World Action Models (WAMs):

• Joint video + action generation from a single DiT backbone
• Autoregressive chunk-wise inference with ground-truth injection
• 2x over VLAs, 38x speedup to real-time 7Hz

Key Paradigm

WAMs generate “how to move” by predicting “what happens next” — inverse dynamics emerge from video understanding.

From DreamZero to DreamDojo

DreamZero (WAM = Policy):

• Joint video + action prediction
• Inverse dynamics: video \rightarrow action
• Zero-shot policies
• Trained on robot data

DreamDojo (World Model):

• Action-conditioned video prediction
• Forward dynamics: action \rightarrow video
• Open-world simulation
• Pretrained on human video

The Shift

Same team, complementary paradigms: DreamZero is a policy, DreamDojo is a simulator.

The Data Gap: Robot Data Is Not Enough

The fundamental bottleneck:

• Existing robot datasets: limited coverage, narrow distributions
• DROID: 350 hours, 86 skills, 564 scenes
• AgiBot World: 2.9K hours, 87 skills, 106 scenes
• World models trained on robot data \rightarrow in-distribution only

The Core Problem

Robot world models plateau at observed setups and are unresponsive to counterfactual actions — actions the robot never performed in training (e.g., patting a toy, reaching and missing). Scaling teleoperation alone cannot cover the long tail.

The Human Video Insight

Despite the embodiment gap, the underlying physics during interactions is largely consistent between humans and robots, enabling effective knowledge transfer.

— Gao et al. (2026)

Key idea: Human videos capture the same physics — contact, gravity, deformation, tool use — just with different end-effectors.

DreamDojo-HV: The Largest WM Dataset

Dataset Scale

• 44,711 hours of egocentric video
• 1,179K trajectories
• ~6,015 unique skills
• >1,135K unique scenes

vs. prior largest (AgiBot World): 15x duration, 96x skills, 2,000x scenes

Three Data Sources

Data Mixture

In-lab (55h)

Manus gloves, 35 skills, 1 scene

EgoDex (829h)

Apple Vision Pro, 194 skills, 5 scenes

HV (43.8Kh)

Crowdsourced, 6,015 skills, 1.1M scenes

Sampling ratio: In-lab : EgoDex : HV = 1 : 2 : 10

EgoDex sample

In-lab PSNR curves

System Overview: Three-Phase Pipeline

Phase 1: Pretrain on human videos with latent actions

Phase 2: Post-train on target robot data (reset action layer)

Phase 3: Distill to autoregressive real-time student

How Does DreamDojo Learn Actions?

Latent Action Model

The Action Label Problem

Human videos are vast but lack action labels.

• Passive video prediction ignores causality between observations and actions
• Off-the-shelf hand pose extractors (HaMeR) miss arm movements, locomotion, and fail under occlusion
• Converting heterogeneous action formats into a unified representation \rightarrow significant engineering burden

The Dilemma

Actionless pretraining wastes interaction knowledge. Ground-truth actions require expensive capture devices. We need a self-supervised middle ground.

Latent Action Model (LAM)

VAE with information bottleneck:

\mathcal{L}^{pred}_{\theta,\phi}(f^{t+1}) = \mathbb{E}_{q_{\phi}(\hat{a}|f^{t:t+1})} \log p_{\theta}(f^{t+1}|\hat{a},f^{t}) - \beta\,D_{KL}(q_{\phi}(\hat{a}|f^{t:t+1})||p(\hat{a}))

\hat{a} \in \mathbb{R}^{32}, \beta=10^{-6} (tight bottleneck \rightarrow forces action disentanglement)

Latent Actions Cross Embodiments

Cross-Embodiment Transfer

The same latent action \hat{a} captures the same physical interaction regardless of whether it is performed by a human hand, a gripper, or a dexterous robot.

LAM Architecture (700M params):

• Spatiotemporal Transformer (Bruce et al. 2024)
• 24 encoder blocks + 24 decoder blocks
• Encoder: (f^t, f^{t+1}) \rightarrow \hat{a} — Decoder: (\hat{a}, f^t) \rightarrow \hat{f}^{t+1}
• Trained on mixture of all datasets (human + robot)

Latent Actions vs. Alternatives

Method	PSNR	LPIPS	Note
w/o pretrain	20.576	0.222	No human data
Action-free	20.797	0.222	Passive video
Latent action	20.913	0.219	Self-supervised
Retargeted GT	20.960	0.219	Requires gloves

Key Finding

Latent actions achieve near-parity with ground-truth action labels — without any capture devices. The gap: <0.05 PSNR, 0.000 LPIPS.

My Take: This is the enabler for 44K-hour scale. You cannot put Manus gloves on a million crowdworkers, but you can run a VAE on their videos.

Building the World Model

Architecture & Training

Cosmos-Predict2.5 Backbone

Foundation

• Latent video diffusion model (Ali et al. 2025)
• WAN 2.2 tokenizer: 4\times temporal compression (video latent x^i corresponds to 4 pixel frames f^{i:i+4})
• DiT blocks (Peebles and Xie 2023) with cross-attention for text, AdaLN for timestep
• Trained with flow matching loss
• Two variants: 2B and 14B parameters

Training setup:

• 140K steps pretrain (256 H100s), 50K steps post-train (128 H100s)
• Resolution: 640 \times 480, sequences of 13 latent frames
• EMA maintained throughout

Flow Matching in 60 Seconds

Intuition

Learn a velocity field that transports noise \rightarrow data along straight paths. Simpler and faster than standard diffusion.

Flow matching loss:

\mathcal{L}_{\text{flow}}(\theta) = \mathbb{E}_{\mathbf{x},\epsilon,\mathbf{c},t} \left\|\mathbf{u}(\mathbf{x}_{t},t,\mathbf{c};\theta) - \mathbf{v}_{t}\right\|^2

where \mathbf{v}_t = \epsilon - \mathbf{x} (velocity = noise - clean data)

• t=0: pure noise. t=1: clean data.
• Model predicts the velocity pointing from noise toward data
• Conditions \mathbf{c}: text, frames, and actions (for world models)

Two Key Innovations for Controllability

Design Choices

1. Relative actions: Rebaseline every 4 timesteps — concentrates action space \rightarrow better generalization to compositions
2. Chunked injection: 4 consecutive actions a^{t:t+4} sent to matching latent frame — respects causality \rightarrow future actions do not leak

Action Embedding: AdaLN Injection

Action injection pathway:

\text{actions} \xrightarrow{\text{MLP}} \text{action embedding} \xrightarrow{+} \text{timestep embedding} \xrightarrow{\text{AdaLN}} \text{DiT blocks}

Scale, shift, and gate modulations in every DiT block

Key implementation details:

• Last layer of action MLP zero-initialized (Zhang et al. 2023) — avoids perturbing pretrained state at start of training
• During post-training: reset first layer only, finetune all weights
• Classifier-free guidance disabled (empirically no benefit)

Action Embedding: Code Insights

Implementation Pattern

• Latent actions projected via lightweight MLP to match timestep embedding dim
• Zero-init on last MLP layer \rightarrow identity-like at init
• Added to timestep embeddings before AdaLN processing
• Dual pathway: actions modulate through both timestep AND scale/shift/gate

My Take: The dual-path action injection is more sophisticated than the paper describes — actions modulate DiT through both timestep AND AdaLN scale/shift/gate pathways. The zero-init trick from ControlNet is crucial for preserving pretrained physics.

Temporal Consistency Loss

Problem: Flow matching supervises frames individually, ignoring temporal correlations.

Temporal consistency loss:

\mathcal{L}_{\text{temporal}}(\theta) = \mathbb{E}\Big[\sum^{K-1}_{i=1} \left\|(z^{i+1}-z^{i})-(v^{i+1}-v^{i})\right\|^2\Big]

Final objective:

\mathcal{L}_{\text{final}} = \mathcal{L}_{\text{flow}} + \lambda\,\mathcal{L}_{\text{temporal}}, \quad \lambda = 0.1

Post-Training: Adapting to Target Robots

Post-Training Recipe

1. Reset first layer of action MLP (new action space)
2. Finetune all weights, target data at ~10 Hz
3. 128 H100s, 50K steps, batch 512

Embodiments: GR-1, G1, AgiBot, YAM — each post-trained separately. CFG disabled (no benefit).

Making It Real-Time

Distillation Pipeline

The Distillation Challenge

Why distill?

• Teacher: 35 denoising steps, bidirectional attention, 2.72 FPS
• Too slow for live teleoperation and online planning

Two Conversions Required

1. Bidirectional \rightarrow Autoregressive (causal attention, streaming)
2. 35 steps \rightarrow 4 steps (few-step generation)

Student initialized from teacher weights, with bidirectional attention replaced by causal attention over a sliding window of 12 frames.

Self Forcing: Warmup Stage

Stage 1: Warmup — student regresses to teacher ODE solutions

\mathcal{L}_{\text{warmup}} = \mathbb{E}_{x,t}\|G_{\text{student}}(x_{t},t)-x_{0}\|^2

Student generates via teacher forcing (clean context)

How it works:

• Student receives clean context frames (teacher forcing)
• Learns to map noisy inputs to clean data in fewer steps
• Provides a stable initialization before self-generated training

Self Forcing: Distillation Stage

Stage 2: Distillation — student trains on its own outputs (Huang et al. 2025)

\nabla\mathcal{L}_{\text{distill}} = -\mathbb{E}_{z,t}\Big[ (s_{\text{real}}(x_{t},t) - s_{\text{fake}}(x_{t},t)) \frac{dG_{\text{student}}}{d\theta}\Big]

KL divergence between teacher and student distributions (Yin et al. 2024)

s_{\text{real}}: score from real (teacher-generated) data | s_{\text{fake}}: score from fake (student-generated) data

Extended rollout: Student generates N'>N frames, loss on last N — reduces long-horizon drift.

Distillation Results

Speed vs. Quality

	PSNR	SSIM	LPIPS	FPS	Pred	Ctx
Teacher	14.09	0.442	0.412	2.72	12	1
Student	13.15	0.379	0.485	~10	4	12

1-minute rollouts

Context advantage

Does It Actually Work?

Experimental Results

Benchmark Construction: 6 OOD Eval Sets

6 Evaluation Sets on GR-1

1. In-lab Eval 2. EgoDex Eval 3. HV Eval 4. Counterfactual Eval

5. EgoDex-novel (Gemini-edited BG) 6. HV-novel (Gemini-edited BG)

Metrics: PSNR, SSIM, LPIPS (auto); human preference for novel scenes.

Data Scaling: More Diversity = Better OOD

Data Mixture	In-lab	EgoDex	HV
No pretrain (Cosmos)	20.58	19.95	18.27
In-lab only	20.91	20.27	18.62
+ EgoDex	20.97	20.33	18.71
+ DreamDojo-HV	21.02	20.41	18.72
DreamDojo-2B	21.11	20.41	18.81
DreamDojo-14B	21.41	20.53	18.92

Monotonic Improvement

Adding more human data consistently improves all OOD benchmarks — both physics modeling and counterfactual action following.

Human Preference: 73.5% Win Rate

Human Evaluation (12 volunteers)

Comparison	Physics	Action
DreamDojo-2B > Cosmos	62.5%	63.5%
DreamDojo-14B > Cosmos	73.5%	72.6%
DreamDojo-14B > DreamDojo-2B	72.5%	65.5%

Two axes evaluated:

• Physics correctness: object permanence, shape consistency, contact causality
• Action following: robot pose accuracy vs. ground-truth

Ablation: What Matters Most?

Counterfactual Eval metrics:

Rel.	Chunk	Temp.	PSNR	SSIM	LPIPS
			19.45	0.768	0.211
✓			19.48	0.772	0.212
✓	✓		20.78	0.790	0.193
✓	✓	✓	20.98	0.796	0.189

Takeaway

Chunked injection is the largest single gain (+1.30 PSNR). Relative actions alone have minimal effect, but temporal loss adds a further +0.20 improvement.

Multi-Embodiment Support

Four Robot Platforms

Post-trained on GR-1, G1, AgiBot, and YAM — each with separate post-training runs using the same pretrained foundation.

Per-embodiment adaptation:

• Reset action MLP first layer for new action space
• Finetune all weights (50K steps)
• Same pretrained physics transfers across all platforms

Policy Evaluation: A Reliable Simulator

Correlation with Reality

• Pearson r = 0.995
• MMRV = 0.003

Protocol:

• AgiBot fruit packing (20 scenes)
• GR00T N1.5 policy checkpoints
• ~80s rollouts in real world and DreamDojo
• Success = fruits packed / 5

DreamDojo can reliably rank policies without deploying on a real robot.

MPC & Live Teleoperation

Model-Based Planning:

• 5 policy checkpoints \rightarrow action proposals
• DreamDojo predicts futures, value model selects
• +17% success rate over best checkpoint

Live Teleoperation:

• PICO VR controller \rightarrow G1 actions
• Distilled model on RTX 5090
• Real-time interactive simulation

World models as simulators unlock planning, evaluation, and teleoperation — all from a single foundation.

Under the Hood

Code-Level Insights

Why Look at the Code?

Production-Grade NVIDIA Code

DreamDojo’s codebase reveals engineering decisions the paper glosses over — these details matter for reproducibility and understanding.

What we can learn:

• How the LAM bottleneck actually works
• How actions modulate the DiT
• How three different data loaders handle heterogeneous sources
• How relative actions use proper SE(3) math

Code: Latent Action Model

Implementation Details (Not in Paper)

• Class LatentActionModel wraps asymmetric encoder–decoder
• Encoder: SpatioTemporalTransformer (joint space-time attention across frames)
• Decoder: SpatioTransformer (spatial-only — no temporal mixing)
• Action prompt initialized as: nn.Parameter(torch.empty(1, 1, 1, patch_token_dim))

Key asymmetry: The encoder sees both frames jointly (spatiotemporal), while the decoder reconstructs from action + single frame (spatial only). This forces all temporal information through the 32-dim bottleneck.

Code: DiT Action Injection

Concrete Code Patterns

• Temporal rearrange: rearrange(action, "b 1 (t d) -> b t d", t=...)
• Two separate MLPs for action embedding: action_embedder_B_D (global timestep path) vs action_embedder_B_3D (per-frame spatial path)
• Each MLP last layer zero-initialized
• Embeddings added to timestep before AdaLN

My Take: Two separate action MLPs — not one — feed into different pathways. This dual-path design is more sophisticated than the paper describes and explains why chunked injection has such a large ablation effect.

Code: Data Pipeline

Three Distinct Loaders

• MANO loader: For EgoDex/In-lab with hand pose annotations
• VideoOnly loader: For DreamDojo-HV (latent actions extracted online)
• LeRobot loader: For target robot data with ground-truth joint actions

Dual-resolution processing: LAM operates at 320x240 (efficiency), while the world model trains at 640x480 (quality). Separate tokenizer passes for each resolution.

Code: Relative Action Computation

SE(3) Math for Relative Actions

• Translation: body-frame deltas, not naive world-frame subtraction
• Rotation: proper rotation matrix computation for relative orientation
• Scale factor: $$20 applied to normalize action magnitudes
• Rebaseline every 4 timesteps (matching tokenizer compression)

My Take: The relative action computation uses proper SE(3) math — translation deltas in body frame, not naive subtraction. The 20x scaling factor is a practical detail the paper omits entirely. These “boring” engineering choices are what make the model actually work.

What Does It All Mean?

Discussion & Takeaways

Limitations & Honest Assessment

Paper-Acknowledged Limitations

• Uncommon actions (slapping, fast waving) still fail
• Policy eval overestimates success rates vs. real world
• No multi-view simulation support
• Post-training knowledge retention not studied

My Take: The proprietary 43.8K-hour dataset is both the biggest strength and weakness — it makes the results hard to reproduce. The eval overestimation suggests the model struggles with nuanced failure modes.

DreamDojo vs. DreamZero: Two Paradigms

DreamZero (WAM = Policy):

• Inverse dynamics
• Video + action jointly generated
• Zero-shot policy transfer
• Trained on robot data only
• 500 hours robot teleoperation

DreamDojo (World Model):

• Forward dynamics
• Action \rightarrow video prediction
• Simulation, planning, teleoperation
• Pretrained on 44K hours human video
• Adapted to robots via post-training

Complementary, Not Competing

DreamZero asks “what action produces this future?”

DreamDojo asks “what future does this action produce?”

DreamDojo vs. DreamZero: Technical Comparison

	DreamZero	DreamDojo
Dynamics	Inverse (IDM)	Forward (WM)
Backbone	WAN 2.1	Cosmos-Predict2.5
Tokenizer	WAN 2.1 (4\times)	WAN 2.2 (4\times)
Conditioning	Joint video+action	Action \rightarrow video
Data	Robot only	Human + robot
Scale	500h robot	44.7Kh total
Latent actions	No	Yes (32-dim VAE)
Distillation	Flash (1-step)	Self Forcing (4-step)
Speed	7Hz (2xGB200)	~10 FPS (1xH100)

My Take: The Cosmos-Predict2.5 backbone gives DreamDojo stronger video priors, while the latent action framework enables human video pretraining at scale. Different tools for different jobs.

The World Model Landscape

DreamDojo is part of a rapid convergence in world models for robotics:

• AdaWorld (Gao et al. 2025): latent actions for adaptable WMs (tabletop)
• Genie (Bruce et al. 2024): interactive environments from video (discrete control)
• DexWM (Goswami et al. 2025): human videos for dexterous manipulation (smaller scale)
• DreamDojo: first foundation WM at 44K hours with continuous latent actions

The trend: Scale up human video, bridge the embodiment gap with latent actions, distill for real-time interaction.

Key Takeaways

1. Human video pretraining works — 44K hours of ego video \rightarrow physics that generalizes
2. Latent actions bridge embodiments — self-supervised, near GT quality, infinitely scalable
3. Chunked + relative actions are the biggest controllability gains
4. Self Forcing distillation enables real-time interaction (~10 FPS)
5. Policy evaluation correlates with reality (r=0.995)
6. World models are becoming general-purpose simulators for planning, eval, and teleoperation

The future of robot world models is human-video-native.

Thank You & References

Paper & Code:

• Gao et al. (2026)
• arXiv: 2602.06949
• Code: github.com/NVIDIA/DreamDojo

DreamZero (predecessor):

• Ye et al. (2026)
• dreamzero0.github.io

Key References:

• Cosmos-Predict2.5 (Ali et al. 2025)
• Self Forcing (Huang et al. 2025)
• AdaWorld (Gao et al. 2025)
• Genie (Bruce et al. 2024)
• DiT (Peebles and Xie 2023)
• Flow matching (Lipman et al. 2022)

Q&A

References

Ali, Arslan, Junjie Bai, Maciej Bala, Yogesh Balaji, et al. 2025. “World Simulation with Video Foundation Models for Physical AI.” arXiv Preprint arXiv:2511.00062.

Bruce, Jake, Michael Dennis, Ashley Edwards, et al. 2024. “Genie: Generative Interactive Environments.” ICML.

Gao, Shenyuan, Seonghyeon Ye, Jiazhi Yang, Kaiyuan Zheng, Joel Jang, et al. 2026. “DreamDojo: A Foundation World Model for Open-World Dexterous Robot Tasks.” arXiv Preprint arXiv:2602.06949.

Gao, Shenyuan, Siyuan Zhou, Yilun Du, Jun Zhang, and Chuang Gan. 2025. “AdaWorld: Learning Adaptable World Models with Latent Actions.” ICML.

Goswami, Raktim Gautam, Amir Bar, David Fan, et al. 2025. “World Models Can Leverage Human Videos for Dexterous Manipulation.” arXiv Preprint arXiv:2512.13644.

Huang, Xun, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. 2025. “Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion.” NeurIPS.

Lipman, Yaron, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. 2022. “Flow Matching for Generative Modeling.” arXiv Preprint arXiv:2210.02747.

Peebles, William, and Saining Xie. 2023. “Scalable Diffusion Models with Transformers.” ICCV.

Ye, Seonghyeon, Yunhao Ge, Kaiyuan Zheng, et al. 2026. “World Action Models Are Zero-Shot Policies.” arXiv Preprint.

Yin, Tianwei, Michaël Gharbi, Richard Zhang, et al. 2024. “One-Step Diffusion with Distribution Matching Distillation.” CVPR.

Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. 2023. “Adding Conditional Control to Text-to-Image Diffusion Models.” ICCV.