PoC Innovation · Team R&D
LeWM-Robot
A world model that teaches a robot arm to pick up an object and drop it off.
- Python
- PyTorch
- LeRobot
- World models
- Period
- Apr – Oct 2026
- Team
- Team project, 4 contributors
The project
We train a LeRobot SO-101 arm on a pick-and-place task, with one goal: generalising to situations never seen during training.
The architecture is a world model inspired by DINO-WM: a frozen DINOv3 visual encoder, a trainable predictor, LoRA for sim-to-real transfer, and CEM planning.
We built and published an augmented dataset of 360 episodes, from 40 recorded episodes, with our own pipeline.
In numbers
- episodes in the published dataset
- 360
- augmentation techniques
- 8
- cameras: wrist and front
- 2
What was hard
- Fusing two cameras. We implemented and compared five fusion strategies: cross-attention wins, with a validation error of 8.45 against 11 to 12 for the others.
- Working without a powerful enough GPU on hand: the pipeline was adapted to an RTX 4090, with a script that first estimates the time of each step.
- Generalising from only 40 demonstrations.
What I learned
- Building and publishing a clean LeRobot dataset on Hugging Face, and multiplying it by 9 with augmentations.
- Keeping a large encoder frozen and training only what needs it: the predictor and LoRA.
- Validating each building block on its own, from encoding to planning, before training the whole.
Try it.
The arm, seen by its two cameras
Episodes from the published dataset: the front camera on the left, the wrist camera on the right.
Dataset episodes
Recorded episode
The arm picks up the duck and drops it in the box.
The same episode, augmented
Brightness, contrast and saturation shift: one of the 8 augmentation techniques.
Another episode
Another of the 40 recorded episodes.
The architecture
An action-conditioned world model, inspired by DINO-WM.
Step 12 caméras
Observation
Frames from the front camera and the wrist camera.
Step 2gelé
DINOv3 encoder
A pre-trained visual encoder whose weights stay fixed, then the fusion of both views.
Step 3entraînable
Predictor
A transformer predicts the next representation from the state and the action. LoRA adapts from simulation to the real arm.
Step 4CEM
Planning
The cross-entropy method searches for the action sequence whose predicted outcome gets closest to the goal.
Step 56 moteurs
Action
The command is sent to the arm’s motors.