Edouard Silvestre
Back to projects

PoC Innovation · Team R&D

LeWM-Robot

A world model that teaches a robot arm to pick up an object and drop it off.

  • Python
  • PyTorch
  • LeRobot
  • World models
Try it
Period
Apr – Oct 2026
Team
Team project, 4 contributors
Code
View the codeDataset on Hugging Face
cam 1SO-101DINOv3 + LoRA · CEMpredictionfig. 03 — LeWM-Robot · a world model drives the arm

The project

We train a LeRobot SO-101 arm on a pick-and-place task, with one goal: generalising to situations never seen during training.

The architecture is a world model inspired by DINO-WM: a frozen DINOv3 visual encoder, a trainable predictor, LoRA for sim-to-real transfer, and CEM planning.

We built and published an augmented dataset of 360 episodes, from 40 recorded episodes, with our own pipeline.

In numbers

episodes in the published dataset
360
augmentation techniques
8
cameras: wrist and front
2

What was hard

  • Fusing two cameras. We implemented and compared five fusion strategies: cross-attention wins, with a validation error of 8.45 against 11 to 12 for the others.
  • Working without a powerful enough GPU on hand: the pipeline was adapted to an RTX 4090, with a script that first estimates the time of each step.
  • Generalising from only 40 demonstrations.

What I learned

  • Building and publishing a clean LeRobot dataset on Hugging Face, and multiplying it by 9 with augmentations.
  • Keeping a large encoder frozen and training only what needs it: the predictor and LoRA.
  • Validating each building block on its own, from encoding to planning, before training the whole.

Try it.

The arm, seen by its two cameras

Episodes from the published dataset: the front camera on the left, the wrist camera on the right.

Dataset episodes

  • Recorded episode

    The arm picks up the duck and drops it in the box.

  • The same episode, augmented

    Brightness, contrast and saturation shift: one of the 8 augmentation techniques.

  • Another episode

    Another of the 40 recorded episodes.

The architecture

An action-conditioned world model, inspired by DINO-WM.

  1. Step 12 caméras

    Observation

    Frames from the front camera and the wrist camera.

  2. Step 2gelé

    DINOv3 encoder

    A pre-trained visual encoder whose weights stay fixed, then the fusion of both views.

  3. Step 3entraînable

    Predictor

    A transformer predicts the next representation from the state and the action. LoRA adapts from simulation to the real arm.

  4. Step 4CEM

    Planning

    The cross-entropy method searches for the action sequence whose predicted outcome gets closest to the goal.

  5. Step 56 moteurs

    Action

    The command is sent to the arm’s motors.