Edouard Silvestre
Back to projects

Epitech Hub · Solo

WorldVideo

A ConvLSTM that watches 10 frames of a video and predicts the next 10.

  • Python
  • PyTorch
  • ConvLSTM
Try it
Period
Jun – Jul 2026
Team
Solo project
Code
View the code
73t173t273t373t4ConvLSTMmemory73t573t673t710 observed frames10 predicted framesMoving MNIST · 64 × 64fig. 04 — WorldVideo · predicting what comes next

The project

WorldVideo predicts what happens next in a video. From the first 10 frames of a sequence, the model generates the next 10.

The model is a ConvLSTM with 4 layers of 128 channels, trained on Moving MNIST: 10,000 sequences of 20 frames at 64 × 64. I evaluate it with MSE and SSIM.

What was hard

  • Training a ConvLSTM with 4 layers of 128 channels needs a GPU: training ran on another machine, with no network between the two.
  • The further the prediction gets from the 10 input frames, the more it blurs, especially when two digits cross.

What I learned

  • Starting from a reference architecture, OpenSTL’s, adapting it and crediting it, rather than rewriting everything.
  • Evaluating image prediction with two complementary metrics: MSE, pixel by pixel, and SSIM, which looks at structure.
  • Tracking a training run from end to end with Weights & Biases.

Try it.

Real against predicted

On the left the real sequence, on the right what the model generates. Pause and step frame by frame: the prediction blurs as it moves away from the starting frames.

Input, real, predicted

For one sequence: the 10 input frames, the 10 real frames that follow, then the 10 predicted frames.

  • Sequence 1: three rows of frames, input, real and predicted
  • Sequence 2: three rows of frames, input, real and predicted
  • Sequence 3: three rows of frames, input, real and predicted
  • Sequence 4: three rows of frames, input, real and predicted
  • Sequence 5: three rows of frames, input, real and predicted

Credits

  • ConvLSTM architecture adapted from OpenSTL (Apache 2.0 licence).
  • Data: Moving MNIST, Srivastava et al., 2015.