Interactive Generative World Model

How it started

In August 2024, I came across GameNGen and the paper Diffusion Models Are Real-Time Game Engines. The demo showed DOOM being generated interactively by a neural network instead of rendered by a normal game engine. The demo ran on a TPU-v5 and used Stable Diffusion as the base model. I wanted to see if I could make a world model from scratch that could train and run at a comparable 20 fps on my RTX 4090.

I started working on my own version in my free time. The paper described their overall training method, but there was no official implementation to build from. Rather than reproducing their Stable Diffusion architecture, I wrote my own training and inference pipeline and experimented with a smaller action-conditioned model that was practical on my hardware.

October 2024: First working DOOM version

By October 28, about two months after the GameNGen paper was released, I had a first version running with DOOM. It was a custom U-Net I built and trained from scratch. The generation was still rough and lacked consistency and fine detail, but I could control the game and move through an environment being generated by the model in real time.

This result proved to me that I could actually create a working world model, and it wasn't some impossible task. Now that I had this baseline, I could start fousing on which parts of the architecture, training, and data were limiting the model, and iteratively improve it.

What happened next

The early version used a U-Net, but as I kept experimenting I ran into quality As ran training tests, I started hitting barriers that didn't have an obvious solution. I started building small synthetic environments so I could isolate parts of the system and test them directly.

Those tests exposed problems with how actions were represented and how previous frames were being used as context. I also experimented with different attention methods that improved the quality, but cost far more memory than the improvement justified.

I eventually ran benchmarks between the U-Net and a DiT-based architecture. The DiT used much less VRAM and produced better samples than the U-Net, but took more training steps to converge. I later adapted a sparse MoE DiT backbone for the world model and kept changing the conditioning and training setup as I found new failure points.

Throughout the project I used new research papers as sources for ideas. I would impliment the part of the paper I was interested in, ensure I could reproduce the behaviour seen in the papers, and then test it inside my world model. Some methods transferred well, others required modifications, and some just didn't meaningfully help.

May 2025: Much better quality

By May 27, 2025, I had a much better model trained on a small custom Unreal Engine environment to stress test fine details. This new model used actual mouse offsets, allowing far more dynamic camera movement compared to the old model which used arrow keys to rotate at a constant speed. The quality had increased while maintaining the ability to run at 20 FPS on my RTX 4090.

The model used its own generated frames as context for the next prediction, so the environment continued to respond as I moved through it. It was still an experimental model trained on a small environment, but it was a large improvement over the first DOOM version and a much better demonstration of the direction I wanted to pursue.

What happened next

As my college workload increased, I had to put this project aside. More recently, I have returned to world model research and have been analyzing the SOTA models. One of the most interesting challenges to me is persistent memory.

A world model can generate a convincing environment and accurately maintain something in it's recent context, but one you leave an area and return, recalling those details becomes exceptionally hard. I've been researching ways to give models a persistant spatial representation of what it has already observed, so previously seen parts of a world don't have to regenerate from scratch when they leave the context window.

My current experiments focus on long-term spatial consisency while remaining efficient. This work is ongoing and unpublished, so I'm keeping the implimentation details private for now.

What I learned

This project is what really got me interested in world models and experimental model architecture work. A lot of the progress came from breaking the system into smaller pieces and testing them individually, as opposed to assuming a larger model or more data would fix the problem.

Breaking down diffucult problems into smaller tests allows me to check my assumptions and compare alternatives before committing to a direction. That approach has carried into the research I do now. The part I enjoy the most is implimenting new ideas, benchmarking them, and figuring out why a change helped or hurt the result.

Public scope

I am keeping the technical details of this project intentionally high level. I am comfortable discussing the overall approach and the experiments I ran, but I am not currently publishing the exact model architecture or training implementation.