Interactive Generative World Model

How it started

In August 2024, I came across GameNGen and the paper Diffusion Models Are Real-Time Game Engines. The demo showed DOOM being generated interactively by a neural network instead of rendered by a normal game engine. The demo ran on a TPU-v5 and used Stable Diffusion as the base model. I wanted to see if I could make a world model from scratch that could train and run at a comparable 20 fps on my RTX 4090.

I started working on my own version in my free time. The paper described their overall training method, but there was no official implementation to build from. Rather than reproducing their Stable Diffusion architecture, I wrote my own training and inference pipeline and experimented with a smaller action-conditioned model that was practical on my hardware.

October 2024: First working DOOM version

By October 28, about two months after the GameNGen paper was released, I had a first version running with DOOM. It was a custom U-Net I built and trained from scratch. The generation was still rough and lacked consistency and fine detail, but I could control the game and move through an environment being generated by the model in real time.

This result proved to me that I could actually create a working world model, and it wasn't some impossible task. With that baseline, I could focus on which parts of the architecture, training, and data were limiting the model, then improve them iteratively.

What happened next

The early version used a U-Net, but as I kept experimenting I started hitting quality limits that didn't have an obvious fix. Instead of continuing to scale the same setup, I started building small synthetic environments so I could isolate parts of the system and test them directly.

Those tests surfaced failure modes around control fidelity and temporal consistency. I also tested alternatives that improved quality but came with memory costs that did not justify the tradeoff.

I eventually ran benchmarks between the U-Net and a DiT-based architecture. The DiT used much less VRAM and produced better samples than the U-Net, but took more training steps to converge. I later adapted a sparse MoE DiT backbone for the world model and kept changing the conditioning and training setup as I found new failure points.

Throughout the project I used new research papers as sources for ideas. I would implement the part of the paper I was interested in, check whether I could reproduce the reported behavior, and then test it in my experiments. Some methods transferred well, others required modifications, and some did not meaningfully help.

May 2025: Much better quality

By May 27, 2025, I had a much better model trained on a small custom Unreal Engine environment to stress-test fine details. It supported more dynamic interaction than the first DOOM version. Visual quality had improved while retaining interactive performance near 20 FPS on my RTX 4090.

The model generated a continuing visual response as I moved through the environment. It was still an experimental model trained on a small environment, but it was a large improvement over the first DOOM version and a better demonstration of the direction I wanted to pursue.

Revisiting the problem

As my college workload increased, I had to put this project aside. More recently, I have returned to world-model research and have been studying newer approaches. One challenge that particularly interests me is persistent memory.

A world model can generate a convincing environment while maintaining its recent context, yet struggle to preserve details when a user leaves an area and later returns. I have been investigating ways to improve consistency across longer interactions.

Recent work includes repeatable RGB-D scene capture, controlled evaluation splits, and tests of how well a model retains scene information across changes in viewpoint and time. Long-horizon consistency remains an open problem, and the current architecture is unpublished.

What I learned

This project is what really got me interested in world models and experimental model architecture work. A lot of the progress came from breaking the system into smaller pieces and testing them individually, as opposed to assuming a larger model or more data would fix the problem.

Breaking down difficult problems into smaller tests allows me to check my assumptions and compare alternatives before committing to a direction. That approach has carried into the research I do now. The part I enjoy the most is implementing new ideas, benchmarking them, and figuring out why a change helped or hurt the result.

Public scope

I am keeping the technical details of this project intentionally high level. I am comfortable discussing the overall approach and the experiments I ran, but I am not currently publishing the exact model architecture or training implementation.