Code World Model: Coding Agent as World Brain

Yiwen Chen1,2, Guosheng Lin2, Chi Zhang1
1 AGI Lab, Westlake University 2 Nanyang Technological University

Abstract

World models aim to simulate how complex environments evolve under actions and events, yet existing video-based world models primarily learn dynamics from visual observations, which reveal outcomes rather than the underlying knowledge, rules, and mechanisms governing world evolution. This makes it difficult to maintain persistent consequences and support coherent, open-ended evolution. We introduce Code World Model, a framework that separates world evolution from visual realization by combining the reasoning and coding capabilities of language models with the generative priors of video models. A coding agent serves as the world brain, reasoning about events and their consequences and generating executable code to maintain persistent world state and perform rule-consistent evolution. To connect executable state with visual generation, we introduce a proxy representation that encodes frame-wise spatiotemporal constraints and is compiled into a proxy video, which conditions a video model to render high-fidelity visual observations. We further develop data pipelines for constructing aligned proxy-observation pairs from gameplay and real-world videos. After fine-tuning on paired gameplay data, MiniMax-H3 follows proxy-based spatiotemporal specifications from simple interactive worlds built by the coding agent while preserving rich visual details and dynamics. These results demonstrate the potential of combining code for persistent world evolution with video models for flexible visual realization, providing a new path toward open-ended world models.

Code World Model Pipeline

Pipeline of the proposed Code World Model
Pipeline of the proposed Code World Model. The coding agent serves as the brain of the world model, translating interaction intent into code that updates the world state. The updated world state is compiled into a proxy and paired with a text prompt to condition the video model, which generates the corresponding visual observations. Generated observations and state feedback then support subsequent interaction and world evolution.

Paired Data Construction

Game RGB target showing a running person on a highway RGB target
Aligned game proxy with semantic scene regions, a human primitive, and vehicle boxes proxy

(a) Game pair data.

Real RGB target showing a residential street, a car, and a walking person RGB target
Aligned real-world proxy with semantic scene regions, a human primitive, and a vehicle box proxy

(b) Real pair data.

Paired data construction. Each adjacent RGB-proxy pair is aligned at the same frame. (a) For game data, code reads structured runtime annotations to automatically produce pixel-wise, instance-level proxy annotations and preserve identity across frames. Note that, to ensure that the coding agent can easily reproduce the proxy at inference time, we exclude skeletal pose, detailed 3D models, and other representations that are difficult for coding agents to construct and maintain. Because extraction and compilation are code-based, a coding agent can easily add or revise the retained annotation channels through small, local program changes. (b) For real data, calibrated 3D reconstruction and object annotations compile the aligned proxy offline, preserving scene layout, entity locations, and occlusion without action labels or access to game-engine runtime state.

BibTeX

@misc{chen2026codeworldmodel,
  title         = {Code World Model: Coding Agent as World Brain},
  author        = {Yiwen Chen and Guosheng Lin and Chi Zhang},
  year          = {2026},
  eprint        = {2608.25927},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2608.25927}
}