DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generation

Jiazhe Guo1,2*, Yikang Ding2*†, Xiwu Chen3, Shuo Chen1,2, Bohan Li2,4, Yingshuang Zou1,2, Xiaoyang Lyu5, Feiyang Tan3, Xiaojuan Qi5, Zhiheng Li1, Hao Zhao1
1Tsinghua University 2MEGVII Technology 3Mach Drive 4Shanghai Jiao Tong University 5University of Hong Kong
*Equal Contribution † Project Leader
Paper Code

Abstract

Current generative models struggle to synthesize dynamic 4D driving scenes that simultaneously support temporal extrapolation and spatial novel view synthesis (NVS) without per-scene optimization. A key challenge lies in finding an efficient and generalizable geometric representation that seamlessly connects temporal and spatial synthesis. To address this, we propose DiST-4D, the first disentangled spatiotemporal diffusion framework for 4D driving scene generation, which leverages metric depth as the core geometric representation. DiST-4D decomposes the problem into two diffusion processes: DiST-T, which predicts future metric depth and multi-view RGB sequences directly from past observations, and DiST-S, which enables spatial NVS by training only on existing viewpoints while enforcing cycle consistency. This cycle consistency mechanism introduces a forward-backward rendering constraint, reducing the generalization gap between observed and unseen viewpoints. Metric depth is essential for both accurate reliable forecasting and accurate spatial NVS, as it provides a view-consistent geometric representation that generalizes well to unseen perspectives. Experiments demonstrate that DiST-4D achieves state-of-the-art performance in both temporal prediction and NVS tasks, while also delivering competitive performance in planning-related evaluations.

Overview

Overview pipeline

Overall framework of the proposed DiST-4D. DiST-4D is a disentangled spatiotemporal diffusion framework for 4D driving scene generation, leveraging metric depth as the core geometric representation to enable both temporal extrapolation and spatial novel view synthesis (NVS). (Top: Temporal Generation) DiST-T employs a diffusion model to predict future multi-camera RGB-D sequences from historical multi-camera images and control signals. The generated RGB-D sequences are then aggregated into point clouds, allowing for bullet time rendering. (Bottom: Spatial Generation) To enable spatial NVS, DiST-S leverages the predicted RGB-D sequences to generate novel viewpoints by first projecting them into sparse conditions and then refining them into dense RGB-D outputs.

Temporal RGB-D Generation

Spatial RGB-D Novel View Synthesis

Comparison with other GS works on Spatial RGB-D NVS

Multi-View Spatial NVS

More DiST-T Results

DiST-T: Zero-shot on Waymo

The number of cameras and their installation positions in the Waymo dataset, as well as the image resolutions, are quite different from those in nuScenes.

More DiST-S Results

BibTeX


        @article{guo2025dist4d,
          title={DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generation},
          author={Jiazhe Guo and Yikang Ding and Xiwu Chen and Shuo Chen and Bohan Li and Yingshuang Zou and Xiaoyang Lyu and Feiyang Tan and Xiaojuan Qi and Zhiheng Li and Hao Zhao},
          journal={arXiv preprint arXiv:2503.15208},
          year={2025}
      }