NeuralSRender: G-Buffer Guided Video Diffusion for Sim-to-Real Rendering

The scarcity of high-quality annotated visual data remains a critical bottleneck for autonomous-driving perception.

Simulators provide training data at scale, but the limited realism of synthetic videos compromises perception performance in real-world applications.

To address this limitation, we propose NeuralSRender, a sim-to-real video translation framework that enhances the photorealism of synthetic videos while maintaining geometric consistency with the source scenes.

Built on a pretrained video diffusion transformer (DiT), NeuralSRender uses a lightweight projector to inject geometric and material priors from G-buffers, and LoRA enables parameter-efficient adaptation.

The model is trained in two stages: it first learns G-buffer conditioning from synthetic videos with accurate G-buffers, then adapts to real-world appearance using real videos with estimated G-buffers.

During the second stage, an additional LoRA module is trained while the projector and first-stage LoRA remain frozen to reduce interference from estimation noise.

NeuralSRender introduces only 1.45% parameter overhead and can be trained on consumer-grade GPUs with 24 GB of memory.

To support training and evaluation, we introduce GenReal, a large-scale driving video dataset comprising synthetic and real-world videos with corresponding G-buffers.

Experiments demonstrate improved photorealism with competitive geometric consistency.

Training perception models on our translated videos instead of raw simulator renders improves detection ${\mathrm{AP}}_{50}$ on nuImages from 35.1 to 40.6 and segmentation mIoU on Cityscapes from 32.4 to 40.8.

Our code, data, and model weights are available at: https://github.com/WHU-USI3DV/NeuralSRender.