SyntheOcc: Synthesize Geometric Controlled Street View Images through 3D Semantic MPIs

arxiv preprint
Leheng Li1, Weichao Qiu3, Yingjie Cai3, Xu Yan3, Qing Lian2, Bingbing Liu3, Ying-Cong Chen1,2
1HKUST(GZ)2HKUST3HUAWEI Noah's Ark Lab
TRI logo

 arXiv  Slides  Code

We enable geometric-controlled generation that conveys the user editing in 3D voxel space to generate realistic street view images.

Teaser Image

Overview: We achieve 3D geometric control in image generation by utilizing our proposed 3D semantic multiplane images to encode scene occupancy. In our framework, we can edit the occupied state and semantics of every voxel in 3D space to control the image generation, thereby opening up a wide spectrum of applications as shown in the top right.

Teaser Image 1

We use 3D semantic MPI to represent the irregular occupancy conditions.

Abstract

The advancement of autonomous driving is increasingly reliant on high-quality annotated datasets, especially in the task of 3D occupancy prediction, where the occupancy labels require dense 3D annotation with significant human effort. In this paper, we propose SyntheOcc, which denotes a diffusion model that Synthesize photorealistic and geometric controllable images by conditioning Occupancy labels in driving scenarios. This yields an unlimited amount of diverse, annotated, and controllable datasets for applications like training perception models and simulation. SyntheOcc addresses the critical challenge of how to efficiently encode 3D geometric information as conditional input to a 2D diffusion model. Our approach innovatively incorporates 3D semantic multi-plane images (MPIs) to provide comprehensive and spatially aligned 3D scene descriptions for conditioning. By doing so, SyntheOcc can generate photorealistic multi-view images and videos that faithfully align with the given geometric labels (semantics in 3D voxel space). Extensive qualitative and quantitative evaluations of SyntheOcc on the nuScenes dataset prove its effectiveness in generating controllable occupancy datasets that serve as an effective data augmentation to perception models.

Video generation

We design our SyntheOcc for CogVideoX to enable controllable video generation.

Applications

Use our synthetic data for downstream evaluations.

Results


Generation with weather variation

From top to bottom, we display images of fusion of 3D semantic MPI, synthesized images of sandstorm, snow, foggy, rainy, day night, day time, and ground truth.

Video demonstration of weather variation

Citation

@article{li2024syntheocc,
            title={SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D Semantic MPIs},
            author={Li, Leheng and Qiu, Weichao and Cai, Yingjie and Yan, Xu and Lian, Qing and Liu, Bingbing and Chen, Ying-Cong},
            journal={arXiv preprint arXiv:2410.00337},
            year={2024}
            }