Scalable Data Engine for Perception Tasks in Autonomous Driving - Joint Image-Mask Generation via Latent Diffusion Models

Abstract

Scaling pixel-level annotated data, e.g, a segmentation map, for autonomous driving perception is constrained by the high cost of manual labelling. This thesis investigates joint image-mask generation via latent diffusion to reduce the annotation bottleneck. We propose a joint latent generation model built on a pretrained Stable Diffusion 1.5 backbone, in which segmentation masks are encoded into RGB space using Fibonacci RGB (F-RGB) encoding and processed together with images through an 8-channel UNet. The model is trained and evaluated on the BDD100K dataset. We validate that F-RGB-encoded masks survive the VAE round trip, and show that the joint model clearly outperforming a pixel-based DDPM baseline trained from scratch. The utility of the generated pairs is evaluated by training four segmentation models, FPN, PSPNet, SegFormer, and SETR-Naive, under synthetic data augmentation and synthetic pretraining conditions. Both strategies yield modest improvements over the real-data-only baseline. The results suggest that joint latent diffusion is a viable approach for scalable data generation in autonomous driving, though synthetic data remains a supplement to real annotated data rather than a replacement.