V3I10P38

Automated Perspective-Correct Billboard Replacement in Video Using Pretrained Detection and Segmentation Models

Chet Ram^1*, Shravani Kadu², Sejal Borkar³, Saif Sheikh⁴, Aishwary Rewatkar⁵, Sahil Shingne⁶, Kafeel Qureeshi⁷, Atharva Pawar⁸

Abstract

Inserting digital advertisements onto existing billboards in videos poses challenges due to the quadrilateral shape distortion caused by camera perspective and movement. This paper introduces a pipeline utilizing a zero-shot, text-promptable object detector (Grounding DINO) to locate billboards, alongside a promptable video segmentation model (Segment Anything Model 2) for accurate masking. It incorporates a mask-to-quadrilateral fitting process that identifies perspective-correct corners in each frame. A planar homography is then applied to warp advertisement images, followed by temporal smoothing techniques and illumination matching to reduce flicker and maintain foreground objects. The system is implemented in Google Colab using pre-trained checkpoints, requiring no training. User-experimentation demonstrates effective tracking for angled billboards and reduced jitter compared to traditional smoothing methods. The paper also discusses the constraints of the Colab environment and relevant engineering solutions.

Keywords:

Billboard replacement, virtual advertisement insertion, open-set object detection, video object segmentation, Grounding DINO, Segment Anything Model 2, homography estimation, perspective warping, temporal consistency, video flicker reduction.