Vanilla ViT for Automotive Point Cloud Semantic Segmentation

Gilles Puy    Nermin Samet    Alexandre Boulch    Spyros Gidaris    Tuan-Hung Vu    Renaud Marlet

WACV 2027

Overview of VaViT: point embeddings are aggregated into non-empty pillar tokens processed by a plain ViT with RoPE, then decoded back to per-point predictions.

Abstract

Plain Transformers have become the de-facto architecture for processing text, audio, image, and video, offering a unified backbone for multimodal learning. However, state-of-the-art architectures for point cloud semantic segmentation remain dominated by U-Nets architectures where convolutions are interleaved with local or windowed attentions. In this work, we show how to effectively leverage vanilla, non-hierarchical ViTs for segmentation of large-scale automotive lidar scenes. We bridge the performance gap thanks to a carefully designed tokenizer, a lightweight decoder segmentation head, and tailored data augmentations. Our approach, VaViT for Vanilla ViT, matches or exceeds the performance of state-of-the-art methods while maintaining the simplicity of ViT architecture. We provide extensive evaluations on nuScenes, SemanticKITTI, and Waymo Open Dataset to validate the efficiency of our method.

BibTeX

@inproceedings{puy2027vavit,
  title     = {Vanilla ViT for Automotive Point Cloud Semantic Segmentation},
  author    = {Puy, Gilles and Samet, Nermin and Boulch, Alexandre and Gidaris, Spyros and Vu, Tuan-Hung and Marlet, Renaud},
  booktitle = {Winter Conference on Applications of Computer Vision (WACV)},
  year      = {2027}
}