AutoExpert: Automating 3D LiDAR Annotation from Expert-Crafted Guidelines

Abstract

The contemporary paradigm of scaling data annotation, crucial for developing machine learning solutions, is to hire ordinary human annotators and instruct them with expert-crafted guidelines to label data. This paradigm is laborious, tedious, and costly, motivating us to study an open problem, auto-annotation with expert-crafted guidelines (dubbed AutoExpert). We develop benchmarks by redesigning the evaluation protocol and re-annotating data with nuScenes and PandaSet, two 3D detection datasets for autonomous driving research that provide expert-crafted annotation guidelines. Their guidelines define 18 and 25 object classes, respectively, using nuanced language descriptions and a few visual examples. Following the guidelines that require using 3D cuboids to label LiDAR data, AutoExpert requires algorithms to learn on few-shot labeled images and texts to perform the task of 3D detection on LiDAR data. Apparently, the challenges of AutoExpert lie in the data-modality and task discrepancy. Nevertheless, public foundation models (FMs) serve as promising tools to tackle these challenges. To address AutoExpert, we adopt a conceptually simple pipeline consisting of three components: (1) 2D object detection and segmentation in RGB images, (2) lifting 2D detections into 3D using known sensor poses, and (3) 3D cuboids generation for the 2D detections. Within this pipeline, we enhance and evaluate a variety of methods such as open-vocabulary detectors, few-shot detectors, and self-supervised learned detectors. We also develop novel techniques, leading to refined components that boost 3D detection mAP from 12.1 to 25.4 on the AutoExpert-nuScenes benchmark.

Benchmarks and protocol

AutoExpert studies auto-annotation from expert-crafted guidelines: algorithms learn textual class definitions and a few federated 2D-labeled RGB examples, then annotate LiDAR data with 3D cuboids. The target classes follow the source annotation guidelines, including rare and nuanced categories.

The training examples contain 4–8 selected RGB images per class. Each example is labeled only for its target class. No 3D annotations are used for training; validation annotations support model and prompt selection.

Correcting PandaSet annotations

AutoExpert-PandaSet uses our manually corrected and re-annotated subset. The original PandaSet cuboids include misaligned or ghost boxes whose objects are not identifiable in the RGB views. We re-annotate the selected frames with aligned 3D boxes for visually identifiable objects.

Method: auto3D

Our pipeline adapts available foundation models to expert-crafted guidelines and lifts their 2D predictions into 3D. auto3D denotes the complete method; v-MHT is its VLM-guided cuboid-generation component.

Multimodal few-shot finetuning

VLM-guided multi-hypothesis testing (v-MHT)

Contribution of 3D refinement components

AutoExpert-nuScenes ablation, following the paper. v-MHT reaches 21.9 mAP; the full auto3D system reaches 25.4 mAP after the additional refinements.

Results reported in the paper

3D mAP averages AP at ground-plane center-distance thresholds of 0.5, 1.0, 2.0, and 4.0 meters. Scores below are on a 0–100 scale. ft-GD denotes GroundingDINO finetuned with refined class names and few-shot guideline examples.

AutoExpert-nuScenes · 18 classes

AutoExpert-PandaSet · our corrected annotations, 25 classes

PandaSet NDS is adapted to use translation, scale, and orientation errors, excluding velocity and attribute errors. NDS values should be interpreted within each dataset's evaluation protocol. The complete paper includes additional enhanced baseline variants.

Qualitative results

CVPR 2026 Workshop Challenge

The Automated 3D Cuboid Annotation on LiDAR Data challenge evaluates AutoExpert-PandaSet using our corrected annotations and ranks entries by NDS. Challenge submissions are separate from the paper's baseline comparisons.

The October 20, 2026 leaderboard snapshot ranks iMerit first, with 23.24 mAP / 29.36 NDS. The organizer entry, listed as AnnoExpert, scores 18.36 mAP / 27.61 NDS; these round to the paper's auto3D PandaSet result of 18.4 / 27.6.

The challenge server displays scores on a 0–1 scale; this page uses 0–100. Visit the live leaderboard for current standings and submission instructions.

Code and data

Citation

@inproceedings{ma2026autoexpert,
  title = {{AutoExpert}: Automating 3D {LiDAR} Annotation from Expert-Crafted Guidelines},
  author = {Ma, Yechi and Hua, Wei and Kong, Shu},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2026},
  url = {https://arxiv.org/abs/2506.02914}
}