A self-driving car sees the road as millions of laser points per second. A warehouse robot sees shelves, pallets and people the same way. Neither machine understands any of those points until a human, or a human-supervised tool, tells it what they are. That process is LiDAR annotation, and it quietly decides how safe and capable an autonomous system becomes.
LiDAR (Light Detection and Ranging) fires laser pulses and measures how long each one takes to bounce back. The result is a point cloud: a precise 3D map of the surroundings that works in darkness and gives exact distances. Cameras give colour and texture. LiDAR gives geometry. Together they form the perception backbone of most autonomous vehicles and a growing share of mobile robots.
But raw point clouds are just coordinates. To train a perception model, every car, cyclist, curb, forklift and shelf in that cloud needs a label. This guide explains how LiDAR annotation works, which annotation types matter for autonomous vehicles and robotics, which public datasets set the standard, and how teams build accurate labeled data at scale.
If you are new to the topic, our guide on what data annotation is and why it decides whether your model works covers the fundamentals. Here we go deeper into the 3D side.
Quick hint: LiDAR annotation is a 3D task, not a harder version of image labeling. Teams that plan it like 2D bounding box work usually underestimate cost, time and QA effort.
What is LiDAR Annotation?
LiDAR annotation is the process of labeling objects, surfaces and boundaries inside LiDAR point clouds so machine learning models can learn to detect, classify and track them in 3D. It is a core part of 3D data annotation, which differs from 2D labeling in tools, effort and use cases.
Every point in a LiDAR frame represents a surface that reflected a laser pulse. According to Kognic's LiDAR annotation best practices guide, each point usually carries:
• X, Y, Z coordinates – the position of the point in 3D space
• **Intensity **– how strongly the surface reflected the laser (useful for spotting lane paint, signs and reflective vests)
• **Return number **– for sensors that record multiple returns per pulse
• **Timestamp **– when the point was captured, which matters for moving objects and sensor sync
Point clouds have their own quirks. They get sparser with distance, so a pedestrian 80 metres away may be only a handful of points. Objects hidden behind others leave holes. Rain, dust and glass create noise. Annotators have to make confident decisions on incomplete shapes, which is why domain training and clear guidelines matter so much.
The terms annotation and labeling are often used interchangeably. If you want the precise difference, read our breakdown of data annotation vs data labeling.
Types of LiDAR Annotation
Different perception tasks need different label types. Most production programs combine two or three of the methods below.
| Annotation Type | What Annotators Do | Typical Output | Best for |
|---|---|---|---|
| 3D bounding boxes (cuboids) | Fit a tight box around each object with position, size and heading (yaw) | Center (x, y, z), length/width/height, rotation, class | Vehicle, pedestrian and cyclist detection |
| Semantic segmentation | Assign a class to every single point | Per-point class label (road, sidewalk, vegetation, building) | Drivable area, free space, scene understanding |
| Instance and panoptic segmentation | Per-point class plus a unique ID for each object | Class + instance ID per point | Crowded scenes, counting and separating objects |
| Object tracking (4D) | Keep the same ID on an object across a sequence of frames | Track IDs, trajectories, velocities | Motion prediction, path planning |
| Polylines and lane marking | Draw lines along lanes, curbs and road edges | Ordered 3D line points | HD maps, lane keeping, localization |
| Sensor fusion (2D-3D linking) | Link the 3D label to the same object in camera images | Cuboid projected to each camera view | Classification of ambiguous objects, multi-sensor models |
3D bounding boxes
Cuboids are the most common LiDAR label. A good box is tight, follows the true heading of the object and keeps the same size across frames for rigid objects such as cars. Research on one-click LiDAR annotation points out that sparse, low-resolution clouds make objects hard for humans to recognize, and that drawing 3D boxes is far more laborious than 2D work.
Semantic and Panoptic Segmentation
Point-wise labels give a model a full understanding of the scene, not just the objects in it. They are slower to produce but essential for separating drivable road from sidewalk, or open floor from racking in a warehouse.
4D tracking
Tracking adds time. The same object keeps one ID across every frame, so the model learns how things move. The Auto4D research paper on auto-labeling 3D boxes over time calls these space-time annotations 4D labels and shows how temporal context improves label quality.
Hint: Decide your label types per model, not per project. A detection model may only need cuboids, while a planning stack needs tracks, attributes and lane polylines on the same data.
LiDAR Annotation for Autonomous ehicles
Autonomous driving was the first big driver of LiDAR annotation, and it is still the most demanding. A perception stack has to locate every road user in 360 degrees, estimate its speed and heading, and do it reliably at night, in rain and at highway speeds. As our article on data collection in the automotive industry explains, collecting and labeling this data is the real bottleneck for most AV programs.
Key annotation tasks in AV programs include:
• **3D object detection **– cuboids for cars, trucks, buses, pedestrians, cyclists, animals and movable objects such as cones and barriers
• Multi-object tracking – consistent IDs across sequences so the planner can predict where each road user will be next
• Drivable area and free space – point-wise segmentation of road, sidewalk, parking areas and terrain
• Lane and road-edge polylines – used to build and validate HD maps
• Attribute tagging – parked vs moving vehicles, pedestrian pose, emergency vehicles, open doors
• Rare scenarios – debris, unusual vehicles, construction zones and jaywalkers that the model rarely sees
That last group is where many models fail. Our guide to edge case data collection for safer AI shows why rare scenarios deserve their own capture and labeling budget.
Most AV teams also annotate LiDAR alongside cameras and radar. Kognic's sensor fusion annotation guide describes the standard approach: one label is drawn in the point cloud and projected into every camera view using calibration data, so annotators can confirm what a sparse cluster of points really is. We cover the bigger picture in sensor fusion in physical AI.
LiDAR Annotation for Robotics
Robots use LiDAR for the same reason cars do: precise geometry that is not affected by lighting. The environments, objects and label taxonomies are different, though.
| Robotics Segment | What LiDAR helps with | Typical Labels |
|---|---|---|
| Warehouse AMRs and forklifts | Navigation, obstacle avoidance, pallet and rack detection | Cuboids for pallets, carts, people; segmentation of floor, racks, walls |
| Delivery and sidewalk robots | Safe movement among pedestrians, curbs and street furniture | Pedestrians, strollers, curbs, kerb ramps, poles |
| Agricultural robots | Row following, crop and tree detection, terrain mapping | Plants, trunks, rows, ground vs vegetation |
| Industrial and manufacturing robots | Cell mapping, collision avoidance, part localization | Machines, fixtures, safety zones, humans |
| Humanoids and mobile manipulators | Scene understanding for walking and grasping | Furniture, doors, stairs, graspable objects |
| Drones and inspection robots | Mapping, power line and structure inspection | Wires, towers, vegetation encroachment |
Warehouses are a good example. A study on LiDAR-based semantic perception for forklifts built a dedicated dual-LiDAR segmentation setup for outdoor industrial sites, because automotive datasets did not cover the objects forklifts deal with. This is why our article on physical AI in logistics calls warehouses the first battleground for robots: the data has to match the site.
The same idea applies across factories and hospitals, covered in physical AI in manufacturing, and in humanoid programs that need data built for how humanoid robots learn. LiDAR annotation for robots usually sits inside a wider multimodal data collection pipeline for physical AI that also includes video, IMU and joint data.
Hint: Do not reuse an automotive taxonomy for robots. A pallet, a tote, a dock door and a reflective safety vest each deserve their own class if your robot must behave differently around them.
Public LiDAR Datasets that Set the Benchmark
Public datasets show what good LiDAR annotation looks like and give teams a baseline before they invest in custom data. They are also a common starting point when evaluating physical AI datasets and their role in robot training.
| Dataset | Released | Sensors | Annotation highlights |
|---|---|---|---|
| KITTI | 2012 | Stereo cameras, Velodyne LiDAR | 22 sequences; the first large multimodal AV benchmark with 3D object labels |
| SemanticKITTI | 2019 | Velodyne HDL-64E LiDAR | Dense point-wise labels over the full 360° field of view, 28 classes, panoptic and 4D tasks added later |
| nuScenes | 2019 | 1 LiDAR, 6 cameras, 5 radars | 1,000 scenes of 20 seconds, 3D boxes for 23 classes with 8 attributes, about 7x the annotations of KITTI |
| Panoptic nuScenes | 2021 | Same as nuScenes | Every LiDAR point in 40,000 keyframes labeled, around 1.4 billion points |
These datasets are excellent for research and pre-training. They rarely match a specific product's sensor setup, geography or object list, which is why most teams still need custom annotation. Combining real data with simulation is another option, and we compare both in synthetic vs real-world data for robotics.
The LiDAR Annotation Workflow, step by step
A production LiDAR pipeline usually follows these stages:
1. Capture and sync the data: Record LiDAR together with cameras, radar, GPS and IMU, with accurate timestamps. Our guide to sensor data collection for robotics covers this stage in detail.
2. Calibrate and validate: Check intrinsic and extrinsic calibration between sensors and flag corrupt frames or drift before anyone starts labeling.
3. Define the taxonomy and guidelines: List every class, attribute and edge-case rule, with visual examples. Decide how to handle occlusion, truncation, groups of people and objects with very few points.
4. Pre-label with models: Run an existing detector or segmentation model to create draft labels. Humans then correct instead of drawing from scratch.
5. Human annotation and refinement: Trained annotators adjust cuboids, fix point-wise labels, keep track IDs consistent and add attributes, using camera views as reference.
6. **Multi-stage quality assurance: **Reviewers check a sample or every frame, run automated checks (box size jumps, ID switches, points outside boxes) and send errors back.
7. Export and feedback: Deliver labels in the format the training team needs (KITTI, nuScenes JSON, custom), then use model errors to choose what to label next.
Hint: Steps 2 and 3 feel slow, but they prevent the most expensive rework. A calibration error or vague guideline found after 50,000 frames means relabeling all of them.
Common challenges in LiDAR annotation (and how to solve them)
LiDAR labeling is slow and expensive. Kognic estimates it takes 6 to 10 times longer than 2D image annotation. A study on semi-automated LiDAR annotation for traffic infrastructure reported needing 960 person-hours from 20 professional annotators for its intersection dataset. Here are the problems teams hit most often.
| Challenge | Why it happens | How to solve it |
|---|---|---|
| Sparse, distant objects | Point density drops with range, so far objects have few points | Use camera views for context, aggregate points across frames, set minimum-point rules in guidelines |
| Occlusion and truncation | Objects hide behind others or leave the sensor's view | Annotate full extent using tracking history; tag occlusion level as an attribute |
| Calibration errors | Small LiDAR-camera misalignment breaks projected labels | Validate calibration at ingestion; reject or fix drifting sequences before labeling |
| Inconsistent labels | Different annotators interpret classes and box fit differently | Detailed guidelines with examples, calibration rounds for annotators, consensus review |
| Track ID switches | Rain, dust, glass and reflective surfaces create false points | Add noise and ghost-point classes; teach annotators what artifacts look like |
| Cost at scale | Manual 3D work per frame is slow | Model-assisted pre-labeling, active learning and keyframe interpolation |
iMerit notes that common dataset problems in dense cities include inconsistent class labels, limited taxonomy detail, LiDAR-camera alignment gaps and broken tracking across frames. Each one feeds straight into model errors. Our article on data quality challenges in physical AI training goes deeper into how these errors spread through training.
How to Measure LiDAR Annotation Quality
You cannot improve what you do not measure. Strong programs track a small set of metrics on every batch:
• 3D IoU (intersection over union) between annotator boxes and reviewer or gold-standard boxes
• Center, size and heading error – position offset in centimetres, dimension error and yaw error in degrees
• Point-wise accuracy and mIoU for segmentation tasks
• Missed and false objects – recall and precision against a gold set
• Track consistency – ID switches and fragmented tracks per sequence
• Attribute accuracy – how often states like moving or parked are correct
• Rework rate – the share of frames sent back by QA
These numbers tie directly to model performance. We explain the link in how to improve data quality in AI training.
In-house Vs. Outsourced LiDAR Annotation
Most teams eventually choose between building an internal labeling team and working with a specialist partner. The right choice depends on volume, data sensitivity and how fast the taxonomy changes.
| Factor | In-house Team | Outsourced Partner |
|---|---|---|
| Setup time | Slow: hiring, training, tooling | Fast: trained 3D annotators and tools ready |
| Control over guidelines | Direct and immediate | Strong with clear SLAs and feedback loops |
| Scaling up or down | Hard and costly | Flexible with project volume |
| Tooling cost | You buy or build 3D tools | Usually included |
| Best fit | Small, highly sensitive or research datasets | Large production volumes and tight timelines |
For a fuller comparison, see in-house vs outsourced data annotation services. If you are shortlisting vendors, our list of the best data annotation companies and services is a useful starting point.
Wrapping It
LiDAR gives autonomous vehicles and robots precise 3D vision, but only accurate labels turn that geometry into understanding. The best programs pick the right label types for each model, invest early in calibration and guidelines, combine model pre-labeling with skilled human review, and measure quality on every batch.
As autonomy spreads from public roads into warehouses, farms, hospitals and homes, demand for high-quality 3D training data will only grow. To see where this is heading, read our view on the future of physical AI.
Gamasome helps autonomous vehicle and robotics teams collect and annotate real-world sensor data, including LiDAR point clouds, with trained annotators and multi-stage QA. Talk to our team about your next LiDAR annotation project.
