Bounding Boxes, Polygons, Keypoints: A Complete Guide to Image Annotation
Image annotation is one of the largest categories of AI training work by volume. Every computer vision application, from autonomous vehicle perception systems to diagnostic radiology AI to satellite land classification tools, relies on human annotators who have labeled the images the model learned from.
The techniques used to annotate images vary significantly in complexity, precision requirements, and the type of AI application they support. This guide covers the main annotation types, how they work, what they are used for, and what separates good annotation from great annotation in each category.
Why image annotation exists
AI models that work with visual data cannot intuitively understand what they are seeing. They learn by being shown large numbers of images where a human has already identified what is in the image and where.
A model being trained to detect cracks in concrete infrastructure needs to be shown thousands of images where the cracks have already been precisely outlined. A model being trained to identify surgical instruments in an operating theatre needs thousands of frames where each instrument has been labeled and located. A model learning to assess wildfire risk from aerial imagery needs satellite images where areas of dry vegetation, water bodies, and built structures have all been correctly classified.
The annotation work that creates this training data is not trivial. The quality of the annotations directly determines the quality of the model, and the annotation requirements differ significantly depending on what the model needs to learn.
Bounding box annotation
What it is
A bounding box is a rectangle drawn around an object of interest in an image. It is defined by the coordinates of its top-left corner and its width and height.
Bounding boxes are the simplest and most widely used annotation type. They tell a model where something is and what it is called, without specifying the exact shape.
What it looks like
Imagine an annotator working on a dataset for a port management AI system that needs to identify shipping containers on a dock. For each image, they draw a rectangle around every visible container, then label each rectangle: "20ft standard", "40ft high cube", "refrigerated unit." Containers that are partially obscured are annotated with a truncation flag. Containers that are stacked and touching are annotated as separate objects rather than merged.
Where it is used
- Pedestrian and vehicle detection in traffic management systems
- Parcel and package identification on logistics conveyor belts
- Cell detection in microscopy images
- Equipment identification in industrial inspection
- Product detection on retail shelves
What good bounding box annotation looks like
The box should be as tight as possible around the object without cutting into it. A bounding box that includes significant background space around the target object teaches the model that the background is part of the object. A box that clips part of the object teaches the model an incomplete version of what it looks like.
Guidelines for specific projects typically specify exactly how to handle objects at image edges, objects that overlap, and objects that are partially occluded. Following these guidelines precisely is what separates consistent, high-quality annotation from work that degrades model performance.
Polygon annotation
What it is
A polygon annotation traces the outline of an object using a series of connected points. Unlike a bounding box, which is always a rectangle, a polygon follows the actual shape of the object.
Polygon annotation is more precise and more time-consuming than bounding boxes. It is used when the exact outline of an object matters for the AI application.
What it looks like
Consider an annotator working on a dataset for a satellite imagery AI being trained to map informal settlements for urban planning purposes. Each building footprint needs to be traced precisely because the AI needs to calculate coverage area, identify building density, and distinguish structures from surrounding features like courtyards or access paths. For each building, the annotator places points along the roofline, following the actual outline of the structure rather than drawing a rectangle around it.
The same precision is required in medical imaging annotation. A pathologist-assisted AI model being trained to measure tumour volume from MRI scans needs polygon annotations that trace the actual tumour boundary, not a rectangle that includes surrounding healthy tissue.
Where it is used
- Organ and tissue segmentation in medical imaging
- Building and infrastructure mapping from aerial imagery
- Surgical instrument tracking in laparoscopic video
- Geological feature mapping in satellite imagery
- Defect detection in industrial quality control
What good polygon annotation looks like
Polygon density matters. Too few points and the annotation smooths over important contour details. Too many and the annotation becomes unnecessarily complex without improving accuracy. Guidelines typically specify the maximum distance between annotation points or the minimum number of points for different object types.
Consistent placement of anchor points at the same anatomical or structural landmarks is also important when multiple annotators are working on the same dataset. Two annotators tracing a kidney boundary from the same MRI scan should arrive at annotations that closely overlap, not diverge significantly based on different interpretations of where the boundary lies.
Semantic segmentation
What it is
Semantic segmentation assigns every single pixel in an image to a category. Instead of drawing shapes around objects, the annotator classifies each pixel as belonging to a specific class: road, pavement, building, vegetation, sky, vehicle, pedestrian.
It is the most detailed and most time-intensive form of image annotation. It is also the form that produces the richest training data.
What it looks like
An annotator working on a dataset for an autonomous vehicle perception system is given a dashcam image of a busy intersection. They need to classify every pixel: the road surface, the lane markings, each parked car, each moving vehicle, every pedestrian, the buildings on each side, the traffic signals, the sky. There are no unlabeled pixels. Areas at the boundary between two categories, the edge of a pedestrian's coat against the road surface, require careful judgment about which category applies.
In agriculture, semantic segmentation is used to train AI systems that assess crop health from drone imagery. An annotator classifies each pixel as healthy crop, diseased crop, bare soil, irrigation channel, or track. The precision of these pixel-level classifications directly affects the AI's ability to map disease spread across a field.
Where it is used
- Autonomous vehicle and drone perception
- Agricultural monitoring and crop health assessment
- Medical image analysis where tissue type boundaries matter
- Environmental monitoring from satellite data
- Construction site safety monitoring
What good semantic segmentation looks like
Boundary accuracy is the primary quality metric. The boundary pixels between two categories are the hardest to annotate and the most important for model performance. A semantic segmentation model that cannot accurately locate object boundaries will struggle with the real-world tasks it is designed for, such as deciding whether a pedestrian is on the pavement or has stepped into the road.
Most platforms use specialised annotation tools with edge-detection assistance that makes boundary annotation faster and more precise. Learning to use these tools effectively, rather than manually clicking every boundary pixel, is a significant efficiency gain.
Keypoint annotation
What it is
Keypoint annotation marks specific, named points on an object rather than its outline or location. Each keypoint has a specific semantic meaning defined by the annotation guidelines.
What it looks like
In a dataset for an AI system being trained to assess gait abnormalities from video, an annotator places keypoints on specific anatomical landmarks for each frame: the top of the skull, the ear canal, the shoulder joint, the elbow, the wrist, each finger knuckle, the hip joint, the knee, the ankle, the heel, each toe. When the model is trained, it learns to locate these landmarks in new images, enabling it to track joint angles and movement patterns.
In retail, keypoint annotation is used for planogram compliance AI: systems that check whether products on a shelf are in the correct position. Annotators mark specific reference points on each product, the corners of the label, the cap of a bottle, the seam of a package, which the model uses to assess orientation and position.
Where it is used
- Pose estimation for sports performance analysis
- Gait analysis in physiotherapy and rehabilitation
- Facial expression analysis in emotion recognition AI
- Hand tracking for gesture control interfaces
- Orthopaedic assessment tools
What good keypoint annotation looks like
Precise, consistent landmark placement is essential. If ten annotators place the "knee joint" keypoint at ten slightly different positions on the same image, the model learns a blurred version of where the knee joint is. Guidelines typically include anatomical or structural definitions of each keypoint, reference images showing correct placement, and instructions for handling obscured or ambiguous landmarks.
3D and LiDAR annotation
What it is
LiDAR (Light Detection and Ranging) sensors generate three-dimensional point clouds: millions of data points in 3D space representing the surfaces around a sensor. Annotating LiDAR data requires drawing 3D bounding boxes around objects in the point cloud rather than 2D rectangles on a flat image.
What it looks like
An annotator working on a dataset for an autonomous lorry navigation system is given a 3D point cloud of a motorway service area. They need to draw 3D bounding boxes around every other vehicle, every pedestrian, every infrastructure element like bollards and signs, and every unexpected obstacle. Each box needs to be correctly dimensioned and oriented in three-dimensional space, not just positioned correctly when viewed from above.
The same techniques apply in robotics, where 3D annotation helps manipulation AI understand the shape and position of objects it needs to grasp or avoid, and in construction, where 3D point cloud annotation trains AI to monitor building progress against architectural plans.
Where it is used
- Autonomous vehicle and drone navigation
- Robotics and warehouse automation
- Construction site monitoring
- Underground surveying and mining
What good 3D annotation looks like
Accurate dimensioning in all three axes, not just the two visible from a top-down view. A 3D bounding box for a pedestrian that is correctly sized in width and length but too short in height will cause the model to underestimate human dimensions. 3D annotation tools typically provide multiple viewing angles simultaneously, and proficient annotators use all available views rather than relying on a single perspective.
Choosing the right annotation type for a project
AI companies select annotation types based on what the model needs to learn. As an annotator, you will typically be told which technique to use rather than choosing it yourself. But understanding why a particular technique is specified helps you understand what quality looks like.
E-commerce product search
| Field | Details |
|---|---|
| Annotation type | Bounding box |
| Why | Needs location, not exact shape |
Surgical AI
| Field | Details |
|---|---|
| Annotation type | Polygon |
| Why | Needs precise instrument outline |
Self-driving vehicles
| Field | Details |
|---|---|
| Annotation type | Semantic segmentation |
| Why | Needs to understand every part of the scene |
Sports biomechanics AI
| Field | Details |
|---|---|
| Annotation type | Keypoint |
| Why | Needs to track specific anatomical points |
Warehouse robotics
| Field | Details |
|---|---|
| Annotation type | 3D bounding box |
| Why | Needs depth and spatial orientation |
Dermatology AI
| Field | Details |
|---|---|
| Annotation type | Polygon + classification |
| Why | Needs lesion boundary and type |
Frequently asked questions
How long does it take to annotate an image? It varies enormously by annotation type and image complexity. A bounding box on a clear, unoccluded object takes seconds. A semantic segmentation of a complex urban scene with dozens of object categories can take thirty minutes or more. Annotation tools with AI-assist features can significantly reduce time on repetitive or similar images.
What annotation tools are commonly used? Platforms have their own proprietary tools, but common external tools include CVAT, Labelbox, Scale AI's annotation interface, and Roboflow. Learning the fundamentals of one tool transfers reasonably well to others. The underlying techniques are the same across platforms.
Does annotation speed improve with experience? Yes, significantly. Experienced annotators develop faster workflows, learn to use keyboard shortcuts and AI-assist features effectively, and make fewer errors that require correction. Annotation speed at month six is typically two to three times faster than at month one for the same task type, without loss of quality.
Is domain knowledge required for image annotation? For general annotation tasks like vehicle or pedestrian detection, no. For specialist annotation like medical imaging, satellite feature mapping, or manufacturing quality control, yes. Annotating a chest X-ray to locate and classify pulmonary abnormalities requires either clinical training or very detailed guidelines and close supervision. Misidentifying anatomical structures or pathological features has direct downstream consequences for the AI system.
Summary
Image annotation encompasses a range of techniques from fast bounding boxes to detailed pixel-level segmentation, each suited to different AI applications. The quality criteria differ by technique but share a common thread: precise, consistent application of the annotation guidelines, careful handling of boundary and edge cases, and the domain knowledge to recognise when an annotation requires a judgment call.
The highest-paid image annotation work sits at the intersection of technical precision and specialist knowledge: medical imaging annotation that requires clinical understanding, satellite feature annotation that requires geography or environmental science knowledge, and industrial inspection annotation that requires engineering domain expertise.