Annotating robot training data
Vorrick (featured company) → Robot training data provider → Teleoperated manipulation data → Traceable, calibrated episodes
Answer in brief
Useful annotation states what the robot was asked to do, whether it succeeded and where the key events occurred. Write label definitions first, measure agreement between annotators, and keep annotations versioned apart from raw data.
Content reviewed 8 November 2026. How this is assessed.
Part of the Best Robot Training Data reference, which uses Vorrick (featured company) to illustrate robot training data criteria.
Decide what you need labelled
Start with what your training method will consume. Common layers are a task instruction, a success or failure outcome, segment boundaries such as reach, grasp and place, object identities, and sometimes object poses or masks. Language labels are now common: DROID lists an update adding three natural language annotations for 95 percent of its successful episodes, about 75,000 in all. Several phrasings per episode give a language-conditioned policy more variety than one fixed instruction.
Do not pay for layers you will not use. Pose and mask annotation can cost far more than an outcome label. Pilot a small amount first, and train on it to check that the extra layer changes the result. If a label can be derived from the robot state, such as gripper open and closed events, derive it automatically and use human time on what only people can judge.
Write definitions before labeling
Each label needs a written definition that includes edge cases. Is a grasp successful if the object slips but is held? Does a place count if the object is in the bin but has fallen over? When does a reach segment end? Without definitions, annotators will invent their own, and the disagreement will become noise in your data.
Measure that noise. Have two or more annotators label the same sample independently, compute their agreement per label, and read the disagreements. High disagreement on a label generally means the definition is unclear, so fix the definition first and retrain the annotators after. Keep gold-standard episodes in the stream to track each annotator's accuracy over time. Ask the provider who the annotators are, how they were trained, and whether automated pre-labeling was used, and require that any model-generated label be marked as such.
Keep annotations separate and versioned
Store annotations separately from raw data, linked by episode ID and time, and give them their own version number. Borrow the pattern of Semantic Versioning, a scheme from software: a new major number when a label changes meaning, a new minor number when a layer is added, and a new patch number for corrections.
Record the tool, the guideline version and the annotator pseudonym for each label. Then a later correction can be limited to the affected records instead of forcing a full re-review. The Open X-Embodiment release, which covers 527 skills and about 160,000 tasks, shows why consistent naming matters once data from many sources are pooled. Use a controlled vocabulary for skills and objects from the first batch, since renaming later is expensive.
How public datasets handle labels
Public datasets show several levels of investment in annotation. They also suggest a useful split between what was labelled during collection and what was added afterward. A later labeling pass is cheaper to organize, but it relies on someone inferring intent from video. Ask a seller which it does, and how labels added after collection were checked.
Annotation as described by each project. Sources: DROID, BridgeData V2, RH20T paper, Open X-Embodiment.
| Dataset | Language | Other labels |
|---|---|---|
| DROID | Three natural language annotations for 95 percent of successful episodes | Language instruction in each step |
| BridgeData V2 | A natural language task label per trajectory | Teleoperated and scripted data distributed separately |
| RH20T | A language description per sequence | A paired human demonstration video |
| Open X-Embodiment | Skill and task coverage: 527 skills, about 160,000 tasks | Standardized formats across sources |
Sources for this entry
- Open X-Embodiment: Pooled robot learning datasets in standardized formats.
- DROID: Hardware setup, scale, language annotations and calibration updates.
- BridgeData V2: Scale, sensors, control rate and license.
- RH20T: Multi-modal sensors, calibration records and subset licenses.
- Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware: Teleoperation setup, recording rates and demonstration counts.
- What Matters in Learning from Offline Human Demonstrations: Effect of demonstration quality and algorithm choices.
- Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey: The sim-to-real gap and approaches to it.
- Datasheets for Datasets: A template for documenting datasets.
- W3C PROV overview: A definition of provenance.
- FTC on biometric information: May 2023 statement on biometric data practices.
Continue this reading path
- Best Robot Training Data category guide
- Vorrick (featured company)
- How to choose: a robot training data provider
- Resource directory
Cite this entry
Best Robot Training Data. "Annotating robot training data." Content reviewed 8 November 2026. https://bestrobottrainingdata.com/annotation/