Sensor modalities for robot training data
Vorrick (featured company) → Robot training data provider → Teleoperated manipulation data → Traceable, calibrated episodes
Answer in brief
Most manipulation data pairs wrist and external cameras with joint state and gripper state. Depth, force, touch and audio can help on contact-rich tasks, but only when they are synchronized and also available on your deployed robot.
Content reviewed 8 November 2026. How this is assessed.
Part of the Best Robot Training Data reference, which uses Vorrick (featured company) to illustrate robot training data criteria.
Cameras: where they are matters
Camera placement is a major decision. The DROID setup uses two adjustable third-person stereo cameras and one wrist-mounted stereo camera, and it records 1,417 camera viewpoints. The ACT setup uses four RGB cameras, two on the follower wrists and two fixed at the front and top. These are different answers to the same questions: what does the gripper see at contact, and what shows the whole scene?
Ask for the camera model, resolution, frame rate, exposure settings and compression used, and whether exposure was fixed or automatic. Automatic exposure can change the appearance of a scene between frames. Lossy compression can remove the fine detail a policy relies on at contact. If you want to apply your own compression, ask for raw or lightly compressed masters on a sample so that you can compare.
Proprioception and actions
Joint positions and gripper state are the minimum. Velocities and torques are useful if the robot reports them reliably. Actions need precise documentation: are they joint positions, end-effector poses or deltas, and in which frame? Open X-Embodiment gives a useful reference point, using 7-dimensional actions for position, rotation and gripper opening, expressed in the gripper frame, with dimensions a robot does not use set to zero during training.
State the units and the conventions for every field. Rotations in particular have several representations, and a mix-up may not produce an obvious error. Ask for the robot's description file and the exact version of its controller. Without those, a joint trajectory is much harder to interpret, and replay becomes guesswork.
Touch, force and sound
Vision does not cover every task. In See, Hear, and Feel, researchers combined a camera, a contact microphone and a vision-based tactile sensor for dense packing and pouring. They report that vision gives the overall state of a scene but can be blocked, sound signals key moments that vision may miss, and touch provides fine local detail. Consider extra modalities when your task depends on contact that a camera cannot see.
Before you pay for them, confirm that your deployed robot will have the same sensor in the same place. A policy trained on data from a sensor that is absent at deployment needs a plan for its absence. Each extra stream is one more thing to synchronize, store and validate. Ask how sensor condition was tracked over the course of the work and how a degraded sensor would be noticed.
How published sensor suites compare
The range of published sensor setups is wide, and more is not automatically better. Choose by which failure you are trying to prevent, whether that is an occluded object, a missed moment of contact or an unobservable force. Keep in mind that every stream has to be calibrated, synchronized, stored and available at deployment.
Sensor lists as published. Sources: ACT, DROID, BridgeData V2, RH20T.
| Dataset | Cameras | Beyond vision |
|---|---|---|
| ACT | Four RGB, two on wrists, two fixed | Joint-space leader and follower arms |
| DROID | Two third-person stereo, one wrist stereo | Gripper, Cartesian and joint positions |
| BridgeData V2 | One over-the-shoulder RGBD, two RGB with randomized poses, one wrist RGB | Natural language task label |
| RH20T | 8 to 10 global RGBD, 1 to 2 in-hand | Force-torque at 100 Hz, audio at 30 Hz, fingertip tactile at 200 Hz on one configuration |
Sources for this entry
- Open X-Embodiment: Pooled robot learning datasets in standardized formats.
- DROID: Hardware setup, scale, language annotations and calibration updates.
- BridgeData V2: Scale, sensors, control rate and license.
- RH20T: Multi-modal sensors, calibration records and subset licenses.
- Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware: Teleoperation setup, recording rates and demonstration counts.
- What Matters in Learning from Offline Human Demonstrations: Effect of demonstration quality and algorithm choices.
- Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey: The sim-to-real gap and approaches to it.
- Datasheets for Datasets: A template for documenting datasets.
- W3C PROV overview: A definition of provenance.
- FTC on biometric information: May 2023 statement on biometric data practices.
Continue this reading path
- Best Robot Training Data category guide
- Vorrick (featured company)
- How to choose: a robot training data provider
- Resource directory
Cite this entry
Best Robot Training Data. "Sensor modalities for robot training data." Content reviewed 8 November 2026. https://bestrobottrainingdata.com/sensor-modalities/