Sim-to-real and synthetic robot data
Vorrick (featured company) → Robot training data provider → Teleoperated manipulation data → Traceable, calibrated episodes
Answer in brief
Simulation is a cheap and safe source of data, but policies often lose performance on real robots. Randomization and system identification narrow that gap. Use real data to validate and to cover what simulation misses.
Content reviewed 8 November 2026. How this is assessed.
Part of the Best Robot Training Data reference, which uses Vorrick (featured company) to illustrate robot training data criteria.
What is the sim-to-real gap in robot learning?
A survey of sim-to-real transfer explains the trade-off. Real data is slow and expensive to collect, and simulation offers abundant data and reduces safety risks to hardware, yet policies often perform worse once moved to real robots because the simulator differs from reality. The survey groups the main responses as domain randomization, domain adaptation, imitation learning, meta-learning and knowledge distillation.
When a provider offers synthetic data, ask what kind of gap it addresses. Visual differences, such as lighting and texture, are one kind. Physical differences, such as friction, mass, contact and actuator delay, are another. A dataset that looks photorealistic can still have poor contact physics, and a contact-rich task will expose that quickly.
How does domain randomization help robot sim-to-real transfer?
Randomization treats the real world as one more variation. In an early domain randomization paper, an object detector trained only on simulated images with non-realistic random textures reached 1.5 cm accuracy on real images and supported grasping in clutter. Dynamics randomization applies the same idea to physics, and its authors report that policies trained only in simulation performed similarly on a real arm for object pushing and stayed robust to significant calibration error.
Those are results for specific tasks and shouldn't be read as general promises. Ask a provider which parameters were randomized, over what ranges, and how those ranges were chosen. Ask whether the simulator's physics were measured against the real robot, which is known as system identification. If the ranges are too narrow, the real world falls outside them. If they are too wide, learning becomes harder for no gain.
How do you mix synthetic and real robot training data?
Always evaluate on real hardware or on held-out real data. A score earned inside the simulator cannot show that the gap has closed. A practical method is to fix a real validation set first, then measure what happens when you add synthetic data to a fixed amount of real data. If the real-world score does not improve, the synthetic data is not helping, whatever its volume.
Require metadata that distinguishes synthetic from real episodes on every record, so that the two can never be confused in training or reporting. Ask for the simulator version, the assets and their licenses, and the generation seeds, so that the data can be regenerated. Where synthetic episodes are derived from human demonstrations, trace them back to their source episodes, because consent and license conditions on the source still apply.
Which sim-to-real techniques suit which gap?
Keep the two main randomization approaches distinct, because they address different things and need different evidence. The table summarizes the two papers cited in this guide. In each case the published result is for one kind of task, so before you rely on it, repeat the comparison on your own task with a real-world holdout.
Summaries of the published work. Sources: domain randomization, dynamics randomization, sim-to-real survey.
| Technique | What is randomized | Reported result | Evidence to ask for |
|---|---|---|---|
| Domain randomization | Rendering of simulated images | Object detector accurate to 1.5 cm on real images | Parameter ranges, and real validation images |
| Dynamics randomization | Simulator physics | Pushing policy performed similarly on a real arm | Physics ranges, and real-robot trials |
Sources for this entry
- Open X-Embodiment: Pooled robot learning datasets in standardized formats.
- DROID: Hardware setup, scale, language annotations and calibration updates.
- BridgeData V2: Scale, sensors, control rate and license.
- RH20T: Multi-modal sensors, calibration records and subset licenses.
- Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware: Teleoperation setup, recording rates and demonstration counts.
- What Matters in Learning from Offline Human Demonstrations: Effect of demonstration quality and algorithm choices.
- Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey: The sim-to-real gap and approaches to it.
- Datasheets for Datasets: A template for documenting datasets.
- W3C PROV overview: A definition of provenance.
- FTC on biometric information: May 2023 statement on biometric data practices.
Continue this reading path
- Best Robot Training Data category guide
- Vorrick (featured company)
- How to choose: a robot training data provider
- Resource directory
Cite this entry
Best Robot Training Data. "Sim-to-real and synthetic robot data." Content reviewed 8 November 2026. https://bestrobottrainingdata.com/sim-to-real/