Synthetic data scales dexterous robot policy pretraining efficiently, but simulators still approximate contact dynamics, friction, and object deformation. Real-world annotated data, including 6-DoF pose annotation, contact region labeling, and grasp taxonomy grounding, closes the sim-to-real gap that synthetic pipelines leave open before production deployment.
Synthetic data has become the default starting point for training dexterous manipulation policies. It is fast to generate, cheap compared to real-world data collection, and scales without the bottleneck of physical robot time. For ML teams building grasping and manipulation systems, this makes simulation an obvious first move, and for many stages of the pipeline it is the right one.
But dexterous manipulation is exactly the kind of task where synthetic data for dexterous manipulation runs into a hard limit. Contact-rich interactions involve friction, deformation, and force feedback that simulators still approximate rather than reproduce. This is the gap that iMerit’s dexterous manipulation annotation programs are built around, pairing 6-DoF pose, grasp taxonomy, and contact region annotation to supply the real-world grounding that simulation leaves out, particularly for policies headed toward production deployment rather than benchmark demos.
Where Synthetic Data for Dexterous Manipulation Works Well
Synthetic pipelines are genuinely useful for certain stages of policy development, and it’s worth being clear about where that value comes from before getting into where it breaks down.
Domain randomization lets teams generate thousands of object and lighting variations for perception training, at a fraction of the cost of physical data collection. A policy exposed to widely varied textures, colors, and camera angles in sim tends to generalize better once it sees the real world’s messier version of the same scene. Coarse policy initialization also benefits heavily from sim. A policy can learn rough reach and approach behaviors, basic collision avoidance, and gross motor sequencing in simulation before ever touching a real object.
Platforms like iMerit’s own synthetic data platform exist for exactly this kind of targeted generation, filling in edge cases and rare scenarios that would otherwise be expensive to capture live.
This is where synthetic data earns its reputation. Bulk pretraining, visual variation, and early-stage policy shaping do not require the sim to get contact physics exactly right. The sim just needs to be close enough to bootstrap learning that real-world fine-tuning can refine later. Teams that lean on synthetic data for these stages are making a reasonable tradeoff, not cutting a corner.
The Physical Properties Simulators Still Can't Model
Once a task becomes contact-rich, the gap widens quickly. A 2026 paper, Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation, describes the simulation-reality gap as the reason information-dense modalities like touch resist effective transfer, since existing methods often compress tactile data into coarse, low-dimensional features that sacrifice the detail complex manipulation actually needs.
Contact dynamics are a central part of this problem. That same paper introduces a physics-grounded contact representation specifically to reduce the tactile sim-to-real gap, aligning real sensor readings with the contact quantities simulators can represent, rather than assuming the simulator’s contact model is close enough on its own. A separate paper, Direction Matters: Learning Force Direction Enables Sim-to-Real Contact-Rich Manipulation, frames the same issue as a conflict between the target forces a policy needs and the unreliability of simulated force magnitudes, since simulators struggle to represent contact-rich physics with the fidelity real force sensing requires. Friction, in particular, tends to be simplified into a single coefficient in sim, when real surfaces behave inconsistently depending on wear, moisture, and material composition.
Deformable and fragile objects compound the problem further. A 2025 paper, Sim-to-Real Gentle Manipulation of Deformable and Fragile Objects with Stress-Guided Reinforcement Learning, notes that excessive stress during handling can cause irreversible damage, and trains a stress-penalized policy specifically to address this. Their approach reduced stress applied to fragile objects by 36.5 percent compared to a standard reinforcement learning baseline, a result that only shows up once real deformation behavior enters the loop. Simulated cloth, food, or soft packaging rarely deforms the way the physical version does under a gripper’s actual force profile.
Separately, the researchers behind SIM1, a deformable-object simulator, have argued that simulation is only valuable once its physics genuinely corresponds to real dynamics, since scaling synthetic data before that correspondence exists just scales the wrong signal faster. None of this means synthetic data is broken. It means contact dynamics, friction, and deformation are the categories where sim alone cannot carry a policy to production, no matter how much synthetic volume gets thrown at it.
What Real-World Annotated Data Adds Back
This is where annotated real-world datasets do work simulation cannot. Three annotation types matter most for dexterous manipulation policies moving toward deployment.
1. 6-DoF Pose Annotation
6-DoF pose annotation gives precise, ground-truth object and gripper state to a manipulation policy. That state comes from real physical interactions, not a simulator’s approximation of them. This grounding matters most exactly where sim struggles: contact-rich moments where small pose errors compound into failed grasps. This is the argument behind object-level labeling in manipulation datasets: a policy trained on accurately annotated real pose data learns the true relationship between approach angle and grasp success, instead of the simulator’s simplified version of that relationship.
2. Contact Region Labeling
Contact region labeling captures where and how a gripper actually contacts an object, including the contact dynamics that simulators simplify. Real annotated contact data reflects true surface friction and force distribution, not a physics engine’s estimate of it. For tasks involving irregular or unfamiliar object geometry, this labeling is often the only reliable source of ground truth on how contact actually unfolds.
3. Grasp Taxonomy
Grasp taxonomy grounds a policy’s grasp vocabulary in categories drawn from real human and robotic grasping behavior. The GRASP taxonomy, originally developed by Feix and colleagues in 2016, remains the reference framework most current grasp research builds on, and it’s the same standard behind how robots learn from human demonstration data. A 2025 paper called Dexonomy applied this taxonomy to synthesize dexterous grasps across all standard grasp types and reported an 82.3 percent success rate in real-world trials for its type-conditional generative model, evidence that taxonomy-grounded training transfers meaningfully to physical execution rather than staying a benchmark artifact.
Together, these three annotation types supply the real contact geometry, force behavior, and grasp structure that synthetic pipelines cannot generate on their own, no matter how sophisticated the simulator becomes.
Building a Hybrid Pipeline: Sim for Scale, Real Data for Grounding
The practical path for most teams is not choosing sim or real data. It is sequencing them correctly.
- Start with simulation for scale: Simulation still handles bulk pretraining and coarse policy shaping efficiently, closing the volume gap that real-world collection cannot match on its own. This stage should focus on the things sim is genuinely good at: visual variation, gross motor behavior, and early policy structure.
- Fine-tune on real, annotated contact data: Real-world annotated data then takes over for the contact-rich fine-tuning stage, closing the sim-to-real gap that pure synthetic pipelines leave open. This is where 6-DoF pose annotation, contact region labeling, and grasp taxonomy work stop being separate capabilities and start functioning as a single grounding layer, delivered through a domain-trained annotation workforce rather than generic crowd labeling.
- Evaluate against real deployment conditions, not benchmark sim: A policy that performs well in simulation but has never been evaluated against annotated real contact data is still an open question, not a finished result. Teams should treat real-world evaluation as a gating step before deployment, not an afterthought once sim metrics look good.
A policy pretrained in simulation and then fine-tuned against precisely annotated real interactions inherits the scale of synthetic data and the physical accuracy of real data, without needing either one to do the other’s job.
For teams evaluating this tradeoff, the question is not whether synthetic data is worth using. It clearly is, for the stages it fits. The question is where real-world annotation needs to enter the pipeline, and how precisely that annotation captures the contact dynamics and grasp behavior a production policy will actually encounter.
