The teams that struggle with robot data almost never have too little of it. They have a great deal of it, collected along one axis, and a policy that still falls over the moment the lighting changes.
A conversation that happens constantly
An applied ML lead sends over an evaluation report. Their team spent seven months collecting roughly 40,000 teleoperated episodes on a bimanual rig. Success rate on the held-out set from the same lab: 94%. Success rate in the customer's facility: 41%.
The first instinct is always the same. Collect more. The second month of collecting more produces a policy at 44%, and now there is a budget conversation.
What actually happened is visible in the metadata, if anyone had been recording it. Nearly all 40,000 episodes came from two tables, in one room, under one lighting rig, from four operators, on a single robot. The dataset was enormous along one axis and roughly one episode deep along the four that mattered.
Scaling robot data collection is not a volume problem. It is a variance problem wearing a volume problem's clothes, and almost every operational decision follows from getting that distinction right.
This is a working guide: what scale actually means, how to budget for it, what the 2026 tool stack looks like, what it costs, and the specific failure modes that appear as you move from a hundred episodes to a hundred thousand.
The short version
Robotics data collection at scale is the operational practice of capturing large volumes of robot demonstration and sensor data with enough diversity, consistency, and metadata integrity to train and evaluate policies that work outside the room they were collected in. The hard parts are not recording. They are time synchronization across streams, calibration integrity over months, operator consistency across shifts, format standardization across platforms, and deliberate coverage of scene and task variation.
Scale has four axes, and you can only saturate one at a time
Volume is the axis everyone measures because it is the only one that is easy to count. It is also the one with the fastest-diminishing returns.
The most useful public evidence on this comes from DROID. Where the aggregated Open X-Embodiment dataset pooled a very large number of trajectories but drew from roughly 300 scenes, DROID deliberately optimized for environmental variety, collecting across 564 scenes in 52 real buildings on three continents. The ablation is the interesting part: co-training on a scene-diverse subset produced better out-of-distribution performance than co-training on a subset restricted to the 20 most frequent scenes. Same platform, same task family, different variance profile, materially different generalization.
That result reframes the planning question. You do not have an episode budget. You have a diversity budget, and episodes are the currency you spend to buy coverage on four separate axes.
Episode count is a single number hiding a four-dimensional allocation decision. Most collection programs are shaped like the left figure and are surprised by the evaluation gap.
The diversity budget
Before a single episode is recorded, decide how the budget splits across the four axes. The right split depends on what the policy has to do, and getting the split explicit on paper is worth more than any tooling decision you will make afterwards.
Volume: repetitions per condition
How many times the same task is performed under materially identical conditions. This buys statistical stability and smooths operator noise. It is also the axis that saturates fastest.
Signal that you have over-spent here: held-out success is high, but the held-out set was randomly sampled from the same sessions as training.
Embodiment: hardware coverage
Different arms, grippers, camera placements, and mounting geometries. Expensive to buy and the axis most often skipped, because it requires owning or accessing more than one rig.
Spend here when: the policy must ship on hardware that differs from the collection rig, or when you intend to reuse the dataset across products.
Scene: environment and condition variation
Lighting, backgrounds, table heights, distractor objects, clutter density, surface materials. Cheap relative to its impact, and the axis with the strongest published evidence behind it.
Spend here when: the deployment environment is not the collection environment, which is nearly always.
Task tail: rare and recovery cases
The awkward grasps, the partially occluded part, the recovery after a slip. Deliberately collecting failure and recovery data is one of the highest-leverage choices available, and competition results in industrial assembly have specifically identified recovery-from-failure curriculum data as a critical factor in deployment success.
Spend here when: the cost of a rare failure in production exceeds the cost of staging it during collection. That is most industrial and clinical settings.
Practical allocation heuristic
For a first production policy on a fixed platform, a defensible starting split is roughly 40% volume, 35% scene, 20% task tail, 5% embodiment. For a dataset intended to outlive the current hardware, shift 15 to 20 points from volume into embodiment. Write the split down, then check the actual distribution monthly. Collection programs drift toward whatever is easiest to record, and the drift is always toward volume.
The 2026 tool stack
The tooling situation has improved substantially in two years. The most consequential change is that the field converged on a small set of formats, which removed a real tax on every new platform integration.
| Layer | What it does | Common choices in 2026 | What to check |
|---|---|---|---|
| Capture middleware | Drives the robot from operator input and logs state, action, and camera streams | ROS 2, LeRobot, vendor SDKs | Logging rate and whether the pipeline drops frames under load |
| Teleoperation input | Human control surface: leader-follower arms, VR controllers, 3D mice, haptic devices | Leader-follower rigs now available under $2,000, VR headsets, handheld UMI-style grippers | Latency budget and whether the input space matches the deployed action space |
| Episode format | How a demonstration is serialized for training | HDF5 per episode, RLDS, LeRobot dataset format; ROS 2 bags at capture time | Hardware timestamps on every stream, not software-assigned ones |
| Quality and review | Scores episodes, flags failures, routes borderline cases to human review | Replay-and-annotation pipelines with semi-automated scoring | Whether failures are retained and labeled rather than discarded |
| Storage and versioning | Holds the dataset and tracks what changed between training runs | Object storage plus dataset versioning; chunked formats such as Zarr for cloud-native training | Whether you can reproduce the exact dataset behind any past checkpoint |
| Simulation | Generates volume and rare scenarios; hardens policies before hardware | Isaac Sim and Isaac Lab, MuJoCo for cross-checking contact physics | Physics validity of assets, not visual fidelity |
One nuance on the simulation row. Agility Robotics runs the same control policy through a separate MuJoCo pipeline specifically because differences in contact physics between simulators expose corner cases and harden the policy. Cross-simulator validation is a cheap practice that catches a class of problem single-simulator pipelines never see.
What it actually costs
~$118 / hour
Fully loaded teleoperation cost, March 2026 benchmark, standard pick-and-place with wrist camera and external RGBD, per SVRC State of Robotics
40–60%
Reduction in annotation labor from mature replay-and-scoring pipelines versus 2024 workflows
< $2,000
Entry cost of leader-follower teleoperation rigs, which is what made multi-site collection economically viable
Three forces compressed that cost in parallel: cheaper and more ergonomic teleoperation hardware, replay pipelines that can ingest raw operator streams and emit formatted episodes with semi-automated quality scoring, and standardization around a small set of episode formats that cut integration work per platform.
What that hourly figure does not include, and what teams consistently underestimate:
Operator qualification and retention
A qualified teleoperator takes weeks to reach consistent output. Turnover resets that clock and introduces a distribution shift in your dataset that nothing in the pipeline will flag.
Calibration maintenance
Multi-camera rigs drift. Somebody has to re-calibrate on a schedule, log the result, and correlate drift events against batch boundaries. This is recurring labor, not a setup cost.
Scene resets
On contact-rich tasks the reset between episodes can take longer than the episode. Reset time is real cost and rarely appears in per-hour quotes.
Storage growth
Multi-camera episodes at high logging rates scale storage faster than most budgets assume, and retention policy decisions made in month two are expensive to reverse in month twelve.
Five things that break as you scale
These appear in a fairly predictable order. The first two are the ones that destroy data irrecoverably.
- Time synchronization, at roughly the second rig. Camera frames, joint states, force readings, and operator inputs get logged by different subsystems with different clocks. Without hardware timestamps on every stream, you cannot establish which action produced which outcome, and the episode is untrainable regardless of how good the demonstration was. Purpose-built capture stacks log at high rates specifically to keep every joint move and camera frame aligned; verify this rather than assuming it.
- Calibration drift, at roughly month three. Mounts shift under vibration, cameras get bumped, a lens gets cleaned and re-seated slightly differently. Without a logged calibration baseline at capture time, a batch of otherwise-good demonstrations silently degrades any model trained on it, and post-hoc diagnosis is close to impossible.
- Operator variance, at roughly the fifth operator. Different people segment the same task differently, apply different force profiles, and disagree on when an episode has succeeded. This shows up as label noise rather than as an obvious defect. Long uncapped sessions make it worse, which is why dedicated teleoperation operations cap session lengths.
- Format and integration tax, at the second hardware platform. Every new embodiment brings a different action space, joint ordering, and camera configuration. Teams that standardize on a common episode schema early pay this once. Teams that write a bespoke converter per platform pay it forever, and their dataset fragments into incompatible islands.
- License and provenance, at the first commercial training run. Public datasets ship under mixed licenses, and near-identically named releases can carry opposite terms, with some permitting commercial use and lookalike variants restricted to non-commercial use. Community mirrors sometimes relicense as well. Dataset landscape analysis flags this as a genuine trap: a name match is not a license match. Track lineage per dataset before it enters a training mixture, not after legal asks.
The one that costs the most to fix
Of these five, unlogged calibration is the most expensive, because it is the only one where the data looks fine. Sync failures produce obviously broken episodes. Operator variance shows up as noise. Calibration drift produces episodes that pass every visual inspection and quietly poison a training run. Log calibration state as episode metadata from day one, even if nothing consumes it yet.
Best practices that survive contact with a real program
- ✓Write the diversity budget before the first session and audit the actual distribution monthly, because programs drift toward whatever is easiest to record
- ✓Hardware timestamps on every stream, with per-episode metadata covering task, operator ID, rig ID, calibration state, environment conditions, and success label
- ✓Keep failures. Failed grasps, recoveries, and near-misses are the highest-value examples you will collect and the first thing most retention policies purge
- ✓Qualify operators before their data enters the main set, using a fixed qualification task scored against the same rubric applied to production episodes
- ✓Cap session length to protect consistency; fatigue produces corrective motions and drift that are invisible unless you are measuring for them
- ✓Build the held-out set from different sessions, operators, and scenes, not from a random split of the training pool, or your evaluation number is measuring memorization
- ✓Standardize on one episode schema early and write converters into it rather than between platforms
- ✓Version datasets against training runs so any past checkpoint can be traced to the exact data behind it
- ✓Run a scoped pilot batch first on one or two tasks to validate protocol, rig setup, and quality bar before committing to weekly volumes
Where synthetic data fits, honestly
Simulation is genuinely useful and routinely oversold. It scales volume cheaply, covers dangerous or rare scenarios safely, and lets you harden a policy before it touches hardware. Structured programs increasingly train in simulation first and validate on hardware second.
The limits are specific. Transfer depends on physics validity rather than visual fidelity, and a CAD-derived asset can render beautifully while carrying wrong mass distribution, missing joint limits, or collision geometry approximated as a bounding box. A grasp policy trained against that asset learns behavior that does not survive contact with the real part. Appearance-level domain randomization helps with lighting and texture gaps; it does nothing for wrong contact dynamics.
The productive framing is that simulation scales the axes you have already characterized, and real demonstrations are what characterize them. Programs that use only one of the two consistently underperform programs that use both. Our post on data collection for robotics goes deeper on how that loop compounds.
What most teams get wrong
Measuring the wrong evaluation number
A random split of the collection pool measures how well the policy memorized your room. Held-out sets must differ by session, operator, and scene to mean anything about deployment.
Treating it like generic annotation work
A robot episode is continuous motion where action boundaries are ambiguous unless the person reviewing understands the task. Workflows built for image bounding boxes do not transfer cleanly.
Deferring the tail
Easy demonstrations get collected first because they are easy. The policy then ships and fails on precisely the rare cases nobody prioritized. Tail coverage is cheaper to plan than to retrofit.
Counting episodes as the KPI
Episode count is a vanity metric. Useful program KPIs are scenes covered, operators qualified, tail cases staged, and evaluation delta per thousand episodes on a proper held-out set.
