In early 2026, a team of robot learning researchers tried something that sounds almost backwards. Instead of adding more demonstrations to their training set, they scored every episode they already had by how much removing it would change the policy's performance on a held-out set of validation demonstrations. Then they trained on only the highest-scoring subset and threw the rest away.
The curated subset beat the full dataset. Not by a small margin that could be noise, but clearly enough that the paper built an entire method around it, called Quality over Quantity, using influence functions to estimate which episodes were actually teaching the policy something useful and which were quietly teaching it to be worse.
That result is uncomfortable for anyone who has ever pitched a data collection budget in terms of hours logged. It means a dataset can get larger and a policy can get worse at the same time, and the two facts are directly connected. This is the problem this guide is about: not how to run a collection program, which we cover in data collection for robotics and robotics data collection at scale, but what actually separates "high-quality" training data from data that merely exists.
The method behind that result is worth a moment, because it's a useful model for how to think about quality generally. Rather than scoring episodes by how they look, the researchers scored each one by how much it actually changed the policy's loss on a held-out set of validation demonstrations once removed, using influence functions borrowed from general machine learning and adapted specifically for robot data. Two adaptations mattered: taking the maximum influence an episode had across every validation sample rather than an average, since a single demonstration is often useful for one specific situation rather than all of them, and averaging influence scores across the state-action pairs within one trajectory to cut down noise. The detail matters less than the principle. Quality, in this framing, isn't a property you can see by watching an episode play back. It's a property you can only measure by checking what the episode actually does to the policy.
The short version
Robot training data collection only produces a working policy when the data clears four quality gates: structural integrity, annotation and label consistency, deliberate diversity coverage, and downstream validation against held-out performance. Hours collected and episode counts predict almost nothing about whether a policy will work outside the room it was recorded in. What predicts it is whether the data passed those four checks, and most programs only check the first one.
Two datasets, same hour count, very different policies
Picture two teams collecting demonstrations for the same pick-and-place task, same robot, same number of recorded hours. Team A ran every session with a fixed grasp strategy, logged hardware timestamps on every stream, and reviewed a sample of episodes against a written rubric. Team B let operators improvise their own approach, recorded whatever camera footage came out, and moved on once the robot "looked like it was working."
Both teams end the month with roughly the same number of hours. Team A's policy generalizes to new object positions. Team B's policy memorizes a handful of trajectories and falls apart the moment something shifts a few centimeters. Nothing about the model architecture differed. The difference lived entirely in the data, and specifically in dimensions that an hour count can't see: consistency, coverage, and whether anyone checked that the data actually taught the policy anything useful.
This is not a hypothetical. Research on operator consistency in imitation learning shows that inconsistent actions at the same state push policies into states they've never handled at test time, a problem the authors call action divergence. We go deeper on what that looks like in practice, including a real case where two operators' combined data taught a robot to hesitate, in human demonstrations for robot training.
The four quality gates
Every trainable episode, regardless of how it was collected, teleoperation, human video, or simulation, has to clear four checks before it earns a place in a training run. Treating them as sequential gates rather than a vague "quality culture" is what makes them actually enforceable.
Most collection programs only enforce the first gate. The episodes that make it through all four are usually a fraction of what was originally recorded, and that fraction is what actually trains a working policy.
Structural integrity
The basic, mechanical check: every expected field exists, array shapes match across episodes, camera and joint streams carry hardware timestamps rather than software ones, and no episode is silently truncated. This is the gate most programs actually run, often automatically, which is exactly why it catches the least interesting problems. Passing this gate means an episode is parseable. It says nothing about whether it's any good.
Annotation and label consistency
Does "grasp" mean the same moment to every operator and annotator who labels it? Measured consistency, not assumed consistency, is the point. That means comparing repeated demonstrations of the same task against each other, tracking inter-annotator agreement on phase boundaries and success labels, and flagging sessions where one operator's strategy quietly diverges from everyone else's. A dataset where five people describe the same action five slightly different ways isn't one dataset. It's five incompatible ones sharing a folder.
Deliberate diversity coverage
Coverage that happens on purpose, not coverage that happens to accumulate. DROID's published ablation found that co-training on a scene-diverse subset produced better out-of-distribution performance than co-training on a subset restricted to its 20 most frequent scenes, same platform and task family. We cover how to budget for this properly, across volume, embodiment, scene, and task-tail axes, in robotics data collection at scale. The short version for this gate: list the conditions the policy will meet at deployment, then check which ones your dataset is actually missing.
Downstream validation
The gate almost everyone skips, and the one the Quality over Quantity research above is really about. Instead of trusting that clean-looking data is good data, score episodes by how much they actually move performance on a held-out validation set, and treat anything that doesn't earn its place with suspicion regardless of how polished it looks on replay. This is also where a separate line of research gets interesting: a 2026 paper on deployment-time policy reliability found that selecting for "quality" in the wrong way can backfire, because demonstrations that look cleaner on the surface sometimes teach a policy a brittle strategy that fails under slightly different conditions at test time. Validation against real downstream behavior, not visual inspection, is the only way to catch that.
What people get wrong about "quality"
Clean-looking is not the same as high-quality
A smooth, well-lit, successful episode can still teach a policy a strategy that only works under the exact conditions it was recorded in. Visual polish is a weak proxy for whether the data actually generalizes.
Filtering too hard can remove the wrong things
Aggressively discarding anything that looks imperfect also discards recoveries and near-misses, which are often the highest-value signal in the dataset for teaching a policy how to handle things going slightly wrong.
Treating annotation like generic labeling
Robot demonstrations are continuous motion, not a bounding box on a still image. Action boundaries are genuinely ambiguous unless the person labeling them understands the task, and workflows built for image annotation don't transfer cleanly.
Assuming consistency instead of measuring it
Most programs never actually compare how differently two operators or annotators handle the same situation. By the time it shows up as a problem, it's already baked into thousands of episodes.
Why this matters more than it sounds
A 2026 survey on the data landscape for embodied manipulation points out that most existing robot learning datasets remain strongly biased toward successful expert demonstrations, and argues that a binary success-or-failure label is too coarse to capture why something went wrong or how it was recovered from. Several research datasets have started closing that gap directly: the REASSEMBLE assembly dataset logged 4,551 demonstrations, of which 4,035 succeeded, with the remainder kept and labeled rather than discarded. That's the practice worth copying, whatever scale you're operating at.
Reading the signal from the red flag
| Quality signal | What good looks like | Red flag to watch for |
|---|---|---|
| Timestamps | Hardware-sourced, aligned across every stream | Software-assigned timestamps that drift under load |
| Annotation consistency | Measured agreement across operators and annotators | Consistency assumed because everyone followed "the same instructions" |
| Diversity coverage | A written coverage matrix with under-filled cells tracked | Diversity that just happened to accumulate as sessions ran |
| Validation evidence | Episodes scored by their actual effect on held-out performance | Approval based on replay review looking clean |
| Failure retention | Failures and recoveries kept, labeled, and usable | Failed or corrected episodes quietly purged as noise |
Gates need owners, not just definitions
A checklist that nobody is accountable for tends to decay into a formality within a few weeks of a program ramping up. The programs that actually enforce these four gates assign each one to someone whose job is to say no, not just to someone who reviews a dashboard occasionally.
In practice that tends to look like a QA lead or senior annotator owning gate two, since consistency across people is fundamentally a people problem, not a tooling problem. It looks like whoever owns the task definition also owning gate three, since they're the only one who actually knows what the deployment conditions look like well enough to say a coverage matrix is complete. And it looks like gate four living with whoever runs training, since they're the only one positioned to run the held-out comparison that actually answers the question. Splitting ownership this way is less about process for its own sake and more about making sure each gate is checked by someone with the context to catch what it's actually there to catch.
What this means for a team building a collection program today
Start by picking the gate your current program is weakest on, because it is almost never all four at once. Most teams have reasonably solid structural checks already, since that part is easy to automate. Far fewer have ever measured annotation consistency with anything more rigorous than "the instructions were clear," and fewer still have run a validation pass that scores episodes by their actual effect on a held-out set rather than how they look on replay.
The uncomfortable implication of the Quality over Quantity result at the top of this piece is that a collection budget spent on volume without spending anything on these four checks can make a policy worse, not just fail to make it better. That reframes the conversation worth having before the next planning cycle: not "how many more hours do we need," but "which of these four gates have we actually enforced, and which have we just assumed." If you're scoping how to collect the data itself, data collection for robotics and robotics data collection at scale cover the operational side in depth. This is the part that decides whether what comes out of that process is worth training on at all. For the specific case of human-operated collection, teleoperation data collection and what actually ends up inside a usable episode, robot demonstration data, are the natural next reads.
