A manipulation team we advised wired up four capture streams for a single insertion task: a wrist camera, a depth sensor mounted on the gripper, joint encoders on the arm, and a microphone for the operator narrating what they were doing and why. On review day, every clip looked clean. The peg slid into the hole smoothly, over and over, exactly like it should.
The policy trained on that data jammed at the exact moment of contact. Every single time.
Nothing about the footage was visibly wrong. The problem was the clock. Each stream had been timestamped by the software that received it, not by the physical moment it happened, and the small, inconsistent delays between a camera buffer, a sensor driver, and a joint readout had quietly rewritten the story of which action caused which outcome. The data looked synchronized. It was not.
That gap, between footage that looks aligned and data that actually is, is the real subject of multimodal data collection for physical AI. Not how many sensor types you can bolt onto a rig. Whether what they captured can still be trusted once you try to train on all of it at once.
The short version
Multimodal data collection for physical AI means capturing camera, depth or LiDAR, motion, and language streams as one time-aligned record of a task, not as separate logs that happen to share a folder. Most programs fail quietly at one of five points: the clock, the calibration, the data format, the language, or the final quality check. Fixing collection hardware without fixing those five is the single most common reason a multimodal dataset looks complete and still trains a broken policy.
Multimodal is not a label for "lots of sensors"
Search for "multimodal data collection" and most of what comes back is generic. Glossary entries define multimodal data as information made of several modalities, and vendor pages list text, image, audio, and sensor data as the ingredients, then stop there. That framing is not wrong. It is just not the part of the job that actually determines whether a physical AI model can learn anything useful.
In physical AI specifically, "multimodal" almost always means some combination of three things covered elsewhere in depth: individual sensor streams like depth, force, and LiDAR, which we walk through in sensor data collection for robotics; video and motion capture, covered fully in how physical AI processes video and motion data; and language, the instructions and annotations that tell a model what a task was meant to accomplish. This piece is not about any one of those in depth. It is about the problem that only shows up once you try to combine them: making sure a frame, a joint angle, a depth reading, and an instruction that were captured within the same few hundred milliseconds of each other are actually treated as describing the same moment.
That alignment work is the operational core of what we mean when we talk about a multimodal data collection program, as distinct from just owning a lot of sensors.
The alignment stack: five places multimodal capture actually breaks
Getting several streams into one trainable dataset is not one problem. It is five, and they happen in a fixed order during the life of a capture program. Miss one near the start, and every later stage inherits the error silently, because nothing downstream has a way to tell that the input it received was already wrong.
Each stage can look clean on its own. A dataset only earns the word "multimodal" once all five agree with each other.
Temporal synchronization
One research write-up on this specifically, framing it as structured label noise, argues that most collection systems do not record an observation and the action taken at the same instant, they record them offset by small, variable, state-dependent delays, and that the size of that offset grows with how fast the end effector is moving and changes sharply at the moment of contact. The same source claims common rigs like ALOHA, UMI, and plain USB cameras carry roughly 20 to 80 milliseconds of unmeasured jitter between a camera frame and a joint-state readback, though it presents this as its own analysis rather than an audited benchmark. A separate practitioner guide recommends keeping cross-modality discrepancy under 20 milliseconds, verified during pilot testing, and running cameras in hardware-triggered sync mode rather than trusting software timestamps alone. Hardware vendors have taken the hint: Orbbec's Dual-Ego rig, shown at ROSCon JP 2026, advertises synchronization accuracy within 1 millisecond, a figure worth treating as a vendor claim until independently verified rather than a settled spec.
Spatial calibration
Two sensors that agree on when something happened still need to agree on where. The ForEnt quadruped dataset calibrates camera and LiDAR extrinsics with the open Kalibr toolbox, then validates the result by projecting LiDAR points back onto RGB-D frames and checking the overlap. Other dataset teams converge on similar discipline from different angles: DurLAR solves for the rigid transform between LiDAR and camera using ArUco markers and RANSAC, while the STONE off-road dataset expresses every sensor's position relative to one shared reference frame so nothing drifts into its own private coordinate system. A calibration done once at setup is not a calibration done forever, which is the same lesson we cover for single-sensor rigs in sensor fusion in physical AI.
Format and rate normalization
Cameras, IMUs, force sensors, and encoders each run at their own native rate, and forcing them onto one shared clock has a real cost. The DAM-VLA paper argues that a single synchronous clock oversamples the slow modalities, undersamples the fast ones, and caps the whole system's action generation at its slowest effective frequency, proposing instead that each modality keep its own buffer refreshed at its own native rate rather than being forced to a shared tick. Simpler collection pipelines take the more common shortcut: the TOMD off-road dataset, for instance, just resamples everything down to 10 hertz using its LiDAR as the master clock. Both are legitimate choices. The mistake is not picking one on purpose.
Semantic and language alignment
Language is a modality too, and it goes stale the same way a miscalibrated camera does. A 2026 analysis of OpenVLA found its language generalization was held back by the Bridge Dataset V2's instructions, which often skip natural phrasing entirely, and proposed using an LLM to generate many general-purpose rewordings of the same trajectories after the fact. A separate project, FineVLA, makes a related point from the other direction: most robot datasets pair a trajectory with only coarse, goal-level language, leaving out exactly the detail, like which arm is active or where contact should happen, that a policy actually needs. FineVLA reports unifying 972,247 trajectories across ten open datasets and hand-verifying 47,159 fine-grained examples to close that gap, though its code and data were listed as forthcoming at the time of writing. Either way, language captured as an afterthought carries less signal than language captured as deliberately as a sensor reading.
Quality gating before training
The first four stages can each pass inspection individually and the combination can still be broken. ForEnt's team handled this by having operators perform a deliberate sit-to-stand motion at the start and end of every recording session, visible to every sensor at once, specifically so reviewers could check temporal alignment after the fact rather than assuming it held. One buyer-side guide to multimodal collection puts the stakes plainly: misaligned examples teach a model the wrong correspondence between inputs, and no amount of additional volume fixes that once it is baked in. Gating has to happen before an episode is marked usable, not months later when a policy's behavior is already hard to explain.
One clock on the whiteboard, four clocks in the rig
Teams plan for a single shared timestamp and then wire up a camera, a depth sensor, and an encoder that each timestamp on arrival rather than on event. The plan was synchronized. The hardware never was, and nobody measured the gap until the model started failing in a specific, repeatable way.
Treating language as metadata instead of a stream
Instructions get typed in after the session, loosely, by whoever is free. Sensor data gets the calibration rigor. Language gets a text box. The research on instruction-following models says this ordering is backwards, not just sloppy.
What we see in the field
We treat clock strategy, calibration baseline, and action or language schema as one checklist a program has to sign off before an operator sits down for the first session, not a cleanup pass after a few hundred hours have already been recorded. Retrofitting alignment onto data you already collected almost always costs more engineering time than deciding it up front would have, and it rarely recovers everything the original gap already cost you.
If any of this sounds like the same discipline that shows up around any single sensor, it is. The difference with multimodal capture is that you are now managing four or five of these problems at once, and they compound. A camera that drifts 10 milliseconds and a depth sensor that drifts 15 milliseconds in the opposite direction do not average out to something fine. They stack into exactly the kind of structured, state-dependent error that is hardest to catch by eye.
Matching the symptom to the broken layer
When a multimodal policy misbehaves, the fix is rarely "collect more data." It is almost always "find which of the five stages above actually failed."
| Symptom you see in training or evaluation | Layer that is actually broken | Where to look first |
|---|---|---|
| Policy looks perfect in review clips, fails at the exact moment of contact | Temporal synchronization | Hardware trigger vs. software-stamped timestamps |
| Depth or point-cloud overlay looks shifted, and worse after weeks of use | Spatial calibration | Re-run extrinsics, log a fresh baseline per device |
| Model seems to only react to the fastest-updating sensor | Format and rate normalization | Per-modality buffering instead of one forced shared tick |
| Model ignores spoken or written instructions when the scene alone predicts the action | Semantic and language alignment | Instruction diversity and grounding, not just presence |
| Some sessions train well and others quietly hurt performance, with no clear pattern | Quality gating | Alignment checks run at capture time, not after the fact |
What this means for a team starting today
If you are deep in getting one modality right, that is the right place to be. Our guides to sensor data collection for robotics and how physical AI processes video and motion data go further into each of those individually. This piece exists for the moment after that, when those streams have to agree with each other inside a single dataset.
Three things are worth deciding before a multimodal capture session is scheduled, not after. Pick a reference clock and a synchronization method, hardware trigger or software timestamp, and know its measured jitter on your actual rig rather than its spec sheet number. Decide on one action and language schema before the first operator starts, because three teams describing "grasp" three different ways do not produce one dataset, they produce three incompatible ones sharing a folder. And build the alignment check into the capture session itself, the way ForEnt's sit-to-stand motion did, instead of discovering the gap once a policy is already failing in a hard-to-explain way.
The teams getting this right are not the ones with the most sensor types bolted onto a rig. They are the ones who treat synchronization, calibration, format, and language as one connected discipline rather than four separate afterthoughts. That is exactly the gap our multimodal data collection team is built to close, whether that means designing the capture rig, running the alignment checks, or auditing a dataset you already have before you train on it again.
