Framing this as build or buy is what makes it hard. Annotation is not one job. It is five, and the teams who get this right make five separate decisions instead of one large one they later regret.
Month six of an in-house build
The decision was made in a fifteen-minute meeting. Four annotators, one contractor tool license, a spreadsheet for tracking. Cheaper than the vendor quote, and the data never leaves the building. Reasonable on the face of it.
By month six the shape of the thing has changed. One annotator left and the replacement is still ramping. Two ML engineers now spend roughly a day a week each reviewing labels and rewriting guidelines that keep failing on edge cases nobody anticipated. There is a running disagreement about whether a partially occluded object counts, and it has been resolved three different ways in three different batches. Nobody is measuring inter-annotator agreement because nobody owns that.
The team did not build an annotation function. They built an unmanaged operations department staffed by people whose job title says something else.
That story is common enough to be a genre, and the usual conclusion drawn from it is "outsource." That conclusion is half right, which is worse than being wrong, because the teams who outsource the whole thing hit a mirror-image failure about nine months later when the vendor's labels are consistent, delivered on time, and subtly wrong in a way that only someone who understands the model would catch.
The short version
Do not decide in-house versus outsourced as a single question. Annotation has five separable layers: guideline authorship, edge-case adjudication, production labeling, quality measurement, and tooling with data custody. Most successful teams keep layers one, two, and four internal and buy layer three, treating layer five as a negotiation. Deciding per layer gives you the cost structure of outsourcing without giving away the judgment that defines your dataset.
Why the binary framing fails
The stakes here are higher than the line item suggests. McKinsey research widely cited in the field puts data preparation and annotation at up to 80% of the time spent on AI projects, and Gartner has projected that through 2026 organizations will abandon 60% of AI projects that are not supported by AI-ready data. This is not a procurement decision. It is the majority of the project.
The binary framing fails because it bundles together things with completely different economics:
- Deciding what a label means is a judgment task that encodes your domain understanding. It does not scale with volume and it is genuinely proprietary.
- Applying labels at volume is a throughput task with a learning curve, management overhead, and turnover. It scales linearly with volume and is not proprietary at all.
- Knowing whether the labels are right is a measurement task that requires model context and should never be delegated to the party being measured.
Bundle them and you get one of two predictable failures. Keep everything in-house and your engineers become annotation managers. Outsource everything and you lose the ability to tell whether your training data is any good.
The annotation stack: five layers, five decisions
Layer three is the one with real economies of scale, and the only one most teams should be outsourcing by default. The layers above and below it are where your dataset's actual value lives.
Guideline authorship
What counts as an instance. What the class boundaries are. What "done" means for a single unit of work. This encodes your understanding of the model's failure modes and the deployment context, and it changes every time you learn something from an evaluation run.
Keep internal. A vendor can help draft, and good ones will push back hard on ambiguity, but the final authority belongs with whoever owns the model.
Edge-case adjudication
The occluded object. The ambiguous action boundary. The sample that fits two classes. Someone has to rule, and crucially, someone has to write the ruling back into the guidelines so it does not get re-litigated in the next batch.
Keep internal, with a hard SLA. Adjudication requests must be answered within a defined window or your vendor's throughput stalls and they start guessing.
Production labeling
Applying the guidelines at volume. This is where real economies of scale exist: a provider amortizes tooling, workforce training, and QA process across many clients, and can surge or shrink without you carrying fixed payroll.
Usually buy, unless data cannot legally leave your environment or the volume is low and highly iterative.
Quality measurement
Gold-standard sets, inter-annotator agreement, acceptance thresholds, and rejection criteria. The party producing the labels should not be the sole party grading them, for the same reason you do not let a supplier run your incoming inspection.
Keep internal. Maintain your own held-out gold set that the vendor never sees in advance, and re-cut it periodically.
Tooling and data custody
Which platform, who hosts it, where raw data lives, and what you get back. Negotiable in principle, but two terms are not: you own the raw data and all labels, and you can export in a standard, documented format without assistance.
Negotiate, but never concede export. Vendor lock-in via proprietary formats is the most common way teams lose optionality without noticing.
The cost math, done properly
The comparison people run is a vendor's per-unit rate against an annotator's salary. That comparison is wrong in a specific and consistent direction.
Industry analysis of fully loaded internal costs notes that a 60,000 or more once benefits, tools, management time, and overhead are counted, and dividing by realistic annual output often puts in-house labels at two to four times a managed vendor's rate. Other practitioner reporting puts a five-person internal annotation team at roughly 300,000 a year all in.
Here is the structure of a defensible comparison. Fill in your own numbers, but do not omit any rows.
| Cost line | In-house | Outsourced | Frequently omitted? |
|---|---|---|---|
| Direct labor / vendor invoice | Salaries plus benefits | Per-unit or per-hour rate | No |
| Management and coordination | Supervisor time, scheduling, performance | 10-20% added PM effort for comms and time zones | Yes, both sides |
| Tooling licenses | Per-seat platform cost, annual | Usually bundled, verify | Sometimes |
| QA infrastructure | Gold sets, agreement tracking, review tooling | Your own acceptance testing, still required | Yes, almost always |
| Ramp time | New annotators well below full output for weeks | Pilot batch plus calibration cycles | Yes |
| Turnover | Rehire plus re-ramp, recurring | Vendor absorbs it, verify continuity of team | Yes |
| Rework | Internal re-labeling cycles | Revision cycles that multiply effective unit cost | Yes, both sides |
| Engineering opportunity cost | ML engineer hours on review and guidelines | ML engineer hours on adjudication and acceptance | Yes, and it is usually the largest single line |
| What to actually compare | Cost per accepted unit, measured on a real pilot | Cost per accepted unit, measured on a real pilot | Yes |
The metric that settles most arguments
Compare cost per accepted unit, not cost per label. A cheaper rate that produces 30% rework is not cheaper. Run both options against the same gold set, count only units that pass acceptance, and divide total spend including your own engineering time by that number. Teams that do this arithmetic honestly usually stop arguing within a week.
The decision, per layer, by situation
| Your situation | Layer 3 recommendation | Why |
|---|---|---|
| Regulated or restricted data — PHI, defense, or contractual data-residency limits | In-house, or an on-premise partner arrangement | Compliance is a hard constraint. Where a vendor can work inside your environment under your controls, that is a middle path worth exploring. |
| Annotation judgment is your moat — specialist domain knowledge others cannot replicate | In-house for the judgment-heavy subset only | Split the work. The hard 15% stays internal; the routine 85% goes out with clear guidelines. |
| Low volume, fast iteration — guidelines change weekly, research stage | In-house | Communication overhead exceeds the labor saving when the spec is unstable. |
| Steady production volume — stable taxonomy, predictable throughput | Outsource | This is exactly where scale economics apply and where fixed internal headcount is hardest to justify. |
| Spiky or seasonal demand — bursts around model releases or data drops | Outsource, with a small internal core | Internal teams size to the peak and idle at the trough, or size to the trough and miss the peak. |
| Physical AI / robot demonstration data — temporal episodes, multi-sensor | Specialist partner, not a generalist BPO | Requires domain grounding to adjudicate action boundaries consistently. General image-labeling workflows do not transfer. |
Why physical AI data breaks the standard vendor model
Most annotation pricing and workflow assumes a static artifact: an image, a frame, a text span. You draw a box, the box is right or wrong, and a second annotator with the same guidelines will usually agree.
Robot demonstration data is not that. An episode is continuous motion in which the start, end, and transition points of an action are genuinely ambiguous unless the person reviewing understands both the task and the hardware. Ask five annotators when a "grasp" begins and you will get five answers, all defensible, and the resulting label noise is invisible in any per-unit quality metric.
Three practical consequences:
Per-unit pricing fits badly
An episode is not a comparable unit to another episode. A 40-second contact-rich insertion and a 6-second pick are priced identically under a per-episode model and cost wildly different amounts to review properly.
Agreement metrics need redefining
Inter-annotator agreement on a temporal boundary needs a tolerance window, not exact match. Vendors who cannot articulate their tolerance policy have not thought about this.
Reviewers need hardware context
Judging whether a trajectory was smooth or whether calibration drifted requires understanding the robot, not just watching the video. That is a materially different hiring profile.
Failures are assets, not defects
Standard annotation QA discards bad samples. In robot learning, failed and recovery episodes are among the most valuable data you have. A vendor whose process silently drops them is destroying value while hitting their SLA.
If your data is robot episodes rather than images, screen partners on those four points specifically. Our breakdown of annotation versus labeling covers why the distinction matters operationally, and the annotation services page lays out how episode-level quality is scored.
How to run a vendor evaluation that tells you something
- ✓Pilot on your hard subset. Send the ambiguous, occluded, badly lit, edge-case material. A clean sample tells you nothing you did not already know.
- ✓Pay for the pilot. Free pilots are staffed with the best people the vendor has and are not representative of production output.
- ✓Extrapolate from measured throughput, never from a rate card, and require the pilot to report units per hour by task type.
- ✓Ask how disagreements get resolved and who writes the ruling back into the guidelines. Vendors with no answer here will produce inconsistency at volume.
- ✓Require inter-annotator agreement reporting, with the metric and tolerance defined up front.
- ✓Hold back a gold set the vendor never sees and score every delivery against it independently.
- ✓Confirm rework terms in writing. Whether revisions are billable changes the effective rate more than the headline number does.
- ✓Require standard-format export and full data ownership, including raw data plus label provenance, testable during the pilot rather than promised for later.
- ✓Meet the actual team lead, not the account manager, and ask about annotator continuity across the engagement.
What most teams get wrong
Outsourcing the judgment with the labor
Handing over guideline authority feels efficient and costs you the ability to improve. The vendor optimizes for consistency against the spec you gave them, which is exactly what they should do, and nobody is left improving the spec.
Treating the guideline document as finished
Guidelines are a living artifact. If yours has not changed in three months and you are still finding label errors, adjudication rulings are not being written back and the same disputes are recurring.
No adjudication SLA
Teams commit their vendor to a delivery SLA and commit themselves to nothing. When edge-case questions sit unanswered for a week, annotators guess, and the guesses are inconsistent by construction.
Optimizing the wrong direction on cost
Chasing the lowest per-label rate reliably increases total cost through rework. The cheapest vendor and the cheapest dataset are rarely the same vendor.
