Autonomous vehicle fleets generate around 4 TB of raw sensor data per vehicle per day, but most of it captures routine driving that adds little value for improving perception models. Reviewing and annotating every mile is not feasible both technically and economically.
Triage addresses this problem by filtering, prioritizing, and routing data for further review, annotation, and potential use in model training. However, the way teams perform it influences which scenarios enter the learning pipeline and which failure modes the model learns.
Teams can make these decisions in real time, close to data collection, and even offline after storing the data and gaining more context. Each approach selects different events and exposes different edge cases, which can shape the training dataset, model coverage, blind spots, and ability to learn from new scenarios.
This article compares real-time vs. offline triage and examines how each approach shapes what an AV model learns.
Why Triage Matters in Autonomous Vehicle Learning Pipelines
Triage is the data-selection layer that identifies which sensor logs or segments warrant further review, annotation, analysis, or inclusion in training and evaluation datasets. It evaluates, scores, and routes logs based on quality, novelty, and training value before annotation and model updates.
Without autonomous vehicle triage, AV teams face two compounding problems:
- Wasted annotation spend: Most fleet data is routine driving that adds little to model improvement.
- Missed edge cases: Rare but safety-critical scenarios (e.g., occluded pedestrians, unusual interactions) are drowned out by high-volume, low-value miles.
Triage addresses this by prioritizing data based on signals such as sensor quality metrics (blur, dropout, misalignment), scenario rarity (animals on road, occluded pedestrians), or scene complexity and actor diversity. These signals can support different levels of analysis depending on whether the goal is to filter large datasets, investigate a specific scenario, or understand an underlying model failure. Depending on the purpose of the review, teams can apply them across three tiers:
- Operational triage: High-volume filtering, deduplication, and basic QA
- Technical triage: Scenario investigation and perception failure analysis
- Engineering triage: Root-cause analysis and regression validation
These levels also connect to the architecture used to process the data. Together, they determine which AV data moves forward into review and training, ultimately influencing the distribution of data an autonomous vehicle perception model learns from.
How Real-Time Triage Shapes Training Data
Real-time triage evaluates sensor and vehicle data on-board or at the edge within seconds or minutes of capture. Lightweight rules, thresholds, classifiers, or anomaly detectors can flag events such as hard braking, lane deviations, near misses, low-confidence detections, and sensor faults for further review and edge case triage. This makes real-time AV data triage effective for identifying known and immediately observable failure signals while a vehicle is still operating.
Tesla’s shadow mode shows this well. The network makes predictions continuously in the background, and any divergence from actual driver behaviour is flagged against more than 200 purpose-built triggers. Each trigger targets a specific potential failure mode.
The main value of real-time triage is its ability to make selection decisions close to the point of capture, using immediately available signals. It can prioritize data while the vehicle is still in operation, making it particularly useful for operational triage, where teams need to filter high volumes of data, identify clear quality issues, and flag events for further review. It also supports active-learning workflows that feed selected events into near-term model improvement. Real-time triage is particularly useful for:
- Capturing high-severity or clearly detectable events during fleet operation
- Prioritizing known failure modes and safety-critical scenarios
- Providing immediate candidate data for review and model feedback
- Supporting active-learning workflows where selected events can inform near-term model updates
However, this architecture can also introduce selection bias. Decisions made with limited context can overlook subtle failures that emerge across longer sequences, multiple drives, or broader scenario comparisons. Hence, real-time triage can concentrate the training distribution around detectable events while leaving less obvious edge cases underrepresented.
How Offline Triage Expands Dataset Diversity
Offline triage analyzes stored driving logs after collection. The team gets to examine data with more compute, longer temporal context, and information from other parts of the fleet. This allows them to find patterns that real-time systems may miss because they only see a limited snapshot during capture.
It can also identify duplicate or highly similar samples, while uncertainty scoring and active learning can surface examples that add new information to the training set. Teams can then organize these events using scenario ontologies aligned with SOTIF (ISO 21448). This helps them look beyond individual anomalies and identify gaps in coverage across known and potentially hazardous scenarios.
This broader analysis can expose long-tail scenarios, recurring localization drift, calibration issues, weather-related failures, and rare pedestrian or cyclist interactions. It can also reveal issues that only become clear when teams compare data across multiple drives. This makes offline triage particularly useful for technical and engineering triage, where teams can investigate specific perception failures, compare patterns across the fleet, and trace recurring issues toward their underlying causes.
Recent research provides evidence for the value of this type of offline data curation. For instance, this paper found that curating offline data through uncertainty-based sampling reduces the collision rate from 16.0% to 5.5% in Waymax-based evaluation. However, the trade-off with offline triage is infrastructure. Retaining and processing more raw data increases storage, data movement, compute, and governance requirements.
Real-Time vs. Offline Triage: How Each Architecture Shapes Model Learning
Autonomous vehicle perception models never learn from everything a vehicle experiences. They only learn from data that survives triage, gets reviewed, annotated, and incorporated into future training datasets. This makes autonomous vehicle triage architecture part of the learning pipeline, not just a data-management decision.
However, the key difference is the selection boundary each architecture creates. Real-time triage makes selection decisions using information available close to capture. Offline triage can revisit the same data later with additional context, updated models, and comparisons across larger datasets. These boundaries can lead to shifts in the training distribution over successive training cycles.
| Learning factor | Real-Time Triage | Offline Triage |
|---|---|---|
| What gets selected | Data matching predefined rules, triggers, or lightweight model signals | Data selected using richer models, historical context, and broader scenario analysis |
| Rare and unexpected scenarios | May be missed if they do not trigger an existing rule | Can be discovered retrospectively as new failure patterns emerge |
| Model failure coverage | Often reflects what the current model already knows how to detect | Can expose failures using newer models or analysis applied to historical data |
| Training data distribution | Can become concentrated around known triggers and recognizable events | Can be deliberately reshaped around gaps, failures, diversity, and underrepresented scenarios |
| Feedback loop | Current model influences which new data is retained | Model improvements can be used to re-examine previously collected data |
| Risk to learning | Valuable examples may be discarded before their training value is understood | More storage and compute are required to preserve and analyze broader data |
| Main learning advantage | Captures known or detectable failures while they are occurring | Expands the training set with newly discovered, difficult, or underrepresented examples |
These differences matter beyond data processing. They influence which failure modes are represented in training data, how well the pipeline covers rare scenarios, and how data selection changes as the model is retrained. Three factors are particularly important:
Selection Bias
Real-time triggers can concentrate training data around known failure modes while missing events that do not produce an obvious signal. Offline analysis can revisit historical data with newer models and broader fleet-level comparisons to identify these gaps. This supports more informative sample selection when annotation resources are limited.
Long-Tail Coverage
More driving data does not automatically mean better coverage. Rare events can remain scarce even in large datasets. Offline analysis can help find these gaps by searching historical data for underrepresented scenarios that real-time triggers did not select.
Model Feedback Loops
Data selection also affects what the model can detect in the next learning cycle, creating a feedback loop. The current model influences which events are selected. Those examples shape retraining for the next model version, which, in turn, determines the next set of failures, and therefore, the next set of events for retraining. Without deliberate offline review, this loop can reinforce failures the model already knows how to detect while leaving less visible blind spots underrepresented.
Why a Hybrid Triage Architecture Can Be More Effective
Real-time vs. offline triage address different parts of the data-selection problem, so combining them can create a more complete learning loop.
The value of this approach is not simply that one method is faster and the other is more thorough. They expose different parts of the data distribution. Real-time selection can capture known or clearly detectable failures, while offline mining can deliberately search for underrepresented scenarios, recurring weaknesses, and new failure patterns.
IEEE’s Autonomous Driving Working Group describes this same split at the architecture level. Data selection comprises an on-vehicle trigger mode for capturing data economically at the moment of an event and a cloud-side database mode for mining stored data more broadly. They both feed into the same closed loop of screening, annotation, training, and deployment.
Together, these capabilities allow selected events to move through validation, annotation, training, and evaluation before the results inform the next data-selection cycle. Model evaluation can reveal gaps in scenario coverage, which can then be reflected in event triggers, scenario definitions, and search criteria. The pipeline therefore becomes iterative, with each training cycle providing evidence for what data should be prioritized next.
Human Review: The Validation Layer Between Triage and Training
Triage determines which events deserve attention, but selection alone does not make an event useful training data. Human review is needed to determine whether a flagged scenario represents a genuine model failure, a sensor issue, an annotation problem, or a previously unseen case. This is particularly important when camera, LiDAR, radar, and vehicle-state signals do not agree or when the correct interpretation depends on context that automated scoring cannot capture.
For AV teams, expert review from domain experts also provides a mechanism for refining taxonomies and annotation rules as new failure modes emerge. The validated results can then feed training datasets, evaluation sets, and edge-case libraries rather than remaining isolated review outcomes.
This review can be managed through structured annotation and QA workflows, such as Ango Hub, which supports task routing, reviewer validation, benchmarks, and quality checks across AI data workflows.
iMerit’s human-in-the-loop workflows support multimodal review, annotation, and quality assurance for fleet-scale AV sensor data. This approach has also delivered measurable efficiency gains. One robotaxi company working with iMerit achieved a 250% improvement in annotation efficiency by directing resources toward high-value data exceptions. iMerit case study
Best Practices for Building a Continuous Learning Triage Pipeline
A continuous learning pipeline should treat triage as an evolving part of model development rather than a one-time filtering step. The goal is not to process the most data, but to consistently identify the data that can improve model performance and coverage. Key practices include:
- Define clear selection criteria: Connect triggers to measurable safety, quality, and learning objectives.
- Combine real-time and offline analysis: Use immediate alerts for known issues and retrospective analysis for emerging patterns.
- Measure training value: Track whether selected samples improve relevant model metrics after retraining.
- Refine sampling strategies: Update rules and scoring methods as new failure modes emerge.
- Prioritize annotation quality: High-quality labels are more useful than simply increasing annotation volume.
- Track edge-case coverage: Monitor representation across weather, geography, road types, traffic conditions, and vulnerable road users.
- Close the feedback loop: Compare post-deployment behavior with previous failures and use the results to guide the next triage cycle.
This creates a learning pipeline that continuously selects, validates, learns from, and reassesses its data rather than repeatedly training on the same types of examples.
Conclusion
The choice between real-time vs. offline triage is not about speed, it is about which parts of the driving distribution a team is willing to leave unselected. Neither architecture is wrong; neither is complete on its own. Closing that gap is what expert-led triage is for.
Key Takeaways
- Triage architecture directly influences training data distribution and model coverage.
- Real-time triage captures immediate, detectable learning signals but can miss subtle or emerging failures.
- Offline triage provides broader context for identifying long-tail scenarios and dataset gaps.
- Human review validates selected events before they become training signals.
- Hybrid pipelines can combine operational responsiveness with broader dataset discovery.
iMerit’s human-in-the-loop triage services help autonomous vehicle teams turn raw event streams into validated, high-value training data. Talk to an iMerit expert to strengthen your AV learning pipeline with structured triage and human review.
