Autonomous vehicle (AV) data pipelines have traditionally used uncertainty signals, edge-case detection, and human review to decide which driving scenarios require annotation or investigation. Directed triage extends this workflow by using AI to identify model weaknesses and request targeted scenarios, while human experts validate and act on those requests.
The shift toward directed triage becomes more important as autonomous fleets scale. While larger fleets create more opportunities to capture valuable scenarios, simply increasing road mileage is insufficient to gather the exact evidence needed to fix known model weaknesses.
So the question for autonomous vehicle data triage is shifting to “What evidence does the model need next, is that request valid, and what should we do with the resulting data?” from “Which scenarios should we label?”
In this article, we’ll explore how directed triage changes autonomous vehicle data collection, why human judgment remains vital, and the capabilities AV teams need to scale this workflow.
Why Autonomous Vehicle Data Triage Is Shifting from Reactive to Directed
Reactive triage starts with what the fleet has already captured. Teams inspect disengagements, interventions, unusual trajectories, and sensor anomalies, then decide which events should enter the autonomous vehicle data annotation pipeline or engineering review.
That approach is effective for surfacing high-value events from existing data, but it is constrained by what the fleet has already encountered. Scenario-mining pipelines can search for environments such as unprotected left turns or tunnels using perception data and HD-map context, yet they still operate on recorded scenarios.
As AV systems mature, failures emerge from combinations of conditions rather than from single variables. For example, a planning model may perform reliably at an unprotected left turn in daylight and in light rain, yet fail only when a specific combination of road geometry, actor behavior, lighting, and sensor conditions occurs together. Collecting more examples of common driving conditions does little to resolve a gap if those conditions are absent from the dataset.
The limitation of reactive triage is that model diagnosis and data acquisition remain loosely coupled. The pipeline can identify valuable events after they appear, but it cannot intentionally acquire the evidence needed to investigate a model weakness.
Directed triage aims to bridge that gap by connecting a known model weakness to the next data acquisition task.
Pony.ai’s PonyWorld 2.0 is one example of this shift. Its Intention layer represents what the driving model intended to do and compares that intent with the observed outcome. The system can distinguish between incorrect execution, a flawed decision, and a mismatch between its representation of the world and real-world behavior.
When PonyWorld detects a recurring weakness, it can generate targeted data collection tasks for testing and operations teams. Those teams collect matching real-world samples to support scenario generation and targeted fine-tuning.
Why Directed Triage Makes Human Judgment More Important
A model-generated request can narrow the search space, but it cannot verify the root cause of its own failure.
For example, a model may attribute an event to poor pedestrian perception. But the actual failure may originate from camera overexposure, LiDAR-camera timestamp misalignment, localization errors, prediction uncertainty, incorrect map context, or an inappropriate planning response despite accurate perception.
The human-in-the-loop layer needs to challenge the model’s diagnosis before acting on it. And that requires answering several questions.
Is the weakness genuine?
The team must distinguish a consistent performance gap from an isolated anomaly, corrupted sensor data, or misleading model signal.Does the evidence reproduce the relevant conditions?
Matching a high-level scenario class is not enough. If the weakness depends on actor trajectory, timing, occlusion, illumination, or sensor conflict, those properties must be present in the acquired sample.Where does the failure originate?
Investigation requires synchronized data from cameras, LiDAR, radar, telemetry, model outputs, and system state.What should happen next?
A validated event might need annotation, further collection, scenario simulation, regression testing, model fine-tuning, or full engineering analysis.
This independent review process also guards against subtle failure modes like self-confirming data collection. When hypotheses determine both the evidence sought and its interpretation, the workflow risks reinforcing the model’s original assumptions instead of exposing its blind spots.
So, directed triage does not reduce the human role but moves judgment toward verifying the trustworthiness, relevance, sufficiency, and actionability of evidence identified by AI.
Operationalizing directed triage at scale requires a structured human-in-the-loop workflow. Explore how iMerit’s autonomous vehicle triage services help AV teams validate and route AI-flagged priority scenarios.
What Must a Directed Autonomous Vehicle Data Pipeline Prove?
Generating targeted data requests is only the first step. A directed triage pipeline must show that each request can be traced from model signal to collected evidence, expert review, downstream action, and post-change evaluation.
The model-generated request is traceable
The pipeline must retain the signal that triggered the request, the model’s intended behavior, the observed outcome, uncertainty indicators, vehicle state, and operational context. This traceability lets reviewers determine why the scenario was prioritized, whether the suspected weakness is credible, and whether additional fleet, annotation, or engineering resources should be committed.
The acquired data matches the request
Targeted acquisition must capture the requested conditions so the collected data can actually test the suspected weakness. The pipeline should preserve synchronized sensor streams, environmental context, vehicle-state data, model outputs, interventions, and relevant metadata. If the weakness depends on temporal interaction, sensor conflict, or a sequence of decisions, the sample must preserve those properties. Otherwise, the pipeline increases dataset volume without generating evidence that addresses the identified weakness.
The scenario reaches the right level of expertise
The workflow should define when an event remains in operational review, when it moves to technical analysis, and when it requires engineering investigation. Structured escalation ensures routine cases are handled at the operational level, while ambiguous scenarios receive specialist review. Additionally, complex cross-system failures reach engineering teams with the relevant evidence already assembled. This prevents both unnecessary engineering workload and prolonged investigation of issues that cannot be resolved operationally.
The result closes the learning loop
Every request should end in a defined outcome and a measurable follow-up. That may include annotation, additional data collection, simulation, regression testing, fine-tuning, or root cause investigation. The pipeline should then record whether the intervention reduced the original weakness or whether further acquisition is required, allowing teams to measure whether the original model hypothesis was actually resolved. Without that feedback, directed acquisition becomes another intake process rather than a learning system.
How iMerit Supports Directed Triage for Autonomous Vehicle Data
Meeting directed triage requirements depends on applying the right level of human expert judgment as AI-flagged scenarios move through the pipeline. iMerit provides a three-tier triage framework to ensure each scenario is validated, investigated, and routed into an outcome for annotation, simulation, or engineering teams.
Tier 1: Operational triage
Data specialists validate completeness and scenario relevance, organize AI-prioritized events, identify annotation or data-quality issues, remove clear noise and duplicates, and prepare qualified scenarios ready for technical review.Tier 2: Technical triage
iMerit Scholars and AV-trained specialists investigate AI-flagged priority scenarios using replay tools and synchronized multimodal data. They validate whether the suspected weakness is genuine, assess its novelty and severity, and identify the likely failure mode. Then, they provide a validated failure assessment with a recommended next action, whether that is annotation, simulation, further data collection, or engineering review.Tier 3: Engineering triage.
For the most complex AI-flagged scenarios, iMerit Scholars and engineering specialists perform full-stack root-cause analysis across perception, localization, prediction, planning, and system interactions. The outcome is root-cause findings with actionable engineering recommendations to improve model performance.
Conclusion
Directed triage changes the role of autonomous vehicle data operations. Instead of searching existing fleet logs for useful scenarios, teams begin with a model-generated hypothesis and work backward to collect, validate, and investigate the evidence required to resolve it. Success will depend on how reliably AI-generated requests are translated into trusted training data, engineering insights, and measurable improvements to the driving system.
Build a tiered human-in-the-loop workflow turning AI-identified weaknesses into validated scenarios and targeted model improvements. Explore iMerit’s autonomous vehicle triage services.
