Scaling-Aware Data Selection for End-to-End Autonomous Driving Systems
MOSAIC cuts autonomous driving training data needs by up to 80% by smartly selecting which data domains matter most for real-world performance.
The Thesis
Training a self-driving car model requires enormous amounts of data, but not all data is equally useful. MOSAIC is a framework that figures out which slices of your dataset — say, highway merges versus parking lots — actually move the needle on the metrics that matter for deployment. It does this by fitting 'scaling laws' (mathematical curves describing how performance improves as you add more data from a given source) to each data domain, then iteratively choosing the most impactful domains to collect from next. The claimed result is matching or beating baseline performance with up to 80% less training data — a meaningful cost and time reduction for an industry where data collection is expensive. The catch is that results are shown on a proprietary evaluation score, which makes independent verification difficult.
Catalyst
End-to-end autonomous driving models — where a single neural network processes sensor input and outputs driving decisions — have grown large enough that training data costs and logistics have become a primary bottleneck. Scaling laws, previously most studied in language models, are now being applied to physical AI systems, giving researchers a principled tool to predict data value before spending money collecting it. The convergence of large E2E driving models and maturing scaling-law theory made this specific approach tractable now.
What's New
Prior data selection methods for autonomous driving treated data points individually, scoring them by diversity or uncertainty without considering how they jointly affect multiple driving metrics. Approaches like coreset selection or active learning pick 'representative' samples but don't model how different data domains scale toward specific evaluation criteria. MOSAIC instead groups data into domains, fits per-domain scaling curves to each metric, and optimizes the mixture iteratively — meaning it explicitly models the tradeoff between, say, night-driving data improving safety scores versus highway data improving comfort scores.
The Counter
The headline '80% less data' is measured against the Extended Predictive Driver Model Score (EPDMS), a proprietary aggregate metric not independently validated outside this paper — so we don't know how well it predicts actual on-road safety. The baselines chosen are described broadly as 'diverse,' but without published code or a public benchmark, it's impossible to verify that the comparison is fair or that obvious alternatives were included. Scaling laws for language models required vast compute and data to fit reliably; fitting them per-domain on driving data with limited budget may produce noisy curves that mislead the optimizer toward data that looks good on scaling extrapolations but underperforms in practice. Finally, the approach assumes data can be cleanly partitioned into meaningful 'domains' — a choice that introduces significant human bias and may not transfer across different vehicle platforms, sensor configurations, or geographic regions without expensive re-tuning.
Longs
- MBLY (Mobileye) — directly competes in E2E autonomous driving model development
- TSLA — operates one of the largest proprietary driving datasets; efficiency gains here compress rivals' catch-up timelines
- APTV (Aptiv) — autonomous driving software unit benefits from lower data collection costs
- KTOS — defense autonomy programs face similar data-efficiency constraints
Shorts
- Data collection fleet operators (e.g., Velodyne-era sensor data companies) — if 80% less data is needed, fleets log far fewer miles for equivalent model quality
- Generic active-learning software vendors — MOSAIC's domain-level approach potentially obsoletes point-level data selection tools
Enablers (Picks & Shovels)
- Scale AI and Labelbox — data labeling infrastructure that MOSAIC's domain-partitioning depends on
- CARLA and nuPlan — open autonomous driving simulation and benchmark platforms used to generate/evaluate training data
- PyTorch and JAX — frameworks used for fitting neural scaling laws at the domain level
- AWS and GCP compute — large-scale training runs needed to generate the scaling-law datapoints
Private Watchlist
- Wayve — E2E autonomous driving startup that would directly benefit from data-efficient training
- Waabi — simulation-heavy AV startup with data selection as a core research theme
- Scale AI — primary vendor for autonomous driving data labeling and curation
- Encord — data-curation tooling for computer vision and physical AI
Resources
The Paper
Large-scale deep learning models for physical AI applications depend on diverse training data collection efforts. These models and correspondingly, the training data, must address different evaluation criteria necessary for the models to be deployable in real-world environments. Data selection policies can guide the development of the training set, but current frameworks do not account for the ambiguity in how data points affect different metrics. In this work, we propose Mixture Optimization via Scaling-Aware Iterative Collection (MOSAIC), a general data selection framework that operates by: (i) partitioning the dataset into domains; (ii) fitting neural scaling laws from each data domain to the evaluation metrics; and (iii) optimizing a data mixture by iteratively adding data from domains that maximize the change in metrics. We apply MOSAIC to autonomous driving (AD), where an End-to-End (E2E) planner model is evaluated on the Extended Predictive Driver Model Score (EPDMS), an aggregate of driving rule compliance metrics. Here, MOSAIC outperforms a diverse set of baselines on EPDMS with up to 80\% less data.