Policy-Aware Design of Large-Scale Factorial Experiments
A two-stage factorial design finds better product combinations faster than standard A/B testing — in theory, at Taobao scale.
The Thesis
Online platforms constantly run overlapping experiments to tune product decisions — button colors, message copy, pricing flows — but most use siloed A/B tests that miss how changes interact. This paper proposes treating all those experiments as one big combinatorial problem and using a mathematical technique called tensor completion (think: filling in a partially-observed multi-dimensional spreadsheet) to predict untested combinations without running them. The goal is not to measure every effect precisely, but to find a high-performing policy fast under a fixed traffic budget. The catch: the method assumes outcomes have low-rank structure, meaning a small number of underlying factors drive most of the variation — an assumption that may not hold in messy real-world products.
Catalyst
Platforms like Taobao, Meta, and Booking.com now run thousands of concurrent experiments on shared user pools, creating interaction effects that naive A/B infrastructure systematically mishandles. Tensor completion algorithms have matured enough to be practically usable at scale, and the existence of large public interaction datasets (such as the 100-million-event Taobao dataset used here) makes offline validation credible for the first time.
What's New
Prior work either treated each experiment independently (standard decentralized A/B testing) or used factorial designs that require exponentially many treatment arms to cover all combinations. Earlier tensor-based approaches to experiment design focused on estimating all treatment effects accurately, not on identifying the single best policy quickly. This paper combines tensor completion for pruning weak options with sequential halving — a bandit algorithm that progressively eliminates poor candidates — to focus the budget where it matters most.
The Counter
The entire method rests on a low-rank tensor assumption — the idea that product outcomes are driven by a small number of latent factors. Real platform experiments involve user segments, seasonal effects, and product interdependencies that routinely violate this assumption. The evaluation is entirely offline: the authors replay logged Taobao data rather than running live experiments, which means the method has never actually touched real traffic or encountered the feedback loops that make online experimentation hard. The paper also requires centralizing experiment design across teams, which is an organizational change, not just a technical one — most large platforms have decentralized product orgs that would resist a single factorial authority. Finally, the strongest baselines compared against are relatively simple; the paper does not benchmark against modern multi-armed bandit systems or Bayesian optimization tools that platforms already deploy.
Longs
- BABA — Alibaba operates Taobao, the exact platform used in the paper's offline evaluation
- META — runs one of the world's largest concurrent-experiment infrastructures across Facebook and Instagram
- SHOP — Shopify merchants increasingly need multi-variate optimization tools at small traffic budgets
- PLTK — Playtika and similar mobile-game companies rely heavily on factorial feature experiments with limited user pools
Shorts
- Optimizely and legacy A/B testing vendors — their core product assumes independent, sequential experiments, which this approach explicitly replaces
- Internal experimentation teams at platforms using decentralized A/B frameworks — their workflows would need substantial re-architecture
Enablers (Picks & Shovels)
- PyTorch and JAX tensor libraries — practical tensor completion depends on autodiff-based optimization
- Taobao/Tianchi public datasets — the offline evaluation benchmark used in this paper
- Apache Spark and Flink — needed to join experiment logs at the 100M-interaction scale the paper targets
Private Watchlist
- Statsig — experimentation platform for product teams, directly relevant to centralized factorial design
- Eppo — modern experimentation infrastructure startup focused on statistical rigor
- Optimizely (private since 2022 acquisition by Episerver) — incumbent experimentation tooling
Resources
The Paper
Digital firms routinely run many online experiments on shared user populations. When product decisions are compositional, such as combinations of interface elements, flows, messages, or incentives, the number of feasible interventions grows combinatorially, while available traffic remains limited. Overlapping experiments can therefore generate interaction effects that are poorly handled by decentralized A/B testing. We study how to design large-scale factorial experiments when the objective is not to estimate every treatment effect, but to identify a high-performing policy under a fixed experimentation budget. We propose a two-stage design that centralizes overlapping experiments into a single factorial problem and models expected outcomes as a low-rank tensor. In the first stage, the platform samples a subset of intervention combinations, uses tensor completion to infer performance on untested combinations, and eliminates weak factor levels using estimated marginal contributions. In the second stage, it applies sequential halving to the surviving combinations to select a final policy. We establish gap-independent simple-regret bounds and gap-dependent identification guarantees showing that the relevant complexity scales with the degrees of freedom of the low-rank tensor and the separation structure across factor levels, rather than the full factorial size. In an offline evaluation based on a product-bundling problem constructed from 100 million Taobao interactions, the proposed method substantially outperforms one-shot tensor completion and unstructured best-arm benchmarks, especially in low-budget and high-noise settings. These results show how centralized, policy-aware experimentation can make combinatorial product design operationally feasible at platform scale.