Ranked-choice conjoint experiments

The Thesis

Conjoint experiments — survey designs that ask respondents to choose between profiles defined by multiple attributes simultaneously — are the dominant tool for measuring what voters, consumers, or policy audiences actually care about. The standard version forces a single binary pick, which wastes information. This paper shows that asking respondents to rank profiles instead of just picking one can cut standard errors by 12–55%, depending on how many profiles are ranked. The catch is that the approach rests on two behavioral assumptions — transitivity (consistent preference ordering) and independence of irrelevant alternatives (adding a new option doesn't change the relative ranking of existing options) — that may not hold in every context. The authors provide both formal proofs and empirical pre-registered validation, plus an open-source R package called cjrank, making the method immediately usable.

Catalyst

Conjoint designs only became mainstream in political science and adjacent social sciences over the past decade, producing a large installed base of researchers now optimizing for efficiency. Pre-registration norms and replication pressures have raised the cost of underpowered studies, creating demand for designs that extract more from the same sample size. The authors' simultaneous release of cjrank as an open-source R package removes the implementation barrier that would otherwise delay adoption.

What's New

Prior conjoint work — exemplified by the Hainmueller, Hopkins, and Yamamoto (2014) framework that established the Average Marginal Component Effect, or AMCE, as the standard estimand — uses forced binary choice between two profiles, discarding any ordinal information respondents might naturally provide. Some researchers have used ratings scales as an alternative, but ratings introduce their own interpersonal comparability problems. This paper formalizes a 'rank expansion' approach that treats each pairwise comparison implied by a full ranking as an independent observation, proves this estimator is algebraically equivalent to the conventional forced-choice AMCE, and shows empirically that the efficiency gains are real and robust across candidate and policy domains.

The Counter

The core statistical result — that more pairwise comparisons per respondent reduce standard errors — is a mathematical identity, not a discovery. Any design that extracts more judgments from each respondent will be more efficient; the interesting question is whether the additional judgments are truthful. The paper's own assumption tests exist precisely because ranked responses can be strategically inconsistent or cognitively corrupted in ways forced-choice responses are not. The 12–55% efficiency gain figure is seductive, but it spans a huge range depending on how many profiles you ask respondents to rank; the high end requires ranking seven profiles simultaneously, which is a substantial cognitive burden that could introduce systematic noise. Meanwhile, the forced-choice conjoint literature is already vast and well-validated — reviewers, editors, and IRBs are familiar with it. The switching costs for researchers who have existing instruments, pre-registrations, or longitudinal designs are real. Finally, the cjrank package is brand new and untested at scale; early adopters will be debugging, not publishing.

The Paper