Learning Who Disagrees: Demographic Importance Weighting for Modeling Annotator Distributions with DiADEM
A new neural model learns which demographic traits drive annotation disagreement — suggesting AI content moderation built on majority-vote labels may be systematically wrong.
The Thesis
Most AI systems that judge whether content is offensive, toxic, or harmful are trained on labels created by averaging out human disagreement. This paper argues that disagreement is not measurement error — it is signal. DiADEM (Demographic-Aware Disagreement Estimation Model) is a neural architecture that learns, for each demographic dimension like race or age, how much that dimension predicts whether annotators will disagree. The practical implication is significant: content moderation and safety classifiers trained on flattened majority labels may be encoding the perspective of a dominant annotator subgroup while suppressing others. The catch is that the evaluation rests on two datasets, and scaling this approach requires demographic metadata that most annotation pipelines do not collect.
Catalyst
Two forces converged. First, the DICES and VOICED datasets — curated benchmarks that retain per-annotator demographic metadata rather than discarding it — became available, giving researchers a way to actually test disagreement models. Second, the rise of LLM-as-a-judge pipelines (using large language models to simulate human raters) created a sharp and timely foil: this paper demonstrates empirically that even chain-of-thought prompted LLMs fail to reproduce the structure of human disagreement, making the gap between current practice and what's needed more visible.
What's New
Prior perspectivist NLP work — approaches that try to preserve label diversity rather than collapsing it — typically treated annotator identity as a flat embedding or ignored it entirely, averaging labels into a single ground truth. LLM-as-a-judge systems, increasingly used as cheap substitutes for human annotation, were assumed to approximate population-level disagreement reasonably well. DiADEM differs by learning a per-demographic importance vector (a set of learned weights that quantifies how much each social dimension like race, age, or gender drives disagreement) and by training with a novel loss function that directly penalizes mispredicted variance rather than mispredicted labels.
The Counter
Two datasets is a thin empirical foundation for a claim this broad. DICES and VOICED are both English-language, US-centric, and cover narrow content domains — conversational safety and political offense — which may not generalize to the full range of subjective labeling tasks. The correlation of r=0.75 on DICES is encouraging, but the paper does not show what happens when demographic metadata is noisy, incomplete, or unavailable, which is the default state in real annotation pipelines. Collecting and storing annotator race and age at scale raises its own ethical and legal complications that the paper does not seriously address. Finally, the 'disagreement is signal, not noise' framing, while compelling, may overstate the case: some annotation variance really is fatigue, ambiguity in task instructions, or inconsistency within a single annotator — none of which maps to social identity in a meaningful way.
Longs
- META — content moderation at scale, direct exposure to annotation quality risk
- GOOGL (Alphabet) — YouTube and Search safety classifiers trained on human labels
- Veeva Systems (VEEV) — regulated content review in life sciences where subgroup disagreement has compliance stakes
- BOTZ (Global Robotics & AI ETF) — broad AI infrastructure exposure
Shorts
- LLM-as-a-judge vendors and workflows — this paper directly shows they fail to recover disagreement structure, undermining a fast-growing evaluation shortcut
- Annotation aggregation services that sell majority-vote labels as ground truth — their core product is reframed here as systematically lossy
- Content moderation API providers (e.g., OpenAI Moderation, Perspective API) — if their training pipelines rely on flattened labels, their classifiers may encode subgroup bias in ways this work makes harder to ignore
Enablers (Picks & Shovels)
- DICES dataset (Google Research) — conversational safety benchmark with annotator demographics retained
- VOICED dataset — political offense benchmark enabling cross-domain generalization tests
- Hugging Face Datasets library — infrastructure for distributing demographically-annotated corpora
- PyTorch — underlying framework for DiADEM's neural architecture
Private Watchlist
- Scale AI — dominant annotation platform; demographic-aware labeling would be a product differentiator
- Surge AI — annotation marketplace with annotator profiling capabilities
- Cohere — enterprise LLM provider whose safety fine-tuning depends on annotation quality
- Jigsaw (Alphabet subsidiary) — Perspective API for toxicity detection is a direct downstream consumer
Resources
The Paper
When humans label subjective content, they disagree, and that disagreement is not noise. It reflects genuine differences in perspective shaped by annotators' social identities and lived experiences. Yet standard practice still flattens these judgments into a single majority label, and recent LLM-based approaches fare no better: we show that prompted large language models, even with chain-of-thought reasoning, fail to recover the structure of human disagreement. We introduce DiADEM, a neural architecture that learns "how much each demographic axis matters" for predicting who will disagree and on what. DiADEM encodes annotators through per-demographic projections governed by a learned importance vector $\boldsymbolα$, fuses annotator and item representations via complementary concatenation and Hadamard interactions, and is trained with a novel item-level disagreement loss that directly penalizes mispredicted annotation variance. On the DICES conversational-safety and VOICED political-offense benchmarks, DiADEM substantially outperforms both the LLM-as-a-judge and neural model baselines across standard and perspectivist metrics, achieving strong disagreement tracking ($r{=}0.75$ on DICES). The learned $\boldsymbolα$ weights reveal that race and age consistently emerge as the most influential demographic factors driving annotator disagreement across both datasets. Our results demonstrate that explicitly modeling who annotators are not just what they label is essential for NLP systems that aim to faithfully represent human interpretive diversity.