Ember: An Extensible Benchmark Suite for Quantum Annealing Embedding Algorithms
Ember standardizes how researchers compare quantum annealing compilers — a missing piece that could accelerate D-Wave hardware adoption.
The Thesis
Quantum annealing computers, like those made by D-Wave, cannot run a user's problem directly. First, the problem's logical structure must be mapped onto the physical hardware's sparse, fixed wiring — a step called minor embedding, and it's a hard combinatorial problem in its own right. The quality of that mapping determines whether the quantum hardware produces good answers, yet researchers have been comparing embedding algorithms using incompatible test sets and inconsistent rules, making it impossible to know which approach actually works best. Ember provides a shared benchmark — 24,016 standardized problem instances, a reproducible testing harness, and support for all three current D-Wave hardware generations — so the field can start making apples-to-apples comparisons. The catch is that this is infrastructure work, not a new algorithm: Ember's immediate value is enabling better science, not delivering a performance leap itself.
Catalyst
D-Wave has shipped three successive hardware generations (Chimera, Pegasus, and the newest Zephyr topology), each with different wiring patterns that scramble comparisons made on older hardware. Meanwhile, reinforcement-learning-based embedding approaches have recently entered the field, making a neutral testing ground urgent before algorithmic claims proliferate without common baselines. The availability of pip-installable quantum SDKs and open-source graph libraries has also lowered the cost of building such infrastructure to a point where a small research team can credibly release it.
What's New
Prior embedding benchmarks were ad hoc: each research group chose its own graph families, metrics, and random seeds, so results rarely transferred across papers. Individual studies typically used small, structured graph sets — such as random Erdős–Rényi graphs or King's graphs — and evaluated only one or two algorithms in isolation. Ember replaces this with a shared library of 24,016 instances spanning structured, random, and physics-motivated problem types, a fixed algorithm interface with seeded execution so runs are reproducible, and a unified pipeline covering all three D-Wave topologies.
The Counter
Ember is a benchmarking framework, not a result — and the history of computing is littered with benchmark suites that researchers quietly ignored when their methods didn't look good on them. The paper's own finding that 'no algorithm dominates universally' is actually a mild result: it means the benchmark reveals no clear winner, which is useful information but doesn't move the needle on quantum utility. Minor embedding is also only one of several compilation bottlenecks in quantum annealing; even a perfect embedding algorithm wouldn't fix noise, limited qubit counts, or the fundamental question of whether annealing outperforms classical methods on practically relevant problems. Adoption requires the community to agree on Ember as the standard, and scientific communities have a poor track record of converging on shared benchmarks quickly — especially when existing groups have invested in their own setups. Finally, D-Wave's hardware is a niche platform; if the broader quantum computing industry shifts toward gate-based systems (IBM, Google, IonQ), the addressable audience for annealing-specific tooling shrinks considerably.
Longs
- QBTS (D-Wave Quantum) — direct beneficiary as Ember targets D-Wave hardware specifically
- IONQ — indirect; better benchmarking culture in quantum broadly raises the credibility floor for the sector
- QTUM (quantum computing ETF) — broad sector exposure as tooling matures
- IBM (IBM) — competes in quantum compilation research; better benchmarks pressure all vendors to improve
Shorts
- Research groups with proprietary embedding benchmarks — Ember commoditizes the comparison infrastructure they previously controlled, reducing the defensibility of claims made on non-reproducible setups
- D-Wave competitors pitching classical heuristic solvers — cleaner benchmarks make it easier to expose cases where embedding overhead erases quantum advantage, which could cut both ways for D-Wave itself
Enablers (Picks & Shovels)
- D-Wave's Ocean SDK — the open-source Python toolkit Ember builds on for hardware interaction and graph embedding
- NetworkX (Python graph library) — underlying data structure for Ember's 24,016 instance library
- PyPI / pip ecosystem — makes Ember installable in one command, lowering adoption friction
- GitHub (microsoft-owned) — hosts the open benchmark suite and enables community contribution
Private Watchlist
- 1QBit — quantum software firm specializing in embedding and optimization on annealing hardware
- Multiverse Computing — quantum optimization software that depends on embedding quality for annealing use cases
- Qu & Co (now part of Pasqal) — quantum algorithm development with optimization focus
Resources
The Paper
Minor embedding is a required compilation step for quantum annealing, mapping logical problem graphs onto sparse hardware topologies. Despite its central role in determining solution quality, no standardized benchmark exists for comparing embedding algorithms: prior studies use incompatible graph libraries, inconsistent metrics, and non-reproducible experimental setups, making cross-algorithm comparisons unreliable. We present Ember (Embedding Minor Benchmark for Evaluative Reproducibility), an open-source benchmarking framework addressing this gap. Ember provides a standardized algorithm interface with seeded, reproducible execution infrastructure; a diverse graph library of 24,016 instances spanning structured, random, and physics-motivated problem types not previously used in embedding benchmarks; and a unified analysis pipeline supporting all three current D-Wave hardware topologies (Chimera, Pegasus, Zephyr). We evaluate five algorithms across the full library on Chimera and find that no algorithm dominates universally: rankings vary systematically with graph structure, and the best algorithm depends on the family being embedded. We also examine the effects of hardware topology (including Pegasus and Zephyr), qubit error rates, and evaluate a reinforcement-learning approach (CHARME) within a narrower test set. Ember is available at https://github.com/zachmacsmith/ember and is installable via pip install ember-qc.