Benchmark Index

A benchmark measures a defined aspect of performance under a specified experiment; certification adds a stated statistical guarantee about a property. Neither word identifies one universal score. This index starts from the quantity to be inferred and links to methods that can support it. Why Benchmarking Is Hard develops the comparison framework, following the distinctions surveyed by Eisert and collaborators.

Choose the quantity to measure

Method	What it measures and what it cannot establish
State Tomography	A density operator within a specified measurement and statistical model Does not establish: Model-independent accuracy, inexpensive scaling, or freedom from preparation/measurement bias.
Process Tomography	A channel model from a spanning family of inputs and measurements Does not establish: Correct separation of preparation, gate, and measurement errors without further characterization.
Classical Shadows	Selected observables from randomized measurements Does not establish: Efficient prediction of every global observable or full state reconstruction.
Randomized Benchmarking	Survival decay under a sampled gate family and noise assumptions Does not establish: The worst-case error of every gate or the failure probability of an arbitrary algorithm.
Cycle Benchmarking	Pauli-orbit decays and model-dependent fidelity bounds for specified dressed cycles Does not establish: Context-free native-gate quality or arbitrary correlated-noise reconstruction.
Cross-Entropy Benchmarking	A score correlating observed samples with ideal circuit probabilities Does not establish: A general total-variation-distance certificate or proof of useful computational advantage.
Quantum Volume	Success on a specified random circuit family with width/depth and confidence rules Does not establish: Logical qubit count, fault tolerance, or performance on every application.
Algorithmic Benchmarking	Accuracy and resources on a task and instance distribution Does not establish: Generality beyond the selected distribution or a fair comparison without matched classical baselines.
Logical Benchmarking	Encoded performance for declared states, rounds, gates, and decoder Does not establish: Universal logical performance from a single protected observable.
Entanglement Certification	A witness, fidelity bound, or other criterion that excludes an identified separable class Does not establish: A complete entanglement measure, Bell nonlocality, or cryptographic security.

Three scores that answer different questions

Randomized-benchmarking decay. The simplest model is

$$P(m)=A\alpha^m+B,$$

where m counts sampled operations and A,B absorb stable preparation/measurement contributions under the model. Converting α into $r=(d-1)(1-\alpha)/d$ as an average infidelity is justified in the gate-independent depolarizing description. Gate dependence, leakage, drift, and sequence context need additional analysis. Magesan, Gambetta, and Emerson develop the RB framework; the detailed RB page states its inference limits.

Linear cross-entropy score. For an n-qubit circuit with ideal distribution p and observed samples $x_1,\ldots,x_N$, a common convention is

$$\widehat F_{\mathrm{XEB}}=\frac{2^n}{N}\sum_{j=1}^N p(x_j)-1.$$

Uniform independent samples have mean score zero. Ideal samples have mean $2^n\sum_xp(x)^2-1$, which depends on the circuit distribution and need not equal one for a fixed circuit. Interpreting the score as a fidelity requires a noise/distribution model and suitable normalization. Scoring also requires access to ideal probabilities for the tested outputs.

Classical-shadow prediction. For M target observables with suitable bounded shadow norms, a typical guarantee has measurement count scaling as

$$N=O\!\left( \frac{\log(M/\delta)}{\epsilon^2} \max_i\lVert O_i\rVert_{\mathrm{shadow}}^2 \right).$$

Here ε is additive accuracy and δ is failure probability. The measurement ensemble determines the shadow norm; it can grow rapidly with observable support. Huang, Kueng, and Preskill’s result does not give dimension-independent cost for arbitrary global observables.

Interpret scores in their experimental setting

Erhard and collaborators’ cycle benchmarking targets simultaneous operations in a chosen cycle; the cycle definition and randomizing gates are part of the reported experiment. Cross and collaborators’ quantum volume uses a heavy-output test on random circuits: $V_Q=2^m$ records a successfully certified square-circuit size m according to its protocol, not $2^m$ physical qubits. Compilation choices and confidence analysis matter.

For an application benchmark, report output error, success probability, all relevant runtime, and the classical comparator. For an encoded memory, report accepted and rejected runs, storage duration, code distance, decoder, and the unencoded baseline with matched physical conditions. A fitted uncertainty interval quantifies sampling uncertainty only within the selected estimator and model; it does not remove model bias.

A counterexample to overinterpreting XEB

Consider an ideal distribution on D outputs with a particularly likely string $x_\star$. A sampler that always outputs that string has score $Dp(x_\star)-1$, which can be large even though it ignores the rest of the ideal distribution. Finding $x_\star$ may itself be difficult; the counterexample concerns what the score logically establishes, not an efficient attack on every circuit ensemble. It shows why a score must be interpreted with distributional assumptions and an explicit claim.

References

• Cross, Andrew W., Lev S. Bishop, Sarah Sheldon, Paul D. Nation, and Jay M. Gambetta, “Validating quantum computers using randomized model circuits,” Physical Review A 100, 032328 (2019), arXiv:1811.12926 — quantum volume and heavy-output protocol.
• Eisert, Jens, et al., “Quantum certification and benchmarking,” Nature Reviews Physics 2, 382–390 (2020), doi:10.1038/s42254-020-0186-4 — inference goals, guarantees, and limitations.
• Erhard, Alexander, et al., “Characterizing large-scale quantum computers via cycle benchmarking,” Nature Communications 10, 5347 (2019), doi:10.1038/s41467-019-13068-7 — cycle-specific characterization.
• Huang, Hsin-Yuan, Richard Kueng, and John Preskill, “Predicting many properties of a quantum system from very few measurements,” Nature Physics 16, 1050–1057 (2020), arXiv:2002.08953 — classical-shadow guarantees.
• Magesan, Easwar, Jay M. Gambetta, and Joseph Emerson, “Scalable and robust randomized benchmarking of quantum processes,” Physical Review Letters 106, 180504 (2011), arXiv:1009.3639 — RB decay models.