Benchmarking, Verification, and Validation

How well does a quantum device perform, and what do its measurements tell us? The answer depends on the question. Benchmarking measures performance on a specified test; characterization estimates a state, operation, or noise model; verification tests a claim; validation asks whether a model or procedure is adequate for its intended use. These activities overlap, but their results answer different questions.

Helpful background. Density operators for QI supplies the objects reconstructed from data, quantum channels for QI supplies process models, and SPAM errors explains uncertainty in preparation and measurement. Basic sampling and estimation are developed in quantum measurement as estimation.

What do the measurements distinguish?

Suppose a source promises to prepare $|+\rangle$. Measuring $Z$ gives outcomes zero and one equally often. Does that confirm the state?

It does not: the completely mixed state $I/2$ gives the same probabilities. An ideal $X$ measurement distinguishes their predictions:

Prepared state	Probability of Z outcome 0	Probability of X outcome +
$\lvert+\rangle\langle+\rvert$	$1/2$	$1$
$I/2$	$1/2$	$1/2$

More repetitions of the $Z$ measurement reduce uncertainty about a probability shared by both states. They cannot reveal the missing coherence. Changing the measurement answers a question that increasing the sample size does not. This example assumes trusted measurement axes; a miscalibrated detector changes the interpretation.

For an arbitrary qubit state, measurements of $X$, $Y$, and $Z$ determine its Bloch vector and hence its density matrix. If only the fidelity to $|+\rangle$ is needed, the ideal relation

$$F_+=\langle+|\rho|+\rangle =\frac{1+\langle X\rangle}{2}$$

uses a single expectation value. Full reconstruction and estimation of one property are different tasks. Nielsen and Chuang develop the underlying state and measurement formulas.

Enter this chapter

Start with Reading Quantum-Technology Claims to connect a result to its task, comparison, assumptions, and uncertainty. Its worked examples distinguish conditional accuracy from throughput and zero observed failures from a zero failure probability.

Then read why benchmarking is hard. Write the estimand—the particular quantity to be inferred—before selecting the protocol. Is the question about an average gate error, an observable, a noise mechanism, a complete state, or success on an application? A metric that is easy to measure may not answer the question that motivated the experiment.

Choose a route by the question you want to answer.

• Reconstruct or diagnose. State tomography reconstructs a state under a measurement model; process tomography reconstructs a channel under preparation and measurement assumptions. Device characterization connects diagnostic experiments to identifiable error mechanisms and model checks.
• Estimate properties or certify a resource. Shadow tomography explains classical shadows and the cost of predicting selected properties. Certification of entanglement matches a witness or correlation test to its null hypothesis and trust assumptions. A successful certificate need not reconstruct the state.
• Benchmark operations and workloads. Randomized benchmarking studies decay under sampled operation sequences; cycle benchmarking uses multi-qubit operation layers as the unit of interest. Cross-entropy benchmarking compares samples with ideal circuit probabilities. Quantum volume and application benchmarks and algorithmic benchmarking connect test ensembles, output quality, and complete execution cost.

All routes lead to reporting standards. For a claimed separation from classical performance, add verification of quantum advantage: correctness, computational hardness, and an empirical comparison are distinct parts of the argument.

Match the protocol to the inference

Full tomography offers detailed information at substantial experimental and classical cost. Property estimation can be more economical when the requested observables have useful structure. The classical-shadow construction of Huang, Kueng, and Preskill (2020) illustrates this distinction: estimating many specified properties does not require reporting every matrix element, but its cost still depends on the measurement ensemble, observables, accuracy, and confidence. “Many observables” does not mean arbitrary observables are free.

Randomized benchmarking uses a different kind of compression. For standard Clifford randomized benchmarking with a stationary, gate-independent Markovian error channel, the sequence-averaged survival signal has the form

$$P(m)=A p^m+B,$$

where $m$ is the number of benchmarked steps, $p$ describes the decay, and $A,B$ absorb aspects of preparation, measurement, and boundary operations. The reference protocol and controlled departures from its simple model are developed by Magesan, Gambetta, and Emerson (2011). A fitted $p$ does not identify every error mechanism, and an acceptable exponential fit is not proof that the assumptions hold in every operating context.

Likewise, an XEB score is a statistic involving ideal output probabilities. Its relationship to circuit fidelity depends on a model; it is not in general a distance between the experimental and ideal distributions. A tomographic fidelity, an RB-derived average error, and an XEB score therefore cannot be compared merely because all are dimensionless numbers.

Choose what to repeat

Shots, random circuits, and time blocks are different sampling units. Repeating one circuit many times estimates that circuit’s success probability accurately. It does not determine how much success varies across other circuits or after the device drifts.

For example, if one tested circuit always succeeds and a second always fails, more shots on either one do not reduce the difference between them. To estimate the average over a specified circuit population, sample circuits from that population and account for both between-circuit variation and within-circuit shot noise. Why Benchmarking Is Hard develops the quantitative distinction.

A confidence interval describes uncertainty under its sampling assumptions. It does not include an unknown detector bias merely because the interval is narrow. Similarly, observing no failures gives an upper confidence limit, not proof that failure is impossible; see the worked binomial example in Reproducible Notebooks.

Compare complete claims

Compare the same task, output criterion, and conditions. If a protocol discards outcomes, give both the fraction retained and the quality of those retained outcomes. A 99% success rate among the 10% of attempts retained corresponds to 9.9% accepted successes per attempt, not 99% per attempt.

For comparative performance, align the output accuracy and task distribution before comparing time or cost. Include compilation, state preparation, shots, optimization, postprocessing, and relevant verification costs within a stated boundary. Classical algorithms and implementations used as baselines must be identified and dated. A favorable component metric does not establish that a complete application is useful or superior.

Use network verification for delivered communication services, verification of quantum simulation for target-model observables, and logical benchmarking for encoded operations. Applications and case studies applies these methods to concrete examples; Research follows changing results and open questions.

Check your understanding

A device achieves a low randomized-benchmarking error and a good tomography fit on one prepared state. Does that verify a long adaptive computation?

Answer

No. The RB result concerns its sampled gate set and noise assumptions; the tomography result concerns one prepared ensemble and measurement model. The adaptive computation may add different operating contexts, correlated errors, measurement and reset faults, leakage, idle time, or feedforward failures. Verification must address the claimed output of that computation, using applicable component measurements and tests of the complete computation.

For a trusted ideal X measurement, a qubit gives outcome + with probability 0.8. What is its fidelity to $|+\rangle$? Does this determine the entire state?

Answer

The fidelity is $0.8$, and $\langle X\rangle=2(0.8)-1=0.6$. The $Y$ and $Z$ components remain undetermined, subject to the Bloch-ball constraint $x^2+y^2+z^2\leq1$. For example, Bloch vectors $(0.6,0,0)$ and $(0.6,0,0.8)$ both fit: the first is mixed and the second pure.

A protocol accepts 100 of 1,000 attempts; 99 accepted outcomes meet the target. Give its acceptance, conditional success, and accepted-success rates.

Answer

The three observed fractions are $100/1000=0.1$, $99/100=0.99$, and $99/1000=0.099$. They describe different events. These frequencies still need uncertainty estimates and an appropriate sampling model when used to infer future performance.

References

• Huang, Hsin-Yuan, Richard Kueng, and John Preskill. “Predicting many properties of a quantum system from very few measurements.” Nature Physics 16, 1050–1057 (2020). doi:10.1038/s41567-020-0932-7; arXiv:2002.08953 — classical shadows and property-dependent sample complexity.
• Magesan, Easwar, Jay M. Gambetta, and Joseph Emerson. “Scalable and robust randomized benchmarking of quantum processes.” Physical Review Letters 106, 180504 (2011). doi:10.1103/PhysRevLett.106.180504; arXiv:1009.3639 — randomized decay models and the conditions for their interpretation.
• M. A. Nielsen and I. L. Chuang, Quantum Computation and Quantum Information, 10th anniversary edition, Cambridge University Press (2010), doi:10.1017/CBO9780511976667 — qubit states, measurements, and fidelity.