Claims and Evidence Checklist

When a paper reports a quantum speedup, a better sensor, or a protected qubit, begin with one concrete question: what result was obtained, under what conditions, and compared with what? A useful conclusion can be narrower than the headline and still represent an important advance. This guide gives a compact way to read such results without either accepting every interpretation or dismissing useful experiments.

Helpful background. Quantum claims and their limits distinguishes experiments, theorems, and projections. Reporting standards explains how measured quantities, uncertainty, and computational costs should be described.

Six questions for a quantum result

Use these questions for a specific statement, rather than assigning one verdict to an entire paper or institution.

1. What is the task? Identify the input, output, and required quality. “Optimization” is a subject; “find a feasible cut of value at least 120” is a task.
2. What was measured or proved? Separate a directly observed count, a parameter inferred through a model, a theorem under assumptions, and a future-hardware estimate.
3. Is the comparison fair? Match input access, output quality, failure probability, tuning, and costs. Distinguish elapsed time from the time spent only executing quantum gates.
4. What does the uncertainty cover? Identify which circuits, devices, days, or problem instances were sampled, as well as the number of repeated shots.
5. How was the answer checked? Look for a task-relevant certificate, trusted reference, controlled limit, or statistical test with stated assumptions.
6. How far does the result extend? A selected gate pair does not describe every gate; a memory experiment does not determine every logical operation; a projection does not demonstrate present hardware.

These questions support a reasoned explanation, not a score obtained by adding check marks. A serious mismatch in the task cannot be repaired by reporting many unrelated calibration details. The distinction between component and application benchmarks is developed by Proctor et al. (2025).

Compare the same output at the same quality

Suppose a quantum optimizer returns a feasible solution of value 120 in $0.5\,\mathrm{s}$, while a classical solver proves the optimum is 128 in $4\,\mathrm{s}$. For this maximization problem, the latter returns both a better solution and a proof. The time ratio is eight, but these measurements do not establish an eightfold speedup for a common task.

A useful next comparison asks how long each method takes to reach value 120 with the same success probability. Alternatively, require both to find and certify the optimum. A success threshold, an average objective, and a proof of optimality are different outputs. Algorithmic benchmarking explains how to compare them across instances.

Also examine data access. An algorithm receiving a prepared quantum state or a sampling oracle may solve a different computational problem from one receiving an explicit classical array. Its access cost can be excluded in a query-complexity theorem, provided that assumption stays visible. An end-to-end runtime claim must explain how the required access is supplied.

Keep accepted trials and all attempts visible

Filtering can improve the quality of retained results while reducing their number. For a task with an explicit pass/fail criterion, let $N_A$ be attempts, $N_K$ kept trials, and $N_G$ good kept results. The observed fractions are

$$\begin{aligned} \widehat p_{\rm keep}&=N_K/N_A,\\ \widehat p_{\rm good\mid keep}&=N_G/N_K,\\ \widehat p_{\rm delivered}&=N_G/N_A. \end{aligned}$$

The last fraction is the product of the first two when their denominators are nonzero. This identity is about counted successes. A quantum-state fidelity is not automatically a per-trial success probability with an observable “good state” label.

For example, 50,000 attempts yield 2,000 kept trials and 1,900 verified good outputs. The retained success fraction is $95\%$, but the good-output fraction per attempt is $3.8\%$. If the complete run takes 100 seconds, its observed good-output rate is $19\,\mathrm{s}^{-1}$. Each number is useful for a different question.

Costs must follow the same set of trials. Include setup, unsuccessful attempts, and any checking needed to identify usable outputs. Setup can be spread across many uses when the application actually reuses it. A hypothetical future workload does not reduce the measured cost of a single current job. Network case studies illustrates this distinction for heralded entanglement and remote gates.

Ask what repeated shots can tell you

A million shots can estimate one circuit’s outcome probabilities precisely during one stable calibration period. They do not create a million independently sampled devices or calibration periods. If the claim concerns typical circuits, the circuit selection matters; if it concerns year-long operation, acquisition across time matters.

The error bar should describe the claimed population. Sampling uncertainty, calibration uncertainty, model bias, and drift are different contributions. A narrow interval obtained by resampling the same data does not automatically cover an incorrect noise model or an unrepresented class of inputs.

Selection also matters. If a team tries many qubit subsets or optimizer settings and reports the best result, that maximum tends to include favorable noise. Test the chosen setting on fresh data or account for the selection procedure. A held-out test set is useful only while it remains separate from tuning; repeatedly using it to choose a method makes it part of the tuning process.

Check the conclusion at the right level

A test must be sensitive to the proposed failure. Measuring the populations of $|0\rangle$ and $|1\rangle$ cannot reveal loss of their relative phase. Agreement on a few observables does not in general establish closeness of a complete many-body state. A benchmark score does not certify every output distribution. Certification results therefore specify both a property and the assumptions under which the measurement checks it. Kliesch and Roth (2021)

Result being read	A useful next question	Example to study
Longer logical memory	Which states and physical reference were compared?	Error correction
Better field sensitivity	Are bandwidth, dead time, and calibration matched?	Sensing
Faster key generation	Is this raw detection rate or finite-size secret-key rate?	Cryptography
Smaller quantum simulation error	Were truncation, finite size, and model error examined?	Materials simulation

Follow important numbers to the original methods and data, and check corrections or later versions. Several articles repeating one estimate are not independent confirmations. Recomputing a result from the same data checks the calculation; a new experiment with new data tests its repeatability in a further setting. The terminology varies by field; the National Academies distinguishes these as computational reproducibility and replicability. Neither alone guarantees that every interpretation of the result is correct. National Academies (2019), chapter 3

State what follows, and what remains open

Write the observation first, followed by the conclusion it supports. For example:

On the selected gate pair and calibration period, the benchmark produced a model-dependent average gate-fidelity estimate of 99.8%. The experiment did not measure a processor-wide success probability.

When information is missing, say which conclusion cannot yet be drawn. “The reported timings use different accuracy targets, so they do not determine the speedup at matched accuracy” is more useful than either “quantum advantage proved” or “the method cannot be faster.” Reserve contradiction for data or reasoning that actually conflict with the claim. An inconclusive comparison can motivate a better experiment.

Future-hardware estimates are similarly conditional. A calculation assuming a lower physical error rate can be valid arithmetic while describing no currently demonstrated machine. Read the assumptions, their sensitivity, and the resources omitted from the estimate. Resource estimation gives worked examples of this dependence.

Exercises

1. A selected gate pair

A report says “the processor has 99.8% fidelity,” based on interleaved randomized benchmarking of one selected two-qubit pair. What conclusion is justified?

Solution

The result concerns that gate pair under the protocol, noise assumptions, and calibration conditions. Quote it as a protocol-derived estimate, with its uncertainty. It does not determine all other gates, simultaneous-operation behavior, or application success. Randomized benchmarking explains the model and the limitations of interleaved estimates.

2. Faster at a different accuracy

A quantum calculation estimates an energy with stated uncertainty $5\,\mathrm{m}E_h$ in 10 seconds. A classical calculation reports $1\,\mathrm{m}E_h$ in 40 seconds for the same Hamiltonian. Is a fourfold speedup established?

Solution

No matched-accuracy speedup follows. Compare time at the same error criterion, confidence level, and treatment of bias. The classical method might reach the looser target much sooner; the quantum method might need more sampling to reach the tighter target. Neither missing runtime can be inferred merely from the two quoted numbers.

3. Which percentage describes delivery?

Of 10,000 attempted runs, 800 are retained and 760 pass a verifiable output test. Calculate the three fractions defined above. If all attempts together cost 200 seconds, find the observed good-output rate.

Solution

The kept fraction is $8\%$, success among kept trials is $95\%$, and good delivery per attempt is $7.6\%$. The observed rate is $760/200=3.8\,\mathrm{s}^{-1}$. These are sample estimates; uncertainty depends on the sampling process and correlations. The $95\%$ number does not describe all attempts.

4. Many shots, one operating period

Twenty circuits each have a million shots, all on one device during one calibration window. What additional sampling would support a claim about performance across devices and months?

Solution

Sample the relevant devices and acquisition periods, and account for their variation. More shots within the existing window primarily improve within-circuit probability estimates. Generalization across circuits also depends on how those twenty circuits were selected; a convenient set is not automatically representative.

5. An assumed future parameter

A resource estimate predicts ten days of runtime with a physical error rate ten times below a cited device result. Does that discrepancy make the estimate false?

Solution

It makes the estimate conditional on an improved error rate. Check the calculation and how the runtime changes as that assumption varies. The cited experiment does not demonstrate the assumed performance, so the estimate alone cannot establish present feasibility. It also does not prove that such performance is impossible or absent from every other device.

6. A memory and a separate gate

A paper shows lower logical memory error at distance 5 than at distance 3 and demonstrates an entangling operation separately. Does this establish sustained universal fault-tolerant computation? Does it establish that the architecture cannot provide it?

Solution

Neither follows from those facts alone. They support the tested memory comparison and logical operation, subject to their assumptions. A universal computation also requires the other logical operations and their reliable composition, including non-Clifford resources and classical control. Missing demonstrations leave that stronger conclusion open; they do not contradict the architecture’s possible future capability.

References

• M. Kliesch and I. Roth, “Theory of quantum system certification,” PRX Quantum 2, 010201 (2021). doi:10.1103/PRXQuantum.2.010201.
• National Academies of Sciences, Engineering, and Medicine, Reproducibility and Replicability in Science, National Academies Press (2019), chapter 3. doi:10.17226/25303.
• T. Proctor, K. Young, A. D. Baczewski, and R. Blume-Kohout, “Benchmarking quantum computers,” Nature Reviews Physics 7, 105–118 (2025). doi:10.1038/s42254-024-00796-z.