Probability, Statistics, and Information

Probability connects a model of an experiment to the relative frequencies of its outcomes. This chapter develops distributions, averages, uncertainty, inference, and sampling. The same tools apply to quantum measurement records once the preparation and measurement have been specified.

Helpful background. Sets, Functions, and Maps explains events and maps; Real Analysis Essentials supplies the integrals and limits used for continuous distributions.

Enter this chapter

Begin with Probability Spaces and Random Variables. Then study Probability Densities, Expectation Values, and Variance and Covariance. Work through their finite examples before moving to continuous distributions.

Keep three objects separate: an outcome, a number assigned to that outcome, and the distribution of that number. A detector click is an outcome; a count or recorded position is a random variable; the distribution describes the relative frequencies predicted over repeated trials.

A density is not a point probability

If $X$ is uniform on an interval of length $a$, its density there is $f_X=1/a$. For a subinterval of length $b\le a$,

$$\mathbb P(X\text{ lies in the subinterval})=\frac{b}{a}.$$

The density can exceed $1$ numerically and carries inverse units of $X$. The probability is a dimensionless area under the density. Changing coordinates changes the density and integration measure together.

Choose a route

• Interpret measurement outcomes: the introductory sequence above → Classical versus Quantum Probability → Probability and the Born Rule. Incompatible sharp observables need not admit one joint measurement, although separate outcome distributions can always be coupled mathematically.
• Infer a parameter: Conditional Probability → Bayes’ Rule → Fisher Information. Distinguish a likelihood from a posterior and check the assumptions of an estimation bound.
• Study fluctuations and sampling: Characteristic Functions → Gaussian Distributions → Monte Carlo Basics.
• Compare information in distributions: Entropy → Relative Entropy. Discrete entropy, differential entropy, and quantum entropy answer different questions.

Check your understanding

A detector records $N$ independent trials. If the sample mean has standard error proportional to $N^{-1/2}$, how many trials reduce that error by a factor of ten?

Solution

One hundred times as many: $1/\sqrt{100N}=(1/10)/\sqrt N$. This reduces random scatter when the single-trial variance is finite and fixed. It does not correct detector bias or an incorrect physical model. Correlated trials require a separate autocorrelation analysis.

Page map

Page	Central question
Probability Spaces, Light Version	What are outcomes, events, measures, and distributions?
Random Variables	How does a measurable map turn outcomes into values?
Probability Densities	How is probability represented relative to a continuous measure?
Expectation Values	How are probability-weighted averages defined and manipulated?
Variance and Covariance	How are spread and linear dependence quantified?
Conditional Probability	How does restricting to known information change a distribution?
Bayes’ Rule	How are likelihood and prior combined into a posterior?
Characteristic Functions	How does Fourier analysis encode distributions and independent sums?
Gaussian Distributions	Why do means and covariances completely determine Gaussian laws?
Entropy	How is average information or uncertainty quantified?
Relative Entropy	How is one distribution compared with another?
Fisher Information	How much local parameter sensitivity does a statistical model contain?
Monte Carlo Basics	How are expectations estimated from samples with controlled uncertainty?
Classical Probability versus Quantum Probability	Which classical rules apply to fixed measurements, and where does quantum structure exceed them?

References

• P. Billingsley, Probability and Measure, 3rd ed., Wiley, 1995.
• G. Casella and R. L. Berger, Statistical Inference, 2nd ed., Duxbury, 2002.
• T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006.
• A. S. Holevo, Probabilistic and Statistical Aspects of Quantum Theory, 2nd ed., Edizioni della Normale, 2011.
• C. P. Robert and G. Casella, Monte Carlo Statistical Methods, 2nd ed., Springer, 2004.
• A. W. van der Vaart, Asymptotic Statistics, Cambridge University Press, 1998.