Skip to content
Quantum Mechanics

Common Variational Pitfalls

An energy calculation can fail even when the variational theorem is correct. The key distinction is between the exact energy of a trial state, the lowest energy available in its trial family, and the number returned by an optimizer or sampler. The cases below show how to tell these apart without mistaking a small numerical change for a small physical error.

Required background. The Variational and Bound Methods overview maps the methods; the Variational Principle, Trial Wavefunctions and Variational Parameters establish admissibility, bounds and stationarity.

An energy below a known ground level

Section titled “An energy below a known ground level”

For a self-adjoint Hamiltonian bounded below, every nonzero admissible trial state satisfies

E[ψ]=qH[ψ]∥ψ∥2≥inf⁡σ(H).E[\psi]=\frac{q_H[\psi]}{\|\psi\|^2}\ge\inf\sigma(H).

A deterministic result below a known exact ground energy therefore means that the computed quantity is not the stated Rayleigh quotient, the state is inadmissible, or the Hamiltonian differs. First rescale the trial state: E[cψ]E[c\psi] must be unchanged for every nonzero complex cc. In a nonorthogonal basis the norm is c†Scc^\dagger Sc, not c†cc^\dagger c.

A constant trial function in an infinite well

Section titled “A constant trial function in an infinite well”

On 0<x<L0<x<L, the normalized constant ψ=L−1/2\psi=L^{-1/2} has zero second derivative inside the interval. Assigning it zero kinetic energy would place it below E0=π2ℏ2/(2mL2)E_0=\pi^2\hbar^2/(2mL^2). The mistake is its nonzero boundary trace: the Dirichlet kinetic form has domain H01(0,L)H_0^1(0,L), which excludes this state.

By contrast, a continuous piecewise-linear function that vanishes at the walls can be admissible even though its second derivative contains delta distributions. Its kinetic form is ℏ2∫∣ψ′∣2dx/(2m)\hbar^2\int|\psi'|^2dx/(2m). Do not impose every matching condition of an exact eigenfunction as a requirement on a variational trial function, or discard singular terms when using an operator formula. Robin or contact interactions can also contribute boundary or contact terms to the form. See Boundary Conditions and Unbounded Operators.

A noisy estimate below the bound

Section titled “A noisy estimate below the bound”

A finite VMC sample can fall below the exact ground energy through ordinary statistical fluctuation. Autocorrelation, equilibration and selection of the lowest among many noisy evaluations matter. Re-evaluate the final state with fresh samples and a justified uncertainty estimate; the theorem constrains the exact expectation, not every random estimate. Variational Monte Carlo also explains when sampled local variance misses residual amplitude outside the trial state’s support.

The optimizer finds the wrong state

Section titled “The optimizer finds the wrong state”

Ordinary energy minimization seeks the lowest accessible energy. If an excited state is intended, an unconstrained family can lower its energy by mixing in lower states. An exact invariant symmetry sector gives a bound on that sector’s lowest energy. Ordered Ritz values give excited-state bounds under the min–max hypotheses.

Orthogonality to an approximate ground vector is not enough: it can leave a small true-ground-state component in the putative excited state and yield an energy below the desired excitation. A penalty objective is a different Hamiltonian unless a theorem relates its optimum to the original target.

Fermionic antisymmetry is an admissibility condition for identical fermions. Finite-system parity, spin or angular-momentum labels may instead identify the target sector. A deliberately broken-symmetry mean-field state can be useful, but its symmetry content and the effect of projecting it should be assessed explicitly.

A stationary energy with a poor wavefunction

Section titled “A stationary energy with a poor wavefunction”

At an interior stationary point, the residual r=(H−E)ψr=(H-E)\psi, when defined, satisfies only

Re⁡⟨∂aψ∣r⟩=0.\operatorname{Re}\langle\partial_a\psi|r\rangle=0.

The missing residual may point outside every direction the family can represent. More coordinates do not necessarily add useful physical directions: a Gaussian can acquire many redundant parameters while still lacking a cusp, a nodal surface or a second separated lobe. Enlarge the family structurally and test observables sensitive to the omitted feature.

For ψ=1−ϵ2∣0⟩+ϵeiϕ∣1⟩\psi=\sqrt{1-\epsilon^2}|0\rangle+\epsilon e^{i\phi}|1\rangle, energy error is ϵ2(E1−E0)\epsilon^2(E_1-E_0), while an observable connecting the two levels can change at order ϵ\epsilon. A justified gap Δ>0\Delta>0 bounds the excited weight by (E−E0)/Δ(E-E_0)/\Delta, but without spectral information a good energy alone does not establish state accuracy.

Zero full energy variance identifies an eigenspace, which may be the wrong excited or degenerate eigenspace. Zero parameter gradient is weaker still. Check symmetry and state identity as well as energy, and distinguish a full residual from one projected into a truncated space.

A larger basis gives a higher energy

Section titled “A larger basis gives a higher energy”

Exact minimization over nested admissible spaces with the same energy form cannot raise a fixed-index Ritz value. If it does, try expressing the old optimized state in the new calculation and evaluating it again. This separates three possibilities:

  • The spaces are not nested because widths, box boundaries or overlap cutoffs changed.
  • At least one nonlinear optimization stopped short of the global minimum.
  • Matrix elements, quadrature, conditioning or arithmetic changed the objective.

Consistently dropping nearly dependent basis combinations preserves the bound for the smaller retained space. Altering overlap eigenvalues without the matching change to the Hamiltonian matrix does not generally represent a valid restriction.

Projection itself also needs care. The polynomial matrix PNX4PNP_NX^4P_N is generally different from (PNXPN)4(P_NXP_N)^4: the latter discards intermediate ladder paths that leave the retained space and return. The oscillator perturbation example explains the required padding.

A conforming Rayleigh–Ritz projection of the original form retains its upper bounds. Other discretizations, effective Hamiltonians or pseudopotentials require their own comparison with the original operator. Finite-box levels above a continuum threshold are not a sequence of infinite-volume bound states; stable values inside an essential spectral gap can also be spectral pollution.

Small optimization steps without convergence

Section titled “Small optimization steps without convergence”

Raw gradients depend on coordinate units. Nondimensionalize widths, energies and lengths; use logarithmic coordinates for positive scales when appropriate. Compare analytic gradients with finite differences at several step sizes and start from physically different states when local minima are plausible.

A near-singular Hessian or state metric may signal redundancy, weak sensitivity or a singular coordinate chart. A pseudoinverse or damping can stabilize updates but changes the numerical search. Inspect the resulting physical state change, rather than judging convergence only by raw parameter increments. Variational Parameters develops these distinctions.

Methods disagree outside their common regime

Section titled “Methods disagree outside their common regime”

For the quartic oscillator, the optimized Gaussian upper bound scales as g1/3g^{1/3} at strong coupling, while first-order fixed-frequency perturbation theory grows linearly in gg. Their disagreement is expected: the weak-coupling truncation is no longer controlled. A finite variational result still has ansatz error, and a higher-order optimized perturbative expression need not retain a variational bound. Compare methods in an overlapping regime and distinguish asymptotic error estimates from exact inequalities.

The Time-Dependent Variational Principle has a different purpose: it chooses an allowed state velocity. Regular action TDVP conserves energy for time-independent HH even on an inaccurate manifold; generic real-manifold McLachlan dynamics need not. Thus energy conservation is not proof of accurate dynamics, and energy drift must be interpreted using the chosen projection and the integrator.

Exercises

Section titled “Exercises”

A kinked trial state in the box

Section titled “A kinked trial state in the box”

Set ψ(x)=Ax\psi(x)=A x for 0<x<L/20<x<L/2 and ψ(x)=A(L−x)\psi(x)=A(L-x) for L/2<x<LL/2<x<L. Normalize it and calculate its kinetic form. Compare with the exact Dirichlet ground energy.

Solution

1=2A2∫0L/2x2dx=A2L3/121=2A^2\int_0^{L/2}x^2dx=A^2L^3/12, so A=12/L3A=\sqrt{12/L^3}. Since ∣ψ′∣2=A2|\psi'|^2=A^2 almost everywhere,

E=ℏ22mA2L=6ℏ2mL2>π2ℏ22mL2.E=\frac{\hbar^2}{2m}A^2L=\frac{6\hbar^2}{mL^2} >\frac{\pi^2\hbar^2}{2mL^2}.

The kink prevents a square-integrable classical second derivative but does not invalidate the form-domain upper bound. Computing ψ′′=0\psi''=0 on each open half and dropping the central delta would falsely give zero.

Exact optimization in the wrong subspace

Section titled “Exact optimization in the wrong subspace”

For E0<E1<E2E_0<E_1<E_2, minimize the energy of ψ(θ)=cos⁡θ∣1⟩+sin⁡θ∣2⟩\psi(\theta)=\cos\theta|1\rangle+\sin\theta|2\rangle. What do its gradient and full variance say at the minimum?

Solution

E=E1cos⁡2θ+E2sin⁡2θE=E_1\cos^2\theta+E_2\sin^2\theta is minimized at θ=0\theta=0 modulo π\pi. The energy is E1E_1, the gradient vanishes and the full variance is zero: this is an exact excited state. The family is entirely orthogonal to ∣0⟩|0\rangle, so none of those successful checks establishes ground-state accuracy.

Recover the old state after enrichment

Section titled “Recover the old state after enrichment”

A calculation gives EN=−1.24E_N=-1.24 and EN+1=−1.22E_{N+1}=-1.22 after adding a basis vector. Re-evaluating the old state in the new code gives −1.24-1.24 again. What has this ruled out, and what remains?

Solution

The old state is representable and retains its energy, so the new search has an available candidate below −1.22-1.22. The reported value cannot be its exact ground-state minimum. Possible causes include incomplete optimization, selection of the wrong eigenvalue or a solver error. This one-state test does not establish that every new matrix element is correct, but it already disproves the claimed optimum.

References

Section titled “References”