Dirac's 1942 Bakerian Lecture problem: a drastic improvement of the quantum-mechanical formalism to a complete, directly interpretable relativistic theory

In his Bakerian Lecture delivered in 1941 and published in 1942, P. A. M. Dirac proposed a probabilistic interpretation of relativistic quantum mechanics based on 'hypothetical worlds', negative probabilities, and an indefinite metric, and closed with a diagnosis that is the problem considered here: the interpretation computes only collision probabilities and does not cover physics at large (the structure of solids being his example); the Schrodinger-style 'description of Nature' survives relativistically only for a hypothetical world, with superposition failing and photons requiring a Gibbs ensemble; and the mathematical methods are not final, since the symmetry between positive and negative energies, arising from the association of energies with the Fourier components of functions of time, is 'the source of all the trouble', so any improvement must be drastic. The problem is open.

Background

By the early 1940s quantum mechanics had a stable non-relativistic formalism but a contested interpretation, and the contest was between two conceptions of what a physical theory should be. In Heisenberg's conception, a theory is a mathematical scheme from which one can consistently calculate the results of experiments; a detailed description of what is taking place in nature is unnecessary and may even be impossible to establish. In Schrodinger's conception, the wave function supplies, in a limited sense, a description of Nature: a calculation that gathers together the probability coefficients which act jointly under given conditions in nature stands in close correspondence with the physical process, so that steps of the calculation have counterparts in the process. The non-relativistic theory rests on the wave function, a complex function of a system's coordinates and time whose squared modulus gives probability densities, evolving by a wave equation $i\hbar\,d\psi/dt=H\psi$ with Hermitian Hamiltonian $H$; linear combinations of solutions are again solutions (the superposition principle).

Relativistic extension broke both the probabilities and the description. Relativistic wave equations admit negative-energy solutions: because quantum mechanics associates the energy $E=\hbar\omega$ with the time dependence $e^{-i\omega t}$, and every function of time decomposes into Fourier components of frequencies $\omega$ and $-\omega$, the formalism pairs every positive energy with a negative one. For electrons (spin one-half) Dirac's hole theory — all negative-energy states assumed filled, in accordance with Pauli's exclusion principle, with an unfilled one observed as a positron — gives a sensible but, in his words, excessively complicated interpretation. For particles of integral spin, such as photons, hole theory is unavailable (Bose statistics allows any number of particles per state, so there is no filled sea), and the negative-energy states moreover occur with negative probability.

Dirac's Bakerian Lecture (the Royal Society's premier annual lecture), delivered in 1941 and published on 18 March 1942, proposed a way out that became the direct ancestor of several modern constructions. Its central device is the hypothetical world: for electrons, a world almost saturated with positrons, in which the calculations are simple; for photons, a world whose initial photon distribution is described by a Gibbs ensemble — a statistical mixture represented by a probability-weighted set of wave functions (equivalently a density operator) rather than a single wave function — with formal initial probabilities taking the values $\pm 1/2$ according to the parity of the photon number. The theory is interpreted by assuming that the transition probabilities computed for the hypothetical world — the collision probability coefficients, i.e. probabilities per unit intensity of incident radiation that enter Einstein's radiation laws — are the same as those of the actual world. This forced Dirac to work with negative probabilities for intermediate, unobservable states and with an indefinite metric (a scalar product that can take negative values, so that some state vectors have negative norm), together with a short-distance cutoff of the interaction, introduced to control the divergences of field theory.

The lecture ends with the problem considered here: the interpretation 'does not provide a complete physical theory, since it enables one to calculate only those experimental results that are reducible to collision probabilities, and some branches of physics, e.g. the structure of solids, do not seem to be so reducible'; the limited kind of description of Nature that Schrodinger's method provides in the non-relativistic case is possible relativistically only for the hypothetical world, and even there is 'rather more indefinite' — the superposition principle fails — because of the need to use a Gibbs ensemble for the photon distribution; and the present mathematical methods are not final, any improvement having to be 'of a very drastic character' because 'the source of all the trouble, the symmetry between positive and negative energies arising from the association of energies with the Fourier components of functions of the time, is a fundamental feature' of them.

For orientation on the later literature: the 1947-1951 renormalization program — the reorganization of perturbation theory in which divergent intermediate quantities are absorbed into a finite number of measured physical parameters — made relativistic collision probabilities computable to unprecedented accuracy without Dirac's hypothetical-world apparatus (the historical account of Buoninfante 2025 documents this trajectory, including the failure of Dirac's alternative); negative-energy solutions were reinterpreted as antiparticles, a symmetry now understood as tied to relativistic invariance and embodied in the CPT theorem — the theorem that any local Lorentz-invariant quantum field theory is invariant under the combined operation of charge conjugation, parity, and time reversal (the trajectory is documented in Sanyuk and Sukhanov 2003); the indefinite metric and negative probabilities of the 1942 lecture became standard tools in covariant quantization (Gupta 1950), in finite electrodynamics with negative-metric states (Lee and Wick 1969, 1970), and in modern non-Hermitian quantum theory — PT-symmetric quantum mechanics (theories with Hamiltonians invariant under combined parity and time reversal but not Hermitian, which can nevertheless have real spectra), Krein-space quantization (quantization on a space with an indefinite metric), pseudo-unitary and time-symmetric generalizations (evolution operators preserving an indefinite scalar product rather than the usual positive-definite one; Tsang 2022), and Lorentz quantum mechanics, whose state space carries an indefinite inner product preserved by Lorentz rather than unitary transformations (Zhang and Wu 2026). The question whether photons admit a Schrodinger-like wave function became its own literature (Bialynicki-Birula 2005). None of these developments claims to have carried out the drastic improvement the lecture calls for.

Problem Statement

Dirac's 1942 Bakerian Lecture ends with three linked unsatisfied demands on quantum theory: the available interpretation computes only results reducible to collision probabilities, while branches of physics such as the structure of solids do not seem so reducible, and the theory should ultimately be extended to include the whole of physics; the limited 'description of Nature' that Schrodinger's method provides in the non-relativistic domain survives relativistically only for a hypothetical world and in a more indefinite form (the superposition of states fails, photons requiring a Gibbs ensemble); and the present mathematical methods are not final — the symmetry between positive and negative energies, arising from the association of energies with the Fourier components of functions of time, is 'the source of all the trouble' and is built into the formalism, so any improvement must be of a very drastic character.

The problem is: construct a mathematical framework for relativistic quantum physics that

(a) is directly physically interpretable in the actual world — every observable probability is non-negative, and no identification rules translating results from a hypothetical world (or from negative-probability intermediate states) into the actual one are needed;

(b) provides a state description of individual relativistic systems, photons included, of the same kind as the non-relativistic Schrodinger wave function — a state on the system's degrees of freedom, a linear evolution law, and a probability or statistical interpretation of that state — stating explicitly which superposition properties hold, which fail, and why, and whether or in what sense a Gibbs ensemble remains necessary;

(c) eliminates the positive-negative energy symmetry at the foundational level, or derives it from independently accepted principles together with a demonstration that it no longer forces negative-norm or negative-probability states in the actual-world description;

(d) recovers the tested predictions of established non-relativistic quantum mechanics and of quantum electrodynamics within their domains, and additionally gives well-defined predictions for the branches of physics, such as the structure of solids, that are not reducible to collision probabilities;

or demonstrate that the existing relativistic quantum formalism — relativistic wave equations and quantum field theory as presently established — already satisfies them.

References

1. P. A. M. Dirac, 'Bakerian Lecture - The physical interpretation of quantum mechanics', Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences 180, 1-40 (1942). DOI: 10.1098/rspa.1942.0023, https://doi.org/10.1098/rspa.1942.0023
2. W. Pauli, 'On Dirac's New Method of Field Quantization', Reviews of Modern Physics 15, 175-207 (1943). DOI: 10.1103/RevModPhys.15.175, https://doi.org/10.1103/RevModPhys.15.175
3. S. N. Gupta, 'Theory of Longitudinal Photons in Quantum Electrodynamics', Proceedings of the Physical Society. Section A 63, 681-691 (1950). DOI: 10.1088/0370-1298-63-7-301, https://doi.org/10.1088/0370-1298-63-7-301
4. T. D. Lee, G. C. Wick, 'Negative metric and the unitarity of the S-matrix', Nuclear Physics B 9, 209-243 (1969). DOI: 10.1016/0550-3213(69)90098-4, https://doi.org/10.1016/0550-3213(69)90098-4
5. T. D. Lee, G. C. Wick, 'Finite Theory of Quantum Electrodynamics', Physical Review D 2, 1033-1048 (1970). DOI: 10.1103/PhysRevD.2.1033, https://doi.org/10.1103/PhysRevD.2.1033
6. V. I. Sanyuk, A. D. Sukhanov, 'Dirac in 20th century physics: a centenary assessment', Physics-Uspekhi 46, 937-956 (2003). DOI: 10.1070/PU2003v046n09ABEH001165, https://ufn.ru/en/articles/2003/9/c/
7. I. Bialynicki-Birula, 'Photon wave function' (2005). arXiv:quant-ph/0508202, https://arxiv.org/abs/quant-ph/0508202 (arXiv version of the review chapter in Progress in Optics, vol. 36, ed. E. Wolf, Elsevier, Amsterdam, 1996)
8. M. Tsang, 'A time-symmetric generalization of quantum mechanics' (2022). arXiv:2206.05058, https://arxiv.org/abs/2206.05058
9. L. Buoninfante, 'Strict renormalizability as a paradigm for fundamental physics', Journal of High Energy Physics 2025, 175 (2025). DOI: 10.1007/JHEP07(2025)175, arXiv:2504.05900, https://arxiv.org/abs/2504.05900
10. Q. Zhang, B. Wu, 'The Power of the Lorentz Quantum Computer', Entropy 28(3), 266 (2026). DOI: 10.3390/e28030266, arXiv:2403.04170, https://arxiv.org/abs/2403.04170