Foundations of Observable Statistical Mechanics: analytic derivation of observable-specific energies, operational meaning of observable temperature and free energy, rigorous Markovian dynamics, and validity boundaries of the Maximum Observable Entropy Principle

Observable Statistical Mechanics (Scarpa, Alhajri, Vedral and Anza, arXiv:2309.15173) predicts the stationary measurement statistics of a coarse, highly degenerate observable $A$ in an isolated many-body quantum system from a Maximum Observable Entropy Principle (MOEP): maximizing the observable entropy of $A$ subject to normalization and fixed average energy yields $p_j = d_j e^{-\beta_A \varepsilon_j}/Z_A$, where $\varepsilon_j$ is the observable-specific energy — the conditional expectation of the system energy given measurement outcome $j$ — and $\beta_A$ the observable-specific inverse temperature. The framework matches exact-diagonalization numerics on interacting spin chains, including regimes where the Gibbs ensemble fails, but it is calibrated empirically and justified only heuristically. The open problem is to complete its foundations: a principled, non-empirical derivation of the $\varepsilon_j$; an operational and kinetic theory of the observable temperature $T_A$ and observable free energy $F_A$; a rigorous theorem — or refutation — connecting MOEP to an effective Markovian master equation with detailed-balance rates $\gamma_{jk}/\gamma_{kj} = e^{-\beta_A(\varepsilon_k-\varepsilon_j)}$; extension to highly degenerate non-local observables; and quantitative characterization of the failure regimes in which the accessible-information ratio $I(H,A)/S(H)$ is not small. The authors' companion extension to non-commuting conserved charges resolves one strand while explicitly leaving the others open.

Background

Consider an isolated (closed) many-body quantum system prepared in a pure state $|\psi\rangle=\sum_n c_n|E_n\rangle$ and evolving unitarily under a Hamiltonian $H$, with no coupling to an environment. The eigenstate thermalization hypothesis (ETH) — the statement that individual energy eigenstates of generic non-integrable Hamiltonians look thermal to few-body observables — and its generalizations explain when observables equilibrate, but constructing the corresponding equilibrium ensembles (Gibbs, microcanonical, generalized Gibbs for integrable systems, or the diagonal ensemble) requires microscopic knowledge: the energy spectrum and the overlaps $|c_n|^2$ of the initial state. The diagonal ensemble is the dephased state $\rho^{DE}=\sum_n|c_n|^2|E_n\rangle\langle E_n|$, whose statistics give the long-time averages of measurements.

Scarpa, Alhajri, Vedral and Anza (arXiv:2309.15173) asked a different question: can one predict the stationary statistics of coarse measurements alone, without microscopic information? Their framework, Observable Statistical Mechanics, targets observables $A$ with few outcomes ($n_A \ll D$, the Hilbert-space dimension) and highly degenerate eigenspaces. Measuring such an $A$ at equilibrium is like sending the energy eigenstates through a noisy communication channel: since the mutual information $I(H,A)$ between the energy and outcome distributions is bounded by the outcome entropy, $I(H,A)\le S(A)\le\log n_A \ll S(H)$ (a form of Holevo's bound on how much information one measurement can carry), the measurement is almost non-informative. Their central ansatz is the Maximum Observable Entropy Principle (MOEP): the stationary outcome distribution is obtained by maximizing the observable entropy

$\tilde{S}(A) = S(A) + \sum_j p_j \log d_j,$

where $S(A)$ is the Shannon entropy of the outcome distribution $p_j$ and $d_j$ is the degeneracy (eigenspace dimension) of outcome $j$. This is the observational entropy of the measurement, introduced by Šafránek, Deutsch and Aguirre in the context of closed-system thermalization. Maximization subject only to normalization and fixed average energy $E$ gives the Gibbs-like solution

$p_j = \frac{d_j\, e^{-\beta_A \varepsilon_j}}{Z_A}, \qquad \varepsilon_j = \frac{R_j^{DE}}{p_j^{DE}}, \qquad R_j^{DE}=\mathrm{Tr}(A_j H \rho^{DE}),$

where $A_j$ is the projector onto the eigenspace of outcome $j$, so that $\varepsilon_j$ is the conditional expectation of the system energy given outcome $j$, $\sum_j p_j\varepsilon_j = E$ fixes $\beta_A$, and $Z_A$ is a measurement-specific partition function. Defining $Z_A = e^{-\beta_A F_A}$ yields an observable free energy obeying the equilibrium relation $F_A = E - T_A \tilde{S}(A)$ with observable temperature $T_A = \beta_A^{-1}$. Numerically, for 7 non-weakly-interacting spin-1/2 chains (integrable and chaotic, $N=10$–$20$), six observable classes (three one-body and three two-body), and 5 initial states, the predicted distributions match the diagonal-ensemble (time-averaged) statistics to small total-variation error — including cases where the Gibbs ensemble fails because the measured region is not weakly coupled to the rest.

Two operational gaps remain in the theory as it stands. First, the $\varepsilon_j$ are computed empirically: guided by a microcanonical argument ($\varepsilon_j$ must lie in the energy window $[E-\Delta E/2, E+\Delta E/2]$, where $\Delta E$ is the width of the initial state's energy distribution), the authors fit the empirically linear form $\varepsilon_j = \gamma_j E + \eta_j \Delta E + \chi_j$ (their Eq. (11), with the $\Delta E$ term negligible in most cases), fixing the coefficients from two or three calibration initial states; no principled analytical derivation from microscopic data exists. Second, the dynamical mechanism is heuristic: the authors conjecture — at the physical level only — that MOEP equilibrium statistics arise from an effective Markovian master equation on outcome space whose rates obey detailed balance, $\gamma_{jk}/\gamma_{kj} = e^{-\beta_A(\varepsilon_k-\varepsilon_j)}$, in line with the work of Strasberg, Winter, Gemmer and Wang showing that classical Markovian behavior with local detailed balance can emerge from coherent pure-state dynamics. A kinetic or operational interpretation of $T_A$ and $F_A$ — how they would appear in effective transition rates, work extraction, or heat-like exchanges defined at the measurement-statistics level — was announced as in development but has not appeared. The framework is also expected to fail when the accessible-information ratio $I(H,A)/S(H)$ is not small — for small systems, measurements whose information capacity is not negligible, or initial states whose energy distribution has low entropy (equivalently, a small effective dimension $d_{\rm eff} = (\sum_n |c_n|^4)^{-1}$, the number of energy eigenstates significantly populated) — and these failure boundaries are uncharacterized.

Problem Statement

The problem is to complete the foundations of Observable Statistical Mechanics — turning the Maximum Observable Entropy Principle (MOEP) from an empirically calibrated, heuristically justified prediction rule into a closed theory with a rigorous dynamical basis and a characterized domain of validity. The components below serve this single objective and are linked by the roles that the observable-specific energies $\{\varepsilon_j\}$, the inverse temperature $\beta_A$, the observable temperature $T_A = \beta_A^{-1}$, and the observable free energy $F_A$ (defined via $Z_A = e^{-\beta_A F_A}$, with $F_A = E - T_A\tilde{S}(A)$ at equilibrium) play across prediction, dynamics, and thermodynamic interpretation. They are, however, independent sub-questions rather than one conjunctive deliverable, each carrying its own completion criterion: components 1 and 3 are the required core — the framework's two acknowledged technical gaps, an empirical calibration of $\varepsilon_j$ and a heuristic account of its dynamics — and a complete solution must resolve each of them under the completion criterion stated with it, whereas components 2, 4 and 5 are optional extensions of the same programme that a submission may address without being required to resolve them for completion.

1. (Analytic inputs; required core) Derive the observable-specific energies $\varepsilon_j$ — and thereby $\beta_A$, $Z_A$, and the predicted stationary distribution $p_j = d_j e^{-\beta_A\varepsilon_j}/Z_A$ — by a principled analytical method from microscopic or experimentally accessible data that does not require diagonalizing $H$ or otherwise resolving the energy spectrum, replacing the current empirical calibration (the linear fit $\varepsilon_j = \gamma_j E + \eta_j \Delta E + \chi_j$ in the average energy $E$ and the width $\Delta E$ of the initial state's energy distribution, fixed with two or three calibration initial states). This component is resolved by a derivation of the $\varepsilon_j$ that requires no calibration fit and no diagonalization of $H$, and whose resulting predictions reproduce the diagonal-ensemble statistics to a stated accuracy on at least one model for which no calibration data were used.

2. (Operational thermodynamics; optional extension) Construct an operational interpretation of $T_A$ and $F_A$ — a thermodynamics defined on measurement statistics rather than density matrices — by exhibiting at least one concrete setting, and demonstrating it on one model, in which these quantities enter with a physically meaningful definition: effective transition rates between measurement outcomes, work extraction, or heat-like exchanges at the measurement-statistics level. Any single one of these settings suffices; the list describes the intended scope rather than three conjunctive deliverables.

3. (Dynamical foundations; required core) Prove or refute the conjecture that the stationary statistics satisfying MOEP are generated by an effective Markovian master equation whose rates obey detailed balance, $\gamma_{jk}/\gamma_{kj} = e^{-\beta_A(\varepsilon_k-\varepsilon_j)}$, or delineate precisely the conditions — on the Hamiltonian, observable, initial state, and timescales — under which such an effective dynamics exists. This component is resolved by carrying out one of these three alternatives rigorously — a proof of the conjecture under explicit hypotheses, an explicit counterexample refuting it, or a precise statement, with proof, of necessary and sufficient conditions; a heuristic or numerical demonstration alone does not resolve it.

4. (Domain extension; optional extension) Test and extend the predictive power of the framework to highly degenerate non-local observables, where the premises of extensivity of $\tilde{S}(A)$ and low accessible information $I(H,A) \ll S(H)$ require re-examination. (The case of non-commuting conserved charges has since been explored at the numerical level by the authors' companion work, arXiv:2507.22882, which leaves the degenerate non-local regime open.)

5. (Validity boundary; optional extension) Characterize quantitatively when the framework fails: the regimes in which $I(H,A)/S(H)$ is not small — small systems, measurements with non-negligible information capacity, initial states with low energy-distribution entropy or small effective dimension $d_{\rm eff}$ — determining the boundary of applicability as a function of system size, number of outcomes $n_A$, and effective dimension.

References

1. L. Scarpa, A. Alhajri, V. Vedral, F. Anza, Observable Statistical Mechanics, arXiv:2309.15173 (v3, revised 29 July 2025), DOI 10.48550/arXiv.2309.15173, https://arxiv.org/abs/2309.15173 (preprint)
2. L. Scarpa, N. Ranabhat, A. Te'eni, A. Alhajri, V. Vedral, F. Anza, L. P. García-Pintos, Non-classicality at equilibrium and efficient predictions under non-commuting charges, arXiv:2507.22882 (2025), DOI 10.48550/arXiv.2507.22882, https://arxiv.org/abs/2507.22882 (preprint)
3. P. Strasberg, A. Winter, J. Gemmer, J. Wang, Classicality, Markovianity, and local detailed balance from pure-state dynamics, Physical Review A 108, 012225 (2023), DOI 10.1103/PhysRevA.108.012225, https://doi.org/10.1103/PhysRevA.108.012225
4. F. Meier, T. Rivlin, T. Debarba, J. Xuereb, M. Huber, M. P. E. Lock, Emergence of a Second Law of Thermodynamics in Isolated Quantum Systems, PRX Quantum 6, 010309 (2025), DOI 10.1103/PRXQuantum.6.010309, https://doi.org/10.1103/PRXQuantum.6.010309
5. J. Schindler, P. Strasberg, N. Galke, A. Winter, M. G. Jabbour, Unification of observational entropy with maximum entropy principles, arXiv:2503.15612 (2025), DOI 10.48550/arXiv.2503.15612, https://arxiv.org/abs/2503.15612 (preprint)
6. P. O'Donovan, P. Strasberg, K. Modi, J. Goold, M. T. Mitchison, Quantum master equation from the eigenstate thermalization hypothesis, Physical Review B 112, 014312 (2025), DOI 10.1103/kh96-ct3y, https://doi.org/10.1103/kh96-ct3y
7. S. Majidy, W. F. Braasch, A. Lasek, T. Upadhyaya, A. Kalev, N. Yunger Halpern, Noncommuting conserved charges in quantum thermodynamics and beyond, Nature Reviews Physics 5, 689 (2023), DOI 10.1038/s42254-023-00641-9, https://doi.org/10.1038/s42254-023-00641-9
8. F. Anzà, V. Vedral, Information-theoretic equilibrium and observable thermalization, Scientific Reports 7, 44066 (2017), DOI 10.1038/srep44066, https://doi.org/10.1038/srep44066
9. D. Šafránek, J. M. Deutsch, A. Aguirre, Quantum coarse-grained entropy and thermalization in closed systems, Physical Review A 99, 012103 (2019), DOI 10.1103/PhysRevA.99.012103, https://doi.org/10.1103/PhysRevA.99.012103
10. T. Nagasawa, K. Kato, E. Wakakuwa, F. Buscemi, Generic increase of observational entropy in isolated systems, Physical Review Research 6, 043327 (2024), DOI 10.1103/PhysRevResearch.6.043327, https://doi.org/10.1103/PhysRevResearch.6.043327
11. A. Rignon-Bret, C. Elouard, Quantum stochastic thermodynamics of macroscopic systems: an algebraic approach, arXiv:2607.09242 (2026), DOI 10.48550/arXiv.2607.09242, https://arxiv.org/abs/2607.09242 (preprint)
12. F. Anza, C. Hahn, Non-equilibrium theory of projected ensembles, arXiv:2608.10110 (2026), DOI 10.48550/arXiv.2608.10110, https://arxiv.org/abs/2608.10110 (preprint)
13. B. Bhattacharjee, R. K. Ray, D. Šafránek, Work and entropy of mixing in isolated quantum systems, arXiv:2507.05054 (2025), DOI 10.48550/arXiv.2507.05054, https://arxiv.org/abs/2507.05054 (preprint)
14. X. Cao, Consistent monitoring of quantum fluctuations, arXiv:2603.15753 (2026), DOI 10.48550/arXiv.2603.15753, https://arxiv.org/abs/2603.15753 (preprint)
15. A. D. Varizi, D. Jonathan, T. R. de Oliveira, Emergence of Thermodynamics from Equilibration in Isolated Quantum Systems, arXiv:2606.28660 (2026), DOI 10.48550/arXiv.2606.28660, https://arxiv.org/abs/2606.28660 (preprint)