Scaling generalized nonorthogonal matrix elements: two-electron intermediate compression and on-the-fly nonorthogonal configuration interaction

The framework of generalized nonorthogonal matrix elements (Burton, J. Chem. Phys. 157, 204109 (2022)) evaluates one- and two-body Hamiltonian and overlap matrix elements between arbitrarily excited configurations built from different nonorthogonal reference determinants at $O(1)$ cost in the system size once intermediates are pre-computed. Its practical bottleneck, identified as such by the source, is the two-electron intermediates: pre-computation scales as $O(16n^{5})$ and storage can reach $256n^{4}$ per determinant pair for $n$ basis functions; individual high-rank matrix elements carry combinatorial prefactors $O(L^{6}\binom{L+2}{m})$ in the excitation number $L$; and the applications that this reduction would enable — on-the-fly NOCI-PT2, NOCI-MP2, NOCIS, and couplings between excited state-specific multiconfigurational wavefunctions — remain to be developed. The open problem is to make this framework large-scale: derive reduced-complexity representations of the two-electron intermediates (the source names the Cholesky decomposition and the resolution-of-the-identity as techniques that 'may provide further computational savings'; other compression routes are equally admissible) together with the required algorithmic strategies, such that the resulting matrix elements preserve the accuracy and numerical stability of the exact determinantal expressions to within a quantified, user-controllable compression error (the representation being exact in the untruncated limit). Zero-overlap biorthogonal orbital pairs ($m>0$), which generate the extra index combinations and the framework's known numerical difficulties, mark the regime in which such a reduction is hardest. As of August 2026 no published work or library release implements such a reduction; the open core survives intact.

Background

In most configuration-interaction (CI) quantum-chemistry methods, the wavefunction is expanded in Slater determinants (antisymmetrized products of one-electron spin-orbitals) that are all built from one orthonormal orbital set. However, some of the most chemically useful qualitatively correct wavefunctions are naturally nonorthogonal: symmetry-broken unrestricted Hartree–Fock solutions (mean-field determinants whose alpha- and beta-spin orbitals are optimized independently, so the determinant is not spin-symmetry adapted), determinants with orbitals separately optimized for different electronic states, localized valence configurations, and symmetry-restored mean-field manifolds. Nonorthogonal configuration interaction (NOCI) generalizes CI to an expansion over such determinants, each with its own orbitals, and is a flexible tool for static (strong) correlation, avoided crossings, spin-symmetry restoration via the generator coordinate method (an expansion over symmetry-broken mean-field states indexed by a continuous coordinate), and core-level spectroscopy with orbital relaxation.

Evaluating Hamiltonian and overlap matrix elements between nonorthogonal determinants is the central computational obstacle. The standard route biorthogonalizes the two orbital sets into mutually orthogonal 'corresponding orbital' pairs (Amos–Hall); pairs with (near-)zero overlap — there are $m$ of them, the 'zero-overlap pairs' — require special treatment and are the source of extra index combinations in every formula. The classical generalized Slater–Condon rules cost $O(n^{2})$ or $O(n^{4})$ per one- or two-body matrix element, which becomes prohibitive when many coupling elements are needed. Burton's generalized nonorthogonal Wick's theorem (J. Chem. Phys. 154, 144109 (2021)), extended to arbitrary excitation rank in the source paper (J. Chem. Phys. 157, 204109 (2022)), instead yields compact determinantal expressions: after pre-computing one-electron intermediates ($O(16n^{3})$ work, $O(16n^{2})$ storage) and two-electron intermediates $\mathcal{J}$, any matrix element between configurations excited by up to $L$ spin-orbitals from their references is evaluated at $O(1)$ cost in the number of basis functions or electrons, with a per-element combinatorial prefactor $O(L^{6}\binom{L+2}{m})$. The open-source C++ library LibGNME implements this framework.

The bottleneck is the two-electron intermediates. Building the full set of $\mathcal{J}$ intermediates costs $O(16n^{5})$ — the same fifth-power scaling as a standard two-electron integral transformation, with the factor 16 counting the $(m_{i},m_{j},m_{k},m_{l})$ combinations of zero-overlap indices — and storing all combinations requires up to $256n^{4}$ memory per determinant pair, a factor 256 beyond the already challenging storage of the two-electron integrals themselves. The source paper notes two available reductions — when there are no zero-overlap pairs ($m=0$) only a subset of combinations is needed, and excitations can be restricted to an active orbital space — and suggests that the Cholesky decomposition (a low-rank factorization of the positive-semidefinite two-electron integral tensor, widely used to compress integrals in conventional correlated methods) or the resolution-of-the-identity/density-fitting approach (expanding orbital-pair products in an auxiliary basis) 'may provide further computational savings'; none of these compression routes has been developed within this framework.

Taming this bottleneck is exactly what the framework's target applications need, because they request very large numbers of matrix elements between excited configurations: on-the-fly implementations of NOCI post-processing methods — NOCI-PT2 (second-order perturbation theory on NOCI references, Burton and Thom 2020), NOCI-MP2 (a Møller–Plesset-type second-order variant), and NOCIS (nonorthogonal configuration interaction singles for core-level spectra with orbital relaxation) — and the evaluation of coupling terms (Hamiltonian and property, e.g. transition dipoles) between large excited state-specific multiconfigurational wavefunctions, such as state-specific CASSCF (complete active space self-consistent field) states whose orbitals differ from state to state. As of 2026, such applications exist only at demonstration scale — transition dipoles between state-specific CASSCF states of ethylene with a (2e, 2o) active space, and spin-generator-coordinate NOCI with NOCI-PT2 for two-electron systems in small basis sets — while adjacent efforts (compressed excitation spaces, alternative Wick reformulations, direct $O(n^{4})$ evaluation of determinant pairs) bypass rather than reduce the intermediate cost, and the post-NOCI method-development agenda is explicitly deferred in the most recent literature.

Problem Statement

The problem is to remove the two-electron intermediate bottleneck of the generalized nonorthogonal matrix-element framework so that large-scale application becomes practical. Concretely: (1) derive reduced-complexity representations of the two-electron intermediates that reduce the $O(16n^{5})$ pre-computation work and the up-to-$256n^{4}$ per-determinant-pair storage beyond what the framework already provides — the reduced set of $(m_{i},m_{j},m_{k},m_{l})$ combinations when $m<4$ (in particular $m=0$) and active-space restriction are the existing baseline, not the target — while preserving the accuracy and numerical stability of the exact determinantal matrix elements to within a quantified, user-controllable compression error (the representation being exact in the untruncated limit, with the tolerance and test regime stated; an exact factorization that does not reduce the $O(16n^{5})$ work or the up-to-$256n^{4}$ storage does not qualify); the Cholesky decomposition and the resolution-of-the-identity are the routes the source names as possible savings, and other reduced-complexity representations are equally admissible. (2) Develop the memory/work trade-off and intermediate-reuse algorithmics this requires, including, where high excitation ranks are needed, strategies for the $O(L^{6}\binom{L+2}{m})$ per-element prefactor; since a low-rank representation replaces the pre-computed four-index intermediates by a sum over an auxiliary index whose count grows with the basis, the per-element scaling of the compressed scheme is to be established relative to both the uncompressed $O(1)$-in-$n$ framework and the classical $O(n^{2})$/$O(n^{4})$ generalized Slater–Condon route, rather than assumed to retain strict $O(1)$ behaviour.

The source presents this reduction of the two-body intermediates as the enabler for its target applications — on-the-fly evaluation for NOCI post-processing methods (NOCI-PT2, NOCI-MP2, NOCIS-type) and couplings between excited state-specific multiconfigurational wavefunctions — which it frames as future method development that rapid matrix-element evaluation 'will allow' or 'will enable', not as part of the intermediate question itself.

References

1. H. G. A. Burton, 'Generalized nonorthogonal matrix elements. II: Extension to arbitrary excitations', J. Chem. Phys. 157, 204109 (2022). DOI: 10.1063/5.0122094, https://doi.org/10.1063/5.0122094 (arXiv:2208.10208)
2. H. G. A. Burton, 'Generalized nonorthogonal matrix elements: Unifying Wick's theorem and the Slater–Condon rules', J. Chem. Phys. 154, 144109 (2021). DOI: 10.1063/5.0045442, https://doi.org/10.1063/5.0045442
3. H. G. A. Burton and A. J. W. Thom, 'Reaching Full Correlation through Nonorthogonal Configuration Interaction: A Second-Order Perturbative Approach', J. Chem. Theory Comput. 16, 5586–5600 (2020). DOI: 10.1021/acs.jctc.0c00468, https://doi.org/10.1021/acs.jctc.0c00468
4. H. G. A. Burton, 'LibGNME: A C++ library for evaluating non-orthogonal matrix elements in electronic structure', open-source software (2021–2025). https://github.com/hgaburton/libgnme
5. S. Saade and H. G. A. Burton, 'Excited State-Specific CASSCF Theory for the Torsion of Ethylene', J. Chem. Theory Comput. 20, 5105–5114 (2024). DOI: 10.1021/acs.jctc.4c00212, https://doi.org/10.1021/acs.jctc.4c00212
6. A. Marie and H. G. A. Burton, 'Excited States, Symmetry Breaking, and Unphysical Solutions in State-Specific CASSCF Theory', J. Phys. Chem. A 127, 4538–4552 (2023). DOI: 10.1021/acs.jpca.3c00603, https://doi.org/10.1021/acs.jpca.3c00603
7. A. Ayati, H. G. A. Burton, P. Bultinck, and S. De Baerdemacker, 'Spin-generator coordinate method for electronic structure', J. Chem. Phys. 163, 044108 (2025). DOI: 10.1063/5.0275507, https://doi.org/10.1063/5.0275507
8. A. Ayati, H. G. A. Burton, and S. De Baerdemacker, 'Spin-Symmetry Projected Constrained Unrestricted Hartree–Fock', J. Chem. Theory Comput. 21, 5409–5424 (2025). DOI: 10.1021/acs.jctc.5c00068, https://doi.org/10.1021/acs.jctc.5c00068
9. C. Sun, F. Gao, and G. E. Scuseria, 'Selected Nonorthogonal Configuration Interaction with Compressed Single and Double Excitations', J. Chem. Theory Comput. 20, 3741–3748 (2024). DOI: 10.1021/acs.jctc.4c00240, https://doi.org/10.1021/acs.jctc.4c00240
10. F. Kossoski and P.-F. Loos, 'State-Specific Configuration Interaction for Excited States', J. Chem. Theory Comput. 19, 2258–2269 (2023). DOI: 10.1021/acs.jctc.3c00057, https://doi.org/10.1021/acs.jctc.3c00057
11. J. E. Arias-Martinez, H. Wu, and M. Head-Gordon, 'Generalization of One-Center Nonorthogonal Configuration Interaction Singles to Open-Shell Singlet Reference States: Theory and Application to Valence-Core Pump-Probe States in Acetylacetone', J. Chem. Theory Comput. 20, 752–766 (2024). DOI: 10.1021/acs.jctc.3c01139, https://doi.org/10.1021/acs.jctc.3c01139
12. C. Giuliani, J. Nys, R. Martinazzo, G. Carleo, and R. Rossi, 'Precise Quantum Chemistry calculations with few Slater Determinants', Nat. Commun. 17, 5613 (2026). DOI: 10.1038/s41467-026-70255-z, https://doi.org/10.1038/s41467-026-70255-z
13. A. M. Tokmachev, 'Separation of metric in Wick's theorem', J. Chem. Phys. 159, 194112 (2023). DOI: 10.1063/5.0174998, https://doi.org/10.1063/5.0174998
14. M. M. F. de Moraes and L. M. Thompson, 'Natural excitation framework for defining the external space: Uncontracted and internally contracted multireference nonorthogonal wavefunction theories', J. Chem. Phys. 164, 014107 (2026). DOI: 10.1063/5.0304585, https://doi.org/10.1063/5.0304585
15. L. Restaino, G. Gamboni, E. Selenius, and G. Levi, 'Orbital-optimized density functional calculations of excited electronic states: Recent advances and perspectives', arXiv:2606.13654 (2026). https://arxiv.org/abs/2606.13654