Global functional independence and optimization-time coupling of the POLCA Net multi-objective losses

POLCA Net (Martin H., Perozo and Lopez, 2024) is an autoencoder trained with a composite multi-objective loss — reconstruction, latent orthogonality, a position-weighted 'center of mass' of latent variances that orders components by importance, and total-variance regularization — designed to reproduce PCA's orthogonal, variance-ordered latent space in a nonlinear regime. Its only theoretical support is a constructive, local argument (Appendix B) that the three losses it analyzes — center of mass, total variance and reconstruction — are functionally independent, and the authors themselves flag as unresolved: whether independence holds globally rather than only in a neighborhood of explicit latent-space transformations, whether it survives capacity-constrained autoencoders whose decoders cannot compensate exactly for a latent transformation, and what indirect couplings specific architectures and optimization procedures induce. The paper also measures pairwise gradient conflicts during training (gradient cosine similarity below $-0.01$), but descriptively only: it draws no conclusion from them about convergence, attainable trade-offs or weight scheduling. The problem asks for a correct, global characterization of when and how the POLCA losses are functionally independent or effectively redundant — including a repair of the source's internally inconsistent definition of the center-of-mass loss — together with the capacity-, architecture- and optimizer-dependent couplings that the source's local argument leaves open. Here the losses are effectively redundant when one is functionally determined by the others, so that no term supplies a distinct optimization signal, or when omitting it from the fixed-weight combination leaves the attained optimum unchanged over the autoencoder class.

Background

Principal Component Analysis (PCA) decomposes data into orthogonal latent directions ordered by explained variance, but it is a strictly linear map. An autoencoder — an encoder–decoder neural network trained to reconstruct its input through a low-dimensional latent (hidden-feature) representation — is the natural nonlinear vehicle for recovering such structure, and several autoencoder lines pursue exactly this: PCA-AE (Ladjal, Newson and Pham, 2019) penalizes the latent covariance to obtain approximately independent components, and the Autoencoder with Ordered Variance (AEO; Augustine, Patil, Bhushan and Bhartiya, 2024) adds a variance-ordering regularizer. POLCA Net (Martin H., Perozo and Lopez, 2024) pursues the same goal with a composite multi-objective loss

$\mathcal{L}_{\text{polca}}=\mathcal{L}_{\text{rec}}+\mathcal{L}_{\text{class}}+\alpha\,\mathcal{L}_{\text{ort}}+\beta\,\mathcal{L}_{\text{com}}+\gamma\,\mathcal{L}_{\text{var}},$

where $\mathcal{L}_{\text{rec}}$ is the mean-squared reconstruction error, $\mathcal{L}_{\text{class}}$ an optional classification loss, $\mathcal{L}_{\text{ort}}$ the average squared off-diagonal cosine similarity between latent components (enforcing orthogonality), $\mathcal{L}_{\text{com}}$ a position-weighted 'center of mass' of the per-dimension latent variances that concentrates information in the earlier latent dimensions, $\mathcal{L}_{\text{var}}$ the total latent variance (which prevents gaming the center-of-mass term by collapsing all variance), and $\alpha,\beta,\gamma$ are loss weights. On 16 image datasets the authors report consistent improvements over PCA in reconstruction and linear-classification metrics, and they monitor cross-loss interactions during training via the cosine similarity $s$ between loss gradients, calling a pair conflicting when $s<-0.01$ — a detection heuristic in the spirit of the multi-task gradient-conflict literature (e.g., PCGrad, which resolves detected conflicts by projecting a task's gradient so that it no longer opposes another task's; Yu, Kumar, Gupta, Levine, Hausman and Finn, 2020). The weights are fixed constants in the source's experiments ($\alpha=\beta=1\mathrm{e}{-2}$, $\gamma=1\mathrm{e}{-7}$; Table 1) and the reported conflict statistics (Figure 3) are descriptive: the conclusion records only that the analysis of the multi-objective loss 'revealed interactions between different loss components during training'.

The paper's only theoretical justification for combining the losses is an appendix-level constructive argument that the three losses it analyzes — $L_{\text{com}}$, $L_{\text{var}}$ and the reconstruction loss $L_2$ — are 'functionally independent' in the sense of the source's Appendix A, Definition 3 (after Hirsch, 1976 and Lee, 2012): that no non-trivial function $\Phi$ satisfies $\Phi(L_{\text{com}},L_{\text{var}},L_2)=0$ for all possible autoencoders. The orthogonality loss $\mathcal{L}_{\text{ort}}$ is part of neither the claim nor the proof. The argument proceeds by three explicit transformations (a permutation of latent coordinates, a latent scaling, and an additive offset on the reconstruction) that change one loss while leaving the others fixed. The authors themselves list the gaps: the scaling step assumes the decoder has enough capacity to compensate exactly; the transformations are local (independence is shown only in a neighborhood of a given configuration); and indirect couplings induced by specific architectures or optimizers are not considered. Definition 3, however, fixes neither the admissible regularity of $\Phi$ nor the meaning of 'non-trivial'. The appendix defines $L_{\text{com}}=\frac{1}{d}\sum_{i=1}^{d} i\,\sigma_{(i)}^2$ over variances $\sigma_{(i)}^2$ sorted in descending order, under which the permutation step is vacuous because sorted variances are invariant under coordinate permutations, whereas the main text (Section 2.1) defines $\mathcal{L}_{\text{com}}$ as the normalized ratio $\sum_{i=0}^{n-1} i^{1+\epsilon}\,\mathrm{Var}(z_i)/(n\,\mathcal{L}_{\text{var}})$ with $\epsilon=0.25$ over unsorted, position-weighted variances — two different functionals, differing in the index range ($i=0,\dots,n-1$ versus $i=1,\dots,d$, so that the leading latent coordinate carries weight $0^{1.25}=0$ in the main-text form), in the normalization (the dimensionless ratio by $n\,\mathcal{L}_{\text{var}}$ versus the factor $1/d$), in the position exponent ($1+\epsilon$ versus $1$) and in the sorting; only the second is compatible with the proof strategy. Whether the POLCA losses are genuinely independent, over which class of autoencoders, and under which capacity-, architecture- and optimizer-induced couplings they become effectively redundant is therefore open even at the level of a correct self-consistent statement.

Problem Statement

The open problem is the question left by the source's constructive argument and the limitations its authors attach to it: fully characterize when and how the POLCA Net losses are functionally independent, and when they interact or become effectively redundant under constrained capacity and under the specific architectures and optimization schemes that the source's local argument does not cover. The open sub-questions are the following.

Functional independence, stated correctly and globally. First fix a self-consistent formalization of the component losses — reconciling the main-text ratio form $\mathcal{L}_{\text{com}}=\sum_{i=0}^{n-1} i^{1+\epsilon}\mathrm{Var}(z_i)/(n\,\mathcal{L}_{\text{var}})$ with $\epsilon=0.25$ and the appendix's sorted form $L_{\text{com}}=\frac{1}{d}\sum_{i=1}^{d} i\,\sigma_{(i)}^2$, whose inconsistency undermines the source's constructive argument. Because the main-text form is the loss actually optimized in the source's experiments while the sorted form appears only inside the appendix argument, the reconciliation must state which conventions it adopts — the index range and the resulting zero weight on the leading latent coordinate, the dimensionless normalization by $n\,\mathcal{L}_{\text{var}}$ versus the factor $1/d$, the position exponent $1+\epsilon$ versus $1$, and whether the variances are sorted in descending order — since each choice yields a different functional and hence a different truth value for the independence claim. The formalization must equally specify the class of autoencoders and data distributions considered and the admissible sense of $\Phi$: the source's Definition 3 fixes neither the regularity of $\Phi$ nor the meaning of 'non-trivial', and the verdict depends on that choice, because for an arbitrary (possibly discontinuous) $\Phi$ the distance to the loss-value image of the class vanishes identically whenever that image is a proper subset of $\mathbb{R}^k$, whereas smooth or analytic $\Phi$ can admit genuine independence. The truth value of the independence claim is therefore relative to the adopted convention.

Then determine, for the three functionals the source analyzes, whether there exists a non-trivial function $\Phi$ with $\Phi(L_{\text{com}},L_{\text{var}},L_2)=0$ (its $L_2$ being the reconstruction loss $\mathcal{L}_{\text{rec}}$) for every autoencoder in the class, or whether these losses are functionally independent — globally rather than only in a neighborhood of explicit latent-space transformations, and in particular for capacity-constrained architectures where the decoder-compensation assumption fails. The source quantifies over 'all possible autoencoders', so a specified class is admissible only as a faithful, non-degenerate sub-domain of that setting: it must cover the source's own POLCA architectures and datasets together with capacity-constrained variants, and a class consisting of a single configuration, or of finitely many, does not answer the question, since a $\Phi$ read off the loss values of those configurations vanishes on the whole class without revealing any coupling. The four-loss relation $\Phi(\mathcal{L}_{\text{ort}},\mathcal{L}_{\text{com}},\mathcal{L}_{\text{var}},\mathcal{L}_{\text{rec}})=0$ is a strictly stronger proposition that the source neither claims nor attempts — independence of a subset does not imply independence of a superset — so it may be pursued as an announced generalization but does not replace the three-loss question, and a three-loss answer leaves the coupling of $\mathcal{L}_{\text{ort}}$ open. Finally, identify the architectures, capacity and data regimes, and optimizer classes under which indirect couplings or effective redundancy among the losses arise; resolving this last part requires a criterion that decides, for a specified such class, whether the losses become effectively redundant, rather than a single example in which coupling is observed.

The source additionally lists the empirical generality of POLCA Net across broader modalities, larger-scale problems, and downstream tasks as future work; that is an open-ended evaluation axis, not a bounded deliverable of this problem, and progress on it (e.g., external baseline evaluations) neither completes nor narrows the loss-interaction objective above.

References

1. Jose Antonio Martin H., Freddy Perozo, Manuel Lopez (2024). Principal Orthogonal Latent Components Analysis (POLCA Net). arXiv:2410.07289, DOI: 10.48550/arXiv.2410.07289. https://arxiv.org/abs/2410.07289
2. Saïd Ladjal, Alasdair Newson, Chi-Hieu Pham (2019). A PCA-like Autoencoder. arXiv:1904.01277, DOI: 10.48550/arXiv.1904.01277. https://arxiv.org/abs/1904.01277
3. Midhun T. Augustine, Parag Patil, Mani Bhushan, Sharad Bhartiya (2024). Discovering Nonlinear Static Relationships in Unlabeled Dataset using Autoencoder with Ordered Variance. arXiv:2402.14031, DOI: 10.48550/arXiv.2402.14031. https://arxiv.org/abs/2402.14031
4. Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, Chelsea Finn (2020). Gradient Surgery for Multi-Task Learning. arXiv:2001.06782, DOI: 10.48550/arXiv.2001.06782. https://arxiv.org/abs/2001.06782