X-radiographs of paintings executed on both sides of their support, or over earlier compositions, superimpose the content of both paintings in a single image. A 2019 Science Advances study introduced a self-supervised deep-learning method that separates such mixed X-radiographs using visible RGB photographs of the contributing sides as side information, demonstrated on the Ghent Altarpiece, and explicitly left open four issues: the unexplained dependence of the learned decomposition on input ordering (an optimization asymmetry), extension to paintings where one contributing composition is visually inaccessible, the incorporation of additional imaging modalities (macro X-ray fluorescence, hyperspectral imaging, spectroscopies) when RGB side information is incomplete or absent, and the replacement of per-pixel MSE/MAE by criteria that reflect expert judgment of correct feature assignment. Follow-up work by the same consortium resolved the concealed-design case in which the visible surface is available as an RGB image (Goya's Dona Isabel de Porcel), but a 2022 consortium survey and a 2025 review confirm that multimodal-guided separation, expert-aligned evaluation, and the asymmetry explanation remain open. This record states the open problem and its surviving core.

Background

X-radiography produces a single grayscale image recording the total attenuation of X-rays through a painting. When a painting is executed on both sides of its support — the wing panels of a polyptych (a multi-panel altarpiece such as the Ghent Altarpiece of 1432), a canvas reused by painting over an earlier composition, or a work with pentimenti (the artist's own revisions of the composition, whose earlier design survives beneath the final paint) — the X-radiograph superimposes the content of both compositions, so neither can be read cleanly by conservators and art historians. Under the standard approximation the observed image xx is the sum x≈x1+x2x \approx x_1 + x_2 of the X-ray images each composition would produce alone; recovering x1x_1 and x2x_2 from xx is severely underdetermined, so methods must exploit side information and structural priors. Earlier approaches include multi-modal dictionary learning (Deligiannis et al., 2017) and Gaussian-mixture-model source separation with side information (Sabetsarvestani et al., 2020), with only partial success.

Sabetsarvestani, Sober, Higgitt, Daubechies and Rodrigues (Science Advances, 2019) introduced a self-supervised deep-learning approach: a convolutional neural network trained, separately for each painting detail, to map the visible RGB photographs of the two sides to estimated X-ray images whose sum reconstructs the observed mixed X-radiograph — self-supervised in the sense that no ground-truth pairs of mixed and separated X-ray images exist or are used. On two details of the Ghent Altarpiece (the Adam and Eve panels) it dramatically improved on all prior separations. The paper's Discussion explicitly left open several issues. First, an unexplained optimization asymmetry: the learned decomposition depends on the ordering of the inputs, and the adopted procedure takes the first output of two runs that differ only in input ordering — behavior the authors could not explain beyond conjectures about the optimizer's implementation or multiple local minima — prompting a call for study of the network design and the evolution of its layers during training. Second, both demonstrated cases have the two compositions visible, whereas many important objects have a contributor that is not visible at all (concealed paintings, reused supports, pentimenti), for which the authors named Goya's Dona Isabel de Porcel, van Gogh's Patch of Grass, and Degas's Portrait of a Woman as examples. Third, when RGB side information is incomplete or absent, additional imaging modalities should be incorporated — macro X-ray fluorescence scanning (MA-XRF, which maps the spatial distribution of chemical elements at and below the paint surface), hyperspectral or reflectance imaging spectroscopy (RIS, which records a reflectance spectrum per pixel), infrared reflectography (IRR, an infrared imaging technique that reveals underdrawing beneath paint), and spectroscopies — requiring new algorithms capable of ingesting such complex, heterogeneous datasets. Fourth, per-pixel mean squared error / mean absolute error (MSE/MAE) is a poor optimization and evaluation criterion — degenerately, the pair consisting of the mixed image itself and a black image attains zero error — and criteria correlated with expert judgment of correct feature assignment and interpretability are needed.

Subsequent work by the same consortium addressed parts of this agenda. A connected-auto-encoder architecture whose encoders are obtained by algorithm unrolling (the conversion of the iterations of an optimization algorithm into trainable neural-network layers), specifically unrolled convolutional sparse coding (Pu et al., IEEE TIP 2023, arXiv 2020), improved the two-sided case but still requires RGB images of both sides. An unrolled network built from learned convolutional iterative shrinkage and thresholding (Pu et al., EUSIPCO 2021 and IEEE TIP 2022) extended separation to paintings with concealed designs using only the RGB image of the visible surface, demonstrated on Goya's Dona Isabel de Porcel — the exact painting named in the source. A survey by the same consortium (Sober et al., IEEE BITS 2022) still lists data fusion across imaging modalities among the open problems for the field, and a 2025 review of machine learning for painting conservation (Van Vijle, Haciguzeller and Van der Snickt, npj Heritage Science) confirms this state of the art and frames the application of separation methods to infrared reflectograms, MA-XRF element maps, and RIS outputs as future work expected to be harder still. No published method ingests such modalities into the self-supervised X-ray separation framework; no expert-aligned separation metric has been validated; and the optimization asymmetry is not revisited anywhere in the follow-up literature.

Problem Statement

The open problem, as posed by the Discussion of Sabetsarvestani et al. (2019) and left open by the subsequent literature, is to extend self-supervised separation of a mixed painting X-radiograph x≈x1+x2x \approx x_1 + x_2 beyond the regime in which a visible RGB photograph of every contributing composition is available as side information. Its coupled components are:

  1. Multimodal-guided separation. Develop self-supervised separation methods for cases in which one contributing composition is completely visually inaccessible (concealed paintings, reused supports, pentimenti) and the available guidance is not, or not only, an RGB image: algorithmic designs that ingest and fuse heterogeneous, high-dimensional imaging modalities — MA-XRF element distribution maps, hyperspectral/reflectance imaging spectroscopy datacubes, infrared reflectograms, and spectroscopic data — to drive the separation. (The subcase in which the visible surface is available as an RGB image has been demonstrated, on Goya's Dona Isabel de Porcel; the multimodal-guided regime is the open core of this component.)

  2. Expert-aligned optimization and evaluation criteria. Define and validate loss functions and quantitative evaluation metrics for this separation task that reflect the actual objective — correct assignment of X-ray features to the correct composition, as judged by conservators and art historians — replacing per-pixel MSE/MAE, which is known to fail (the pair consisting of the mixed image and a black image attains zero error).

  3. Understanding the optimization dynamics. Explain the optimization asymmetry observed in the original architecture — the dependence of the learned decomposition on the ordering of the inputs, and the empirical benefit of combining the first outputs of two order-swapped runs — which the source conjectured to arise from optimizer implementation details or multiple local minima but never established; more broadly, characterize the optimization dynamics and the layer-wise evolution during self-supervised training on which such separations rest.

Components 1 and 2 are, respectively, the capability objective and its validation dimension for one goal — trustworthy computational unmixing of painting X-radiographs in the practically common situations where visible side information is missing; component 3 concerns the reliability and understanding of the underlying optimization, raised by the source as a call for further study of the network design and layer evolution. Components 1 and 2 form one coupled target, while component 3 is an independent second target.

References

  1. Z. Sabetsarvestani, B. Sober, C. Higgitt, I. Daubechies, M. R. D. Rodrigues, "Artificial intelligence for art investigation: Meeting the challenge of separating x-ray images of the Ghent Altarpiece", Science Advances 5(8), eaaw7416 (2019). DOI: 10.1126/sciadv.aaw7416, https://doi.org/10.1126/sciadv.aaw7416
  2. N. Deligiannis, J. F. C. Mota, B. Cornelis, M. R. D. Rodrigues, I. Daubechies, "Multi-Modal Dictionary Learning for Image Separation With Application in Art Investigation", IEEE Transactions on Image Processing 26(2), 751-764 (2017). DOI: 10.1109/TIP.2016.2623484, https://doi.org/10.1109/TIP.2016.2623484
  3. Z. Sabetsarvestani, F. Renna, F. Kiraly, M. Rodrigues, "Source Separation With Side Information Based on Gaussian Mixture Models With Application in Art Investigation", IEEE Transactions on Signal Processing 68, 558-572 (2020). DOI: 10.1109/TSP.2020.2964195, https://doi.org/10.1109/TSP.2020.2964195
  4. W. Pu, B. Sober, N. Daly, C. Zhou, Z. Sabetsarvestani, C. Higgitt, I. Daubechies, M. R. D. Rodrigues, "Image Separation With Side Information: A Connected Auto-Encoders Based Approach", IEEE Transactions on Image Processing 32, 2931-2946 (2023). DOI: 10.1109/TIP.2023.3275872, https://doi.org/10.1109/TIP.2023.3275872 (preprint: arXiv:2009.07889, https://arxiv.org/abs/2009.07889)
  5. W. Pu, J.-J. Huang, B. Sober, N. Daly, C. Higgitt, I. Daubechies, P. L. Dragotti, M. R. D. Rodrigues, "Mixed X-Ray Image Separation for Artworks With Concealed Designs", IEEE Transactions on Image Processing 31, 4458-4473 (2022). DOI: 10.1109/TIP.2022.3185488, https://doi.org/10.1109/TIP.2022.3185488 (preprint: arXiv:2201.09167, https://arxiv.org/abs/2201.09167)
  6. W. Pu, J. Huang, B. Sober, N. Daly, C. Higgitt, P. L. Dragotti, I. Daubechies, M. R. D. Rodrigues, "A Learning Based Approach to Separate Mixed X-Ray Images Associated with Artwork with Concealed Designs", Proceedings of the 29th European Signal Processing Conference (EUSIPCO), 1491-1495 (2021). DOI: 10.23919/EUSIPCO54536.2021.9616096, https://doi.org/10.23919/EUSIPCO54536.2021.9616096
  7. B. Sober, S. Bucklow, N. Daly, I. Daubechies, P. L. Dragotti, C. Higgitt, J.-J. Huang, A. Pizurica, W. Pu, S. Reynolds, M. R. D. Rodrigues, C.-B. Schonlieb, S. Yan, "Revealing and Reconstructing Hidden or Lost Features in Art Investigation", IEEE BITS the Information Theory Magazine (2022). DOI: 10.1109/MBITS.2022.3207125, https://doi.org/10.1109/MBITS.2022.3207125
  8. C. Zhou, Z. Lyu, M. R. D. Rodrigues, "BITS-Net: Blind Image Transparency Separation Network", Proceedings of the 2023 IEEE International Conference on Image Processing (ICIP), 375-379 (2023). DOI: 10.1109/ICIP49359.2023.10222918, https://doi.org/10.1109/ICIP49359.2023.10222918
  9. A. Van Vijle, P. Haciguzeller, G. Van der Snickt, "Machine learning for painting conservation: a state-of-the-art review", npj Heritage Science 13 (2025). DOI: 10.1038/s40494-025-01924-3, https://doi.org/10.1038/s40494-025-01924-3
  10. Z. Preisler, R. Andolina, A. Busacca, C. Caliri, C. Miliani, F. P. Romano, "Deep learning for enhanced spectral analysis of MA-XRF datasets of paintings", Science Advances 10(39) (2024). DOI: 10.1126/sciadv.adp6234, https://doi.org/10.1126/sciadv.adp6234