Frank and Frank (2020) showed that a convolutional neural network (CNN) trained on small tiles of digitized paintings can distinguish Rembrandt's and van Gogh's hands from other artists' with roughly 90% accuracy and can produce within-painting probability maps suggesting where different hands contributed. The study's open problem is the interpretation of such maps: a region classified as "not by the master" is compatible with several unrelated causes (a workshop assistant, later restoration, retouching of the digitized image, or an atypical passage by the master himself), and the network's discriminative cues resist reverse engineering, so no validated method can discriminate these causes or state what features drive the decision. This record poses the problem: develop and validate interpretability and causal-disambiguation methods for region-level CNN attribution and quantify cross-genre generalization, so that computational attribution can serve as defensible evidence about a painting's creation rather than an invitation to storytelling.

Background

Since the mid-2000s, computer vision and machine learning have been applied to questions that art historians traditionally answered by connoisseurship — expert judgment of authorship from close visual study. Early work used hand-designed, interpretable descriptors: wavelet statistics of high-resolution scans to identify van Gogh (Johnson et al. 2008), automatically extracted brushstrokes whose rhythmic statistics separated van Gogh from his contemporaries (Li et al. 2012), and stroke-level analysis of drawings for attribution and forgery detection (Elgammal et al. 2018). A second wave used convolutional neural networks (CNNs) — image classifiers whose features are learned rather than designed — trained on digitized artworks: PigeoNET (van Noord et al. 2015) classified paintings by artist and, using occlusion sensitivity (systematically masking image regions and measuring the change in the classifier's output to map which regions matter), could attribute regions of a work to different hands.

Frank and Frank (2020) trained CNNs on small tiles (image patches on the order of 100–150 pixels) of Rembrandt portraits and van Gogh landscapes, reaching roughly 90% test accuracy in distinguishing the master from comparable artists, and aggregated tile-level classification probabilities into within-painting probability maps — images in which each region is colored by the probability that the master painted it. Such maps suggested mixed authorship (regions likely painted by workshop assistants), restoration, and other creation stories. Rembrandt's studio practice makes this concrete: he ran an active workshop whose tuition-paying students copied his works and assisted in production, so many paintings plausibly contain passages by other hands.

The paper's central unresolved problem is the interpretation of these maps. First, a "not-master" classification of a region is causally ambiguous: the region may have been painted by an assistant or collaborator; it may have been altered by later restoration (conservation treatment such as cleaning, retouching, or structural repair) or by retouching of the digitized photograph itself (for example, to remove glare); or it may be an atypical passage by the master. The method offers no way to discriminate these causes. Second, the network's cues are opaque: unlike object recognition, where visualizing intermediate layers reveals recognizable features (a cat's ears), the subtle brushstroke and texture statistics that distinguish an artist's hand "do not cohere into any sort of unifying theme" and resist reverse engineering; spatial importance maps (occlusion, saliency) say where the network looks, not what feature it uses or which cause it evidences. Third, generalization is doubly unquantified: the paper's single anecdotal test of a model trained on landscapes against van Gogh's still life "Three Pairs of Shoes" carried what the authors called "intrinsic risks," and the full range of an artist's style and the limitless ways it can be imitated always surpass any training corpus, so the boundary of trustworthy application is unknown.

Later literature confirms this is a live, unsolved interface. An overview by Castellano and Vessio (2021) states that although saliency and attention maps can highlight neural responses, "explaining the logic behind what the algorithm discovers is still very difficult." Controlled studies with paintings by known art students, scanned with chromatic confocal optical profilometers (instruments measuring surface-height topography), show that different hands can be distinguished from surface relief alone (Ji et al. 2021) — realizing the controlled-experiment direction for different artists, but not the cause taxonomy on real Old Master works. The most direct engagement to date (Afifi et al. 2025) adds ensemble uncertainty and confidence maps for Rubens and localizes probable master-versus-workshop zones, yet still resolves ambiguous cases by expert narrative — attributing classifier uncertainty on "Samson and Delilah" simultaneously to a 1982 restoration and to the painting's stylistic eclecticism — and positions the system as an auxiliary tool requiring specialist validation.

Problem Statement

Determine what a region-level "not-master" classification by a tile-based painting-attribution CNN means, by jointly solving, with validation, the following source-faithful subquestions:

  1. Causal disambiguation. Given a trained attribution model and a region of a painting classified as not by the master, develop and validate a method that discriminates among — and where the evidence permits, identifies — the compatible causes: execution by a workshop assistant or collaborator; later restoration, retouching, or revision; artifacts of the digitized image (such as photographic retouching to remove glare); or an atypical passage by the master himself. Validation must demonstrate the discrimination rather than rely on the narrative plausibility of a creation story; as the source notes, the compatible explanations coexist until resolved by records or authoritative scholarship — if ever — so evidence may range from controlled experiments with known hands to painstaking corroboration from conservation records and scholarly catalogues.

  2. Interpretability. Characterize, in human-verifiable terms, the visual features the trained network actually uses to distinguish the master's hand — going beyond spatial importance maps (occlusion, saliency, ensemble disagreement), which indicate where the network looks but not what it uses — with demonstrated faithfulness to the model's decisions.

  3. Generalization scope. Quantify the reliability of transferring a model trained on one genre of an artist's work (e.g., van Gogh landscapes) to other genres (e.g., still lifes), and characterize the training-distribution conditions under which the resulting attribution probabilities are dependable.

These deliverables jointly serve one overarching objective: making the meaning of a "not-master" region classification determinate and scientifically defensible, so that computational attribution can serve as evidence about a painting's creation. The full range of an artist's style and the limitless ways it can be imitated always surpass the training distribution, so validity claims hold only within characterized limits.

References

  1. S. J. Frank and A. M. Frank (2020). Analysis of Dutch Master Paintings with Convolutional Neural Networks. arXiv:2002.05107, DOI 10.48550/arXiv.2002.05107. https://arxiv.org/abs/2002.05107
  2. C. Johnson, E. Hendriks, I. Berezhnoy, E. Brevdo, S. Hughes, I. Daubechies, J. Li, E. Postma, and J. Wang (2008). Image processing for artist identification. IEEE Signal Processing Magazine 25(4):37-48, DOI 10.1109/MSP.2008.923513. https://doi.org/10.1109/MSP.2008.923513
  3. J. Li, L. Yao, E. Hendriks, and J. Z. Wang (2012). Rhythmic Brushstrokes Distinguish van Gogh from His Contemporaries: Findings via Automated Brushstroke Extraction. IEEE Transactions on Pattern Analysis and Machine Intelligence 34(6):1159-1176, DOI 10.1109/TPAMI.2011.203. https://doi.org/10.1109/TPAMI.2011.203
  4. N. van Noord, E. Hendriks, and E. Postma (2015). Toward Discovery of the Artist's Style: Learning to recognize artists by their artworks. IEEE Signal Processing Magazine 32(4):46-54, DOI 10.1109/MSP.2015.2406955. https://doi.org/10.1109/MSP.2015.2406955
  5. Elgammal, Kang, and Den Leeuw (2018). Picasso, Matisse, or a Fake? Automated Analysis of Drawings at the Stroke Level for Attribution and Authentication. Proceedings of the AAAI Conference on Artificial Intelligence 32(1), DOI 10.1609/aaai.v32i1.11313. https://doi.org/10.1609/aaai.v32i1.11313
  6. G. Castellano and G. Vessio (2021). Deep learning approaches to pattern extraction and recognition in paintings and drawings: an overview. Neural Computing and Applications 33(19):12263-12282, DOI 10.1007/s00521-021-05893-z. https://doi.org/10.1007/s00521-021-05893-z
  7. F. Ji, M. S. McMaster, S. Schwab, G. Singh, L. N. Smith, S. Adhikari, M. O'Dwyer, F. Sayed, A. Ingrisano, D. Yoder, E. S. Bolman, I. T. Martin, M. Hinczewski, and K. D. Singer (2021). Discerning the painter's hand: machine learning on surface topography. Heritage Science 9, article 152, DOI 10.1186/s40494-021-00618-w. https://doi.org/10.1186/s40494-021-00618-w
  8. S. J. Frank and A. M. Frank (2022). Analysis, attribution, and authentication of drawings with convolutional neural networks. International Journal of Arts and Technology 14(3), p. 192, DOI 10.1504/ijart.2022.128455. https://doi.org/10.1504/ijart.2022.128455
  9. Y.-S. Choi, J. W. Kim, S. Jung, H.-J. Jeon, and J. S. Kim (2022). Efficient Analytical Data Processing and Visualization Methods for Supporting Scientific Connoisseurship on Oil Paintings. Proceedings of the 2022 13th International Conference on Information and Communication Technology Convergence (ICTC), pp. 2373-2376, DOI 10.1109/ICTC55196.2022.9952845. https://doi.org/10.1109/ICTC55196.2022.9952845
  10. L. Schaerf, C. Popovici, and E. Postma (2023). Art Authentication with Vision Transformers. arXiv:2307.03039, DOI 10.48550/arXiv.2307.03039. https://arxiv.org/abs/2307.03039
  11. S. Boccuzzo, D. D. Meyer, and L. Schaerf (2024). Art Forgery Detection using Kolmogorov Arnold and Convolutional Neural Networks. arXiv:2410.04866, DOI 10.48550/arXiv.2410.04866. https://arxiv.org/abs/2410.04866
  12. T. Fu, J. Conde, G. Martínez, P. Reviriego, E. Merino-Gómez, and F. Moral (2025). Artificial Intelligence and Misinformation in Art: Can Vision Language Models Judge the Hand or the Machine Behind the Canvas? arXiv:2508.01408, DOI 10.48550/arXiv.2508.01408. https://arxiv.org/abs/2508.01408
  13. A. Van Vijle, P. Hacıgüzeller, and G. Van der Snickt (2025). Machine learning for painting conservation: a state-of-the-art review. npj Heritage Science 13, article 437, DOI 10.1038/s40494-025-01924-3. https://doi.org/10.1038/s40494-025-01924-3
  14. A. Afifi, A. Kalimullin, S. Korchagin, and I. Kudryashov (2025). A deep learning perspective on Rubens' attribution. arXiv:2511.22667, DOI 10.48550/arXiv.2511.22667. https://arxiv.org/abs/2511.22667
  15. N. Fanelli, P. De Marinis, R. Scaringi, E. Cetinic, G. Vessio, and G. Castellano (2026). Understanding How MLLMs Describe Artworks Using Token Activation Maps. arXiv:2606.27947, DOI 10.48550/arXiv.2606.27947. https://arxiv.org/abs/2606.27947