A 2020 ACM TOG paper introduced a differentiable compositing operator that recovers the layered structure of a 2D graphic pattern — which image patches constitute its elements and where, in what orientation, color, and depth order each instance sits — but requires the user to mark one exemplar of every element type and only handles patterns made of exact repetitions of those patches. Later work by the same authors (Search for Concepts, BMVC 2022) and by others (MarioNette, NeurIPS 2021) showed that the element patch library and the element parameters can be jointly discovered without exemplars when element instances are exact repetitions under rigid transformations. The open problem is the remaining core: joint, exemplar-free discovery of pattern structure at the generality of real graphic patterns — elements whose shapes vary continuously rather than repeating exactly, robust recovery of faint, few-pixel, and near-rotationally-symmetric elements whose gradients are weak or ambiguous, and a compositing formulation with per-pixel rather than per-element depth so non-planar element surfaces and local inter-element layering can be represented.

Background

In computer graphics, a pattern is a collection of graphical elements arranged in a regular or near-regular arrangement over a background, ubiquitous in graphic art, textile and surface design. When such a pattern is stored as a flat raster image, the underlying structure — which distinct motifs exist and how each instance is placed — is lost, making structure-preserving editing tedious. Pixel-level deep generative or editing methods tend to distort the individual element shapes or their arrangement, because they have no representation of the repeated discrete elements.

Reddy, Guerrero, Fisher, Li and Mitra (ACM Trans. Graph. 39(6), 2020) addressed this with a differentiable compositing operator. The pattern is modeled as a composition of pattern elements: each element is an instance of one patch from a small library H\mathcal{H} of atomic image patches (one exemplar per element type, e.g. one flower and one leaf), transformed by continuous element parameters E\mathcal{E} — position, orientation, and color — and assigned a scalar depth that fixes the occlusion order in which elements are alpha-composited (alpha compositing is the standard operation of blending semi-transparent image layers back-to-front). The paper's key contribution is a soft, differentiable relaxation of this compositing function, so that gradients of an image-reconstruction loss with respect to the element parameters can guide a gradient-based search. This turns pattern decomposition into a high-dimensional version of template matching (finding where a known template appears in an image), handles element orientation and color that classical template matching cannot, and supports applications such as pattern expansion to a larger canvas.

The method, however, has stated restrictions. The user must manually mark one instance of each element type in advance, because the patch library H\mathcal{H} is extracted from those markings rather than discovered; the paper lists solving for H\mathcal{H} jointly with E\mathcal{E} as an interesting avenue for future work. The compositing model assigns a single depth per element, so non-planar element surfaces and local layering between overlapping parts of one element cannot be represented; a per-pixel depth within an element's occupancy is suggested instead. The output domain is restricted to patterns composed of repeated instances of a small set of atomic patches, excluding patterns with continuously changing element shapes (e.g. gradual deformations between a circle and a rectangle). Finally, the optimization is nonconvex with no global-optimum guarantee: faint elements or elements with very few visible pixels produce weak gradients and can be missed, and elements close to rotationally symmetric produce strong local minima in orientation space, yielding incorrectly oriented instances.

Subsequent work resolved parts of this. MarioNette (Smirnov, Gharbi, Fisher, Guizilini, Efros and Solomon, NeurIPS 2021) jointly learns a dictionary of possibly transparent sprites together with a network that places them, self-supervised, for sprite-based animations — joint element-library discovery via amortized inference, i.e. training a predictor network across a dataset instead of optimizing per input. Search for Concepts (Reddy, Guerrero and Mitra, BMVC 2022), by the original authors, performs the joint discovery by direct, unamortized optimization: it alternates between gradient-based updates of a dictionary of RGBA 'visual concept' patches (RGBA = red–green–blue plus a per-pixel transparency channel) and their placement parameters (translation, rotation, hue, and a discrete occlusion-layer order found by swap search), growing the dictionary by cloning frequently used concepts. It explicitly positions itself against the 2020 paper's manual input ('extensive information about the content of the decomposed elements is needed as input') and demonstrates the joint discovery on synthetic object-decomposition benchmarks (Tetrominoes, Multi-dSprites, CLEVR6, MNIST, 3D sprites) — but only for exact repetitions of elements, explicitly deferring deformation-tolerant concept discovery to future work. On the generative side, Im2Vec (Reddy, Gharbi, Lukáč and Mitra, CVPR Workshops 2021) embeds the differentiable compositor inside a variational generative model for vector graphics, and Riso, Sforza and Pellacini (Computer Graphics Forum 2022) plus Riso, Vecchio and Pellacini (STAG 2025; journal version spEx, Computers & Graphics 2026) couple structured, differentiable pattern representations with diffusion generative models (networks trained to generate images by reversing a gradual noising process) for pattern expansion — realizing the paper's suggestion to use the compositor inside generative modeling, with DiffBMP (arXiv 2602.22625, 2026) additionally optimizing scale and opacity of bitmap primitives. No existing work addresses the full generality of the original setting for artistic patterns: joint exemplar-free discovery when element shapes vary continuously, robustness in the weak-gradient failure regimes, and per-pixel depth in the compositing model.

Problem Statement

The open core of the 2020 paper's stated limitations, after the exact-repetition rigid-transform case was resolved by Search for Concepts (BMVC 2022) and MarioNette (NeurIPS 2021) and the generative-integration and scale/transparency suggestions were pursued (Im2Vec 2021; pOp 2022; Riso et al. STAG 2025 / Computers & Graphics 2026; DiffBMP 2026), is the following single objective: fully unsupervised discovery of pattern structure in the differentiable-compositing formulation at the generality of real graphic patterns. Concretely, given only raw pattern images — with no user-marked element exemplars — recover jointly the element patch library H\mathcal{H} (a small set of atomic image patches, one exemplar per element type) and all element parameters E\mathcal{E} (for each instance: its element type, its position, orientation, color, and occlusion-layer order (i.e. which front-to-back compositing layer it occupies), and its continuous shape/deformation state relative to the exemplar of its element type) such that the following restrictions of the original formulation are removed: (i) element instances are not exact repetitions of a small set of atomic patches but vary continuously in shape across the pattern (non-rigid deformation between instances, e.g. gradual shape interpolation), requiring a principled treatment of the ill-posedness in deciding whether two differing instances belong to one deforming element type or two distinct types; (ii) the discovery is robust in the regimes where the original optimization struggles — faint elements, elements with very few visible pixels, and elements close to rotationally symmetric, whose weak or ambiguous gradients cause missed or incorrectly oriented instances; and (iii) the compositing model supports a depth value per pixel of an element's occupancy rather than one depth per element, so that non-planar element surfaces and local inter-element layering can be represented and recovered. These three components serve one common objective — removing the remaining manual input and domain restrictions that separate a flat pattern image from an automatically discovered, editable layered representation. For a full resolution, a single joint discovery pipeline must remove all three restrictions together; work that addresses only one or two of them is partial progress toward the objective, not a resolution of it. The question asks for a formulation and algorithm with demonstrated reach over these regimes, together with whatever guarantees or failure-mode characterizations can be established for the joint nonconvex optimization.

References

  1. Pradyumna Reddy, Paul Guerrero, Matt Fisher, Wilmot Li, Niloy J. Mitra. Discovering pattern structure using differentiable compositing. ACM Transactions on Graphics 39(6), 1–15 (2020). DOI: 10.1145/3414685.3417830. https://doi.org/10.1145/3414685.3417830 (arXiv:2010.08788, https://arxiv.org/abs/2010.08788)
  2. Pradyumna Reddy, Paul Guerrero, Niloy J. Mitra. Search for Concepts: Discovering Visual Concepts Using Direct Optimization. arXiv:2210.14808 (2022); published as 'Search for Concepts: Learning Visual Concepts Using Direct Optimization', BMVC 2022. https://arxiv.org/abs/2210.14808
  3. Dmitriy Smirnov, Michaël Gharbi, Matthew Fisher, Vitor Guizilini, Alexei A. Efros, Justin Solomon. MarioNette: Self-Supervised Sprite Learning. NeurIPS 2021. arXiv:2104.14553. https://arxiv.org/abs/2104.14553
  4. Pradyumna Reddy, Michaël Gharbi, Michal Lukáč, Niloy J. Mitra. Im2Vec: Synthesizing Vector Graphics without Vector Supervision. IEEE/CVF CVPR Workshops 2021, 2124–2133. DOI: 10.1109/CVPRW53098.2021.00241. https://doi.org/10.1109/CVPRW53098.2021.00241
  5. Sharon Zhang, Jiaju Ma, Jiajun Wu, Daniel Ritchie, Maneesh Agrawala. Editing Motion Graphics Video via Motion Vectorization and Transformation. ACM Transactions on Graphics 42(6), 1–13 (2023). DOI: 10.1145/3618316. https://doi.org/10.1145/3618316
  6. Marzia Riso, Davide Sforza, Fabio Pellacini. pOp: Parameter Optimization of Differentiable Vector Patterns. Computer Graphics Forum 41(4), 161–168 (2022). DOI: 10.1111/cgf.14595. https://doi.org/10.1111/cgf.14595
  7. Marzia Riso, Giuseppe Vecchio, Fabio Pellacini. Structured Pattern Expansion with Diffusion Models. Smart Tools and Applications for Graphics (STAG), Eurographics Association (2025). DOI: 10.2312/stag.20251330. https://doi.org/10.2312/stag.20251330 (Eurographics Digital Library record: https://diglib.eg.org/handle/10.2312/stag20251330)
  8. Marzia Riso, Giuseppe Vecchio, Fabio Pellacini. spEx: Structured Pattern Expansion with Diffusion Models. Computers & Graphics 139, 104682 (2026). DOI: 10.1016/j.cag.2026.104682. https://doi.org/10.1016/j.cag.2026.104682
  9. Seongmin Hong, Junghun James Kim, Daehyeop Kim, Insoo Chung, Se Young Chun. DiffBMP: Differentiable Rendering with Bitmap Primitives. arXiv:2602.22625 (2026). https://arxiv.org/abs/2602.22625