Completing the statistical foundations of shape from texture via the Texture Gradient Equation: homogeneity characterization, two-dimensional consistency, identifiability, and non-Lambertian texture models

Shape from texture recovers the three-dimensional shape of a surface from a single image by analyzing how perspective projection metrically deforms the surface texture. Clerc and Mallat (2002) recast this as a statistical estimation problem: the image is a realization of a deformed stochastic texture, a warplet transform (a two-dimensional wavelet-like atom whose scalar scale parameter is replaced by an invertible warping matrix) yields a warpogram of local second moments, and this warpogram satisfies a transport equation — the Texture Gradient Equation — whose velocity converges, under a local homogeneity condition on the surface texture, to the deformation gradient from which surface orientation and depth are recovered. The authors themselves identify the unfinished foundations of this framework, which are not yet established. The open problem is to complete the estimation-theoretic foundations of this framework: (i) give necessary and sufficient geometric and stochastic conditions on surface-texture pairs for the texture-gradient velocity to equal the deformation gradient at a quantified rate; (ii) determine whether, and under what conditions, the two-dimensional deformation-gradient estimator is consistent, with convergence rates and bias-variance trade-offs, beyond separable deformations; (iii) characterize identifiability conditions and design adaptive warping-matrix selection that avoid aperture-type degeneracy; (iv) extend the texture model beyond the Lambertian, locally multiplicative intensity model to three-dimensional and non-Lambertian surface appearance; and (v) quantify finite-resolution and discretization effects into practical performance bounds.

Background

Shape from texture is the inverse problem of recovering the three-dimensional geometry of a surface from a single monocular image of its textured surface, exploiting the fact that perspective projection changes the apparent scale and shape of surface markings in a way that depends on surface orientation and distance. Classical treatments assumed deterministic or isotropic textures; Clerc and Mallat instead model the surface texture as a stochastic process, so that shape recovery becomes a statistical estimation problem from a single realization.

The geometric backbone is the following. Under perspective projection, a neighborhood of the surface is mapped to the image by a local map $d(x)$ whose Jacobian $J_d(x)$ describes the position-dependent metric distortion. What is needed for shape recovery is not $d$ itself but the deformation gradient, the matrix-valued field $g^k(u)=J_d(u)^{-1}\partial_{u_k}J_d(u)$ (for $k=1,2$), which measures relative metric change and is related by differential geometry to the local surface orientation (slant and tilt, i.e., the angle between the surface normal and the view direction and the direction of maximal slope) and to the normal curvatures of the surface.

The measurement tool introduced in the source paper is the warplet transform. A warplet is a two-dimensional zero-mean analysis atom $\psi_{u,S}(x)=\det(S)^{-1}\psi(S^{-1}(x-u))$ parameterized by a position $u$ and an invertible $2\times 2$ warping matrix $S$, which replaces the scalar scale of a one-dimensional wavelet and encodes local dilation, rotation, and shear, so that warped atoms migrate under local affine deformations of the image. The warpogram $w(u,S)=\mathbf{E}\{|\langle I,\psi_{u,S}\rangle|^2\}$ is the variance of warplet coefficients of the image $I$; its one-dimensional analogue is the scalogram, the distribution of wavelet-coefficient energies across position and scale. The central result is that the warpogram satisfies a transport (conservation) equation, the Texture Gradient Equation, in the position-warping parameter space: in one dimension $\partial_u w(u,s)-v(u,s)\,\partial_{\log s}w(u,s)=0$, and in two dimensions $\partial_{u_k}w(u,S)=\sum_{i,j}v_{ij}^k(u,S)\,a_{ij}(u,S)$, where the $a_{ij}$ are partial derivatives of $w$ with respect to the logarithmic warping parameters and the velocities $v$ measure how local spectral energy is transported across analysis parameters. For a stationary texture (a stochastic texture whose statistics are invariant under spatial translations) whose covariance satisfies $C(0)-C(\tau)=|\tau|^{\gamma}\eta(\tau)$ with $\gamma>0$ and $\eta$ differentiable near $0$, the one-dimensional velocity converges at fine scales to the deformation gradient $d''(u)/d'(u)$.

For general nondevelopable surfaces (surfaces that cannot be flattened without stretching), the surface texture $\tilde R$ is pulled back to the tangent plane at each point through the exponential map (the map that sends a tangent vector at a surface point to the surface point reached by following the geodesic, i.e., the locally shortest surface path, that it determines), giving local tangent-plane processes $R_{p(u)}$, where $p$ denotes the map from an image point to the corresponding surface point. A local homogeneity (weak stationarity) condition — equation (38) of the paper — requires that a normalized local covariance of warplet responses of $R_{p(u)}$ be of order $O((\det S)^{1/2})$ as the analysis scale shrinks. Under this condition the two-dimensional texture-gradient velocity equals the deformation gradient up to an error of the same order. The condition covers stationary textures on developable surfaces, restrictions of three-dimensional stationary isotropic processes to a sphere, and textures oriented along parallels or meridians, but the class of surface-texture pairs satisfying it was left uncharacterized.

On the estimation side, the paper assumes a Lambertian surface (a matte surface whose apparent brightness does not depend on the viewing direction) with a locally multiplicative intensity model $I(x)=a(x)\tilde R(p(x))$ (a shading/contrast factor times the texture value at the corresponding surface point), builds unbiased estimators of warpogram derivatives from warplet inner products of a single image, smooths them with a kernel of width $\Delta$ to control variance, and recovers each deformation-gradient matrix $g^k(u)$ (four unknowns per $k$) by solving a linear system (equation (31) of the paper) built from warpogram derivatives at a collection of warping matrices $\{S_i\}$. Slant and tilt then follow from a one-dimensional least-squares fit, and depth by integrating the resulting needle map (the field of recovered surface normals over the image). Statistical consistency of the estimator was proven only in one dimension (it extends trivially to separable two-dimensional deformations $d(x_1,x_2)=(d_1(x_1),d_2(x_2))$); the general two-dimensional case was explicitly left open. The linear system can be underdetermined for certain texture orientations even with arbitrarily many warping matrices — an analogue of the aperture problem known from optical flow, where one-dimensional image structure cannot determine a two-dimensional motion field. The authors close by identifying the remaining scientific issues that define the present problem.

Problem Statement

Complete the estimation-theoretic foundations of shape from texture within the warplet/Texture-Gradient-Equation framework of Clerc and Mallat (2002), so that recovering surface shape from a single image of a stochastic texture becomes a procedure with characterized validity conditions, statistical guarantees, and quantified finite-resolution behavior. The common objective uniting the requested results is a closed chain from assumptions to guarantees: which surface-texture pairs are admissible, whether the estimator is consistent and at what rate, when the recovery problem is well posed, which physical appearance models the theory covers, and how accurate it is at finite image resolution. Specifically, determine: (1) necessary and sufficient geometric and stochastic conditions on the surface texture $\tilde R$ and on the surface geometry (including curvature and the slant-tilt frame) under which the local homogeneity condition (equation (38) of the source) holds and the Texture Gradient velocity converges to the deformation gradient at a prescribed rate; (2) a statistical consistency theory in the full two-dimensional setting, determining whether and under what conditions the two-dimensional deformation-gradient estimator is consistent, with convergence rates and variance-bias trade-offs, under realistic stochastic texture models, beyond separable deformations $d(x_1,x_2)=(d_1(x_1),d_2(x_2))$; (3) identifiability and degeneracy conditions — conditions on texture orientation and on the selected set of warping matrices $\{S_i\}$ that prevent aperture-like underdetermination of the linear system (equation (31) of the source), together with adaptive strategies for selecting $S_i(u)$ that avoid local degeneracies; (4) texture models beyond the Lambertian, locally multiplicative intensity model $I(x)=a(x)\tilde R(p(x))$, including three-dimensional texture models and estimation procedures that accommodate non-Lambertian reflectance and more general surface appearance; and (5) quantified finite-resolution and discretization effects — analysis of the choice of scales, the smoothing width $\Delta$, and numerical stability, yielding practical guidelines and performance bounds for typical image sizes. The problem is taken at the source's full generality; no component is restricted to a particular surface class, texture model, or estimator variant.

References

1. Maureen Clerc and Stéphane Mallat, "The Texture Gradient Equation for Recovering Shape from Texture", IEEE Transactions on Pattern Analysis and Machine Intelligence 24(4), 2002, pp. 536-549. DOI: 10.1109/34.993560, https://doi.org/10.1109/34.993560.
2. Maureen Clerc and Stéphane Mallat, "Estimating deformations of stationary processes", The Annals of Statistics 31(6), 2003, pp. 1772-1821. DOI: 10.1214/aos/1074290327, https://doi.org/10.1214/aos/1074290327.
3. Ethan Anderes and Sourav Chatterjee, "Consistent estimates of deformed isotropic Gaussian random fields on the plane", The Annals of Statistics 37(5A), 2009, pp. 2324-2350. DOI: 10.1214/08-AOS647, https://doi.org/10.1214/08-AOS647 (arXiv:0710.0379, https://arxiv.org/abs/0710.0379).
4. Ethan B. Anderes and Michael L. Stein, "Estimating deformations of isotropic Gaussian random fields on the plane", The Annals of Statistics 36(2), 2008. DOI: 10.1214/009053607000000893, https://doi.org/10.1214/009053607000000893.
5. Dor Verbin and Todd Zickler, "Toward a Universal Model for Shape From Texture", IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 419-427. DOI: 10.1109/CVPR42600.2020.00050, https://doi.org/10.1109/CVPR42600.2020.00050.
6. Eva-Maria Didden, Thordis Thorarinsdottir, Alex Lenkoski, and Christoph Schnörr, "Shape from Texture using Locally Scaled Point Processes", Image Analysis and Stereology 34(3), 2015, pp. 161-170. DOI: 10.5566/IAS.1078, https://doi.org/10.5566/IAS.1078 (arXiv preprint arXiv:1311.7401, 2013, https://arxiv.org/abs/1311.7401).
7. Harold Omer and Bruno Torrésani, "Time-frequency and time-scale analysis of deformed stationary processes, with application to non-stationary sound modeling", Applied and Computational Harmonic Analysis 43(1), 2017, pp. 1-22. DOI: 10.1016/j.acha.2015.10.002, https://doi.org/10.1016/j.acha.2015.10.002.
8. Golnoosh Elhami, Adam James Scholefield, and Martin Vetterli, "Shape From Bandwidth: Central Projection Case", IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 1808-1812. DOI: 10.1109/ICASSP40776.2020.9053365, https://doi.org/10.1109/ICASSP40776.2020.9053365.
9. A. M. E. Loh and R. Hartley, "Shape from Non-homogeneous, Non-stationary, Anisotropic, Perspective Texture", Proceedings of the British Machine Vision Conference (BMVC), 2005, pp. 8.1-8.10. DOI: 10.5244/C.19.8, https://doi.org/10.5244/C.19.8.
10. Fabio Galasso and Joan Lasenby, "Fourier analysis and Gabor filtering for texture analysis and local reconstruction of general shapes", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009, pp. 2342-2349. DOI: 10.1109/CVPR.2009.5206591, https://doi.org/10.1109/CVPR.2009.5206591.