Visual brain decoders and image-reconstruction models are trained on fMRI activity evoked by controlled viewing of stimuli, yet the visual experiences most worth reading out — mental imagery, illusory percepts, and dreams — are generated internally rather than by the retina. Whether and how far decoders trained solely on stimulus-evoked activity can faithfully read out such endogenous experience, and what differences in neural representation between stimulus-evoked and endogenous activity account for the observed transfer failures, was explicitly deferred by the foundational studies of the field and remains open. The literature lineage shows repeated partial success and repeated deferral: waking-trained decoders classified reported dream categories during sleep-onset NREM sleep (2013); dreamed-object features decoded only at mid- to high-level DNN layers with report-derived ground truth and low-level features unresolved (2017); imagined-object decoding was demonstrated for volitional imagery while spontaneous experience was named as requiring further work (2017); pixel-level imagery reconstruction remained 'rudimentary' (2019); illusion reconstruction assumed an untested common coding scheme between veridical and illusory perception (2023). A 2025 benchmark (NSD-Imagery) quantifies that seen-to-imagery transfer of state-of-the-art decoders is largely decoupled from their perception performance, and 2024–2026 work achieves only above-chance semantic reconstruction when adapting perception decoders to imagery. The problem asks how far stimulus-evoked-trained decoders and reconstruction models transfer to endogenous visual experience — volitional imagery, illusory percepts, and dream experience — at which levels of the visual representation, including beyond a pre-specified training category set, and what differences in neural representation between stimulus-evoked and endogenous activity explain the observed failures.

Background

Functional magnetic resonance imaging (fMRI) measures blood-oxygenation signals that indirectly report the activity of local neural populations, and machine-learning 'decoders' trained on these multivoxel activity patterns can identify which image a person is viewing. The dominant paradigm trains decoders on activity evoked by controlled stimulus presentation and validates them on held-out stimulus data. 'Generic decoding' extends this beyond a fixed category set: instead of classifying into the training categories, the decoder predicts the feature representation that a deep neural network (DNN, an image-recognition network whose layered features roughly mimic the stages of the visual hierarchy) computes for the seen image, so that arbitrary images can be identified by matching the predicted features against a large image database. Image 'reconstruction' methods go further and synthesize a pixel image (typically via a generative model) whose DNN features match the decoded ones. The open question is whether this stimulus-trained machinery can read out internally generated visual experience. Three families of endogenous visual experience are distinguished in this literature. First, mental imagery: voluntarily visualized content with no corresponding stimulus on the retina. Second, illusory percepts: experiences produced by a physical stimulus whose content diverges from that stimulus, such as illusory contours that appear where only offset grating patches are displayed, or neon color spreading, a perceived tinted surface induced by colored line endings. Third, spontaneous dream experience during sleep, studied by letting participants fall asleep in the scanner, awakening them at electrophysiologically defined moments, and collecting verbal reports of the preceding experience. The founding result for the endogenous case was Horikawa, Tamaki, Miyawaki, and Kamitani's 2013 Science study: decoders trained on waking, stimulus-evoked activity classified the object and scene categories reported from dreams in pre-awakening fMRI volumes, but only at category level (about twenty categories per participant), only during non-rapid-eye-movement (NREM) sleep-onset periods — rapid-eye-movement (REM) dreaming was never tested — and only against ground truth taken from post-awakening verbal reports. Miyawaki et al.'s 2008 reconstruction paper had earlier demonstrated constraint-free reconstruction of viewed images from fMRI; no extension to internally generated experience was demonstrated there. Follow-up work deepened rather than closed the question. Horikawa and Kamitani's 2017 study of dreamed objects (Frontiers in Computational Neuroscience) showed that DNN features of dreamed content were decodable from mid- to high-level layers but could not establish low-level feature representation (color, contrast) in early visual areas, and its validation compared decoded features to category-average features derived from the verbal reports. Their companion 2017 Nature Communications paper decoded volitionally imagined object categories beyond the training set, while stating explicitly that decoding spontaneous experience 'requires distinguishing the conceivable differences in neural representation between volitional and spontaneous mental imagery.' Shen et al. (2019) achieved pixel-level DNN-feature reconstruction but reported only rudimentary reconstructions of simple imagined figures; Senden et al. (2019) reconstructed imagined letters at 7 Tesla (a high magnetic field strength giving finer spatial resolution) by inverting a stimulus-trained encoding model — a model that predicts brain activity from stimulus features, inverted here by searching for the stimulus whose predicted activity best matches the measured activity — combined with a learned imagery-to-perception projection, covering four letter shapes. Cheng et al. (2023, Science Advances) trained a reconstruction model on natural-image viewing and recovered illusory-consistent lines and colors for two illusion families, under an explicitly assumed (and only partially supported) common coding scheme between veridical and illusory perception. Meanwhile the dramatic recent progress in perceived-image reconstruction driven by the Natural Scenes Dataset (a large shared fMRI dataset of natural-image viewing) and diffusion generative models — reviews such as Lu (2023) still describe reconstruction of subjective visual experience as 'a challenge to be actively explored' — has been evaluated almost exclusively on seen images. A 2025 benchmark, NSD-Imagery, began quantifying the transfer gap from seen to imagined images, finding that state-of-the-art perception decoders transfer poorly and that transfer performance is largely decoupled from perception performance. What remains unresolved is the original question at its full scope: how far can decoders trained solely on stimulus-evoked activity faithfully read out endogenous visual experience, at which representational levels and beyond which category sets — and where transfer fails, what differences in neural representation between stimulus-evoked and endogenous activity account for the failure?

Problem Statement

Can decoders — category classifiers, DNN-feature decoders, or full image-reconstruction models — trained solely on brain activity evoked by veridical viewing of visual stimuli faithfully read out internally generated visual experience? The endogenous states at issue span three families: volitional mental imagery; illusory percepts whose experienced content diverges from the physical stimulus; and spontaneous dream experience during sleep, for which decoding to date covers reported categories and mid- to high-level features during sleep-onset NREM periods, with REM and later sleep stages not yet examined. The question asks how far stimulus-evoked-trained decoding transfers to these states; at which levels of the visual representation it succeeds — low-level features such as orientation, color, contrast, and shape, versus mid- to high-level object and semantic features; whether it extends to arbitrary content rather than classifying into a closed training category set; and, where transfer fails, what differences in neural representation between stimulus-evoked and endogenous activity account for the failure. These facets are jointly at issue rather than alternatives: the question spans all three endogenous families and all three decoder classes, so a result confined to a single family, a single representational level, or a single decoder class is a partial answer, and the question is fully resolved only when the standing of every facet is fixed across that span. A faithful readout is one validated against the experienced content, using the best available ground truth for the state in question — illusory percepts predictable from stimulus configuration, experimentally cued imagery targets, and post-awakening verbal reports for dreams. Characterizing the representational difference between stimulus-evoked and endogenous activity — in the framing of Horikawa & Kamitani (2017), 'distinguishing the conceivable differences in neural representation between volitional and spontaneous mental imagery' — is part of this open question rather than an alternative terminal answer: where and why transfer fails is as much at issue as whether it succeeds.

References

  1. Miyawaki, Y., Uchida, H., Yamashita, O., Sato, M., Morito, Y., Tanabe, H. C., Sadato, N., & Kamitani, Y. (2008). Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders. Neuron 60(5), 915–929. DOI: 10.1016/j.neuron.2008.11.004. https://doi.org/10.1016/j.neuron.2008.11.004
  2. Horikawa, T., Tamaki, M., Miyawaki, Y., & Kamitani, Y. (2013). Neural Decoding of Visual Imagery During Sleep. Science 340(6132), 639–642. DOI: 10.1126/science.1234330. https://doi.org/10.1126/science.1234330
  3. Horikawa, T., & Kamitani, Y. (2017). Hierarchical Neural Representation of Dreamed Objects Revealed by Brain Decoding with Deep Neural Network Features. Frontiers in Computational Neuroscience 11:4. DOI: 10.3389/fncom.2017.00004. https://doi.org/10.3389/fncom.2017.00004
  4. Horikawa, T., & Kamitani, Y. (2017). Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications 8:15037. DOI: 10.1038/ncomms15037. https://doi.org/10.1038/ncomms15037
  5. Shen, G., Horikawa, T., Majima, K., & Kamitani, Y. (2019). Deep image reconstruction from human brain activity. PLOS Computational Biology 15(1): e1006633. DOI: 10.1371/journal.pcbi.1006633. https://doi.org/10.1371/journal.pcbi.1006633
  6. Senden, M., Emmerling, T. C., van Hoof, R., Frost, M. A., & Goebel, R. (2019). Reconstructing imagined letters from early visual cortex reveals tight topographic correspondence between visual mental imagery and perception. Brain Structure and Function 224(3), 1167–1183. DOI: 10.1007/s00429-019-01828-6. https://doi.org/10.1007/s00429-019-01828-6
  7. Horikawa, T., & Kamitani, Y. (2022). Attention modulates neural representation to render reconstructions according to subjective appearance. Communications Biology 5, 34. DOI: 10.1038/s42003-021-02975-5. https://doi.org/10.1038/s42003-021-02975-5
  8. Lu, Z. (2023). Visualizing the mind's eye: a future perspective on applications of image reconstruction from brain signals to psychiatry. Psychoradiology 3. DOI: 10.1093/psyrad/kkad022. https://doi.org/10.1093/psyrad/kkad022
  9. Cheng, F. L., Horikawa, T., Majima, K., Tanaka, M., Abdelhack, M., Aoki, S. C., Hirano, J., & Kamitani, Y. (2023). Reconstructing visual illusory experiences from human brain activity. Science Advances 9(46): eadj3906. DOI: 10.1126/sciadv.adj3906. https://doi.org/10.1126/sciadv.adj3906
  10. Caselles-Dupré, H., Mellerio, C., Hérent, P., Lopez-Persem, A., Béranger, B., Soularue, M., Fautrel, P., Vernier, G., & Cord, M. (2024). Mind-to-Image: Projecting Visual Mental Imagination of the Brain from fMRI. arXiv:2404.05468. https://arxiv.org/abs/2404.05468
  11. Shirakawa, K., Nagano, Y., Tanaka, M., Aoki, S. C., Majima, K., Muraki, Y., & Kamitani, Y. (2024; revised 2025). Spurious reconstruction from brain activity. arXiv:2405.10078. https://arxiv.org/abs/2405.10078
  12. Kneeland, R., Scotti, P. S., St-Yves, G., Breedlove, J., Kay, K., & Naselaris, T. (2025). NSD-Imagery: A benchmark dataset for extending fMRI vision decoding methods to mental imagery. arXiv:2506.06898. https://arxiv.org/abs/2506.06898
  13. Spera, F., Boccato, T., Olak, M., Cammarota, S., Ciferri, M., Tronti, M., Toschi, N., & Ferrante, M. (2026). Seeing the imagined: a latent functional alignment in visual imagery decoding from fMRI data. arXiv:2604.15374. https://arxiv.org/abs/2604.15374