Visual mental imagery - seeing with the mind's eye - produces no external signal, so whether its subjective content can be read out from human brain activity is a foundational open question for neural decoding. Koide-Majima, Nishimoto and Majima reconstructed both seen and cued imagined images from fMRI by decoding deep-network and CLIP semantic features and sampling a VQGAN latent vector with stochastic gradient Langevin dynamics, reporting pairwise identification of imagined images at 75.6% against a 50% chance level, and proposed the framework as a technology for directly investigating the subjective contents of the brain. Later work has both contested and advanced the enterprise: an independent reanalysis reports selective reporting, circular metrics and a functionally inert sampling component; benchmark studies show that mental-image reconstruction performance is largely decoupled from seen-image performance; and critiques show that reconstructions can be a blend of category classification and generative hallucination. The open problem is to establish mental-image reconstruction as a reliable, validated readout of subjective visual mental content: reliable for arbitrary mental images across subjects and acquisition protocols, and validated against behavioral reports of what is subjectively experienced. Extending reconstruction beyond cued replay of seen images toward self-generated imagery, and clarifying whether the reported sharpening effect for geometric shapes reflects genuine top-down processing or a pipeline artifact, are related open directions rather than required components of an answer.

Background

Visual mental imagery - the internal generation of a visual experience without corresponding retinal input - is a core capacity of human cognition, engaged in memory, planning and creativity, and it is altered in conditions such as aphantasia (the absence of voluntary visual imagery) and hallucinations. Its contents are private: unlike a photograph on a screen, an imagined image produces no external signal against which a readout can be checked. Functional magnetic resonance imaging (fMRI) measures brain activity indirectly through blood-oxygenation-level-dependent (BOLD) signals, and over the past decade 'deep image reconstruction' methods have attempted to invert these signals into images. In the foundational framework of Shen et al. (2019), linear regression models trained on paired fMRI-and-image data first translate voxel activity patterns into the layer-wise feature activations of a pretrained vision network (VGG19, a 19-layer convolutional network trained on object recognition); an image is then synthesized so that its network features match the decoded ones, typically by optimizing the latent code of a pretrained deep generator network. Koide-Majima, Nishimoto and Majima argued that this pipeline underuses the semantic (meaning-level) information that dominates mental imagery, and extended it in two ways: they additionally decoded image embeddings from CLIP (Contrastive Language-Image Pretraining, a model trained to place images and text in a shared semantic vector space), and they reformulated image synthesis as Bayesian estimation, sampling the latent vector of a VQGAN generator (a vector-quantized generative adversarial network mapping a discrete latent code to an image) with stochastic gradient Langevin dynamics (SGLD), a sampling algorithm that injects Gaussian noise into gradient updates so that the iterates converge to samples from a posterior distribution rather than to a single point estimate. Using fMRI from three subjects who either viewed or were cued to imagine images, they reported that their framework identified the correct imagined image in pairwise forced-choice comparison at 75.6% against a 50% chance level (seen images: 90.7%), while the earlier method performed near chance on imagery (50.4%). They concluded that the framework could provide a technology for directly investigating the subjective contents of the brain, such as illusions, hallucinations and dreams. The demonstration rests on a three-subject dataset, the authors note that reconstruction quality varied significantly across imagery samples, the reconstructions were never tested against the participants' subjective experience, and reconstruction of non-visual or spontaneous mental content was left as a motivating prospect rather than demonstrated; while the published abstract presents the visualization of arbitrary imagined natural images as an achieved milestone, subsequent work has both contested these claims and advanced neighboring parts of the enterprise. What remains open is whether mental-imagery reconstruction can be made reliable for arbitrary images and validated as a faithful readout of what a person actually experiences.

Problem Statement

In the context of Koide-Majima, Nishimoto and Majima (Neural Networks 2024): establish whether reconstruction of mental images from human brain activity can serve as a reliable and validated readout of subjective visual mental content. This objective has two constitutive dimensions. (1) Reliability and generalization - reconstruction of imagined images whose fidelity holds for arbitrary mental images, beyond small trained image domains and beyond best-case samples, demonstrated across multiple subjects and fMRI acquisition settings with quantitative metrics that are not circular, meaning that evaluation does not reuse the same feature representations that the reconstruction algorithm optimized (the failure mode documented in the 2025 reanalysis). (2) Subjective-content validation - an experimental test of whether the reconstructed images actually reflect the imagers' subjective mental content, by combining brain-to-image reconstruction with behavioral reports or other objective behavioral measures of what was imagined, with controls that rule out category classification and generator hallucination as alternative explanations. Two related open directions are scientifically significant in their own right but are not required components of an answer to the target question: extending reconstruction beyond cued replay of seen images toward internally generated and spontaneous visual content, and determining whether the emphasized line components found in imagery reconstructions of geometric shapes reflect genuine top-down processing in visual cortex or an artifact of the decoding and generation pipeline, and whether the effect generalizes to natural images. Engineering avenues - replacing the computationally expensive 500-iteration SGLD sampler with more efficient sampling or optimization, customizing the pretrained models (VGG19, VQGAN, CLIP) or training end-to-end decoders, and scaling to larger cohorts - remain scope notes rather than required deliverables.

References

  1. Koide-Majima, N., Nishimoto, S., & Majima, K. (2024). Mental image reconstruction from human brain activity: Neural decoding of mental imagery via deep neural network-based Bayesian estimation. Neural Networks. DOI: 10.1016/j.neunet.2023.11.024. https://doi.org/10.1016/j.neunet.2023.11.024
  2. Koide-Majima, N., Nishimoto, S., & Majima, K. (2023). Mental image reconstruction from human brain activity. bioRxiv preprint. DOI: 10.1101/2023.01.22.525062. https://doi.org/10.1101/2023.01.22.525062
  3. Shen, G., Horikawa, T., Majima, K., & Kamitani, Y. (2019). Deep image reconstruction from human brain activity. PLOS Computational Biology. DOI: 10.1371/journal.pcbi.1006633. https://doi.org/10.1371/journal.pcbi.1006633
  4. Shirakawa, K., Nagano, Y., Tanaka, M., Cheng, F. L., & Kamitani, Y. (2025). Advancing credibility and transparency in brain-to-image reconstruction research: Reanalysis of Koide-Majima, Nishimoto, and Majima (Neural Networks, 2024). arXiv:2511.07960. https://arxiv.org/abs/2511.07960
  5. Shirakawa, K., Nagano, Y., Tanaka, M., Aoki, S. C., Majima, K., Muraki, Y., & Kamitani, Y. (2024). Spurious reconstruction from brain activity: The thin line between reconstruction, classification, and hallucination. Journal of Vision, 24(10), 321. DOI: 10.1167/jov.24.10.321. arXiv:2405.10078. https://arxiv.org/abs/2405.10078
  6. Kneeland, R., Scotti, P. S., St-Yves, G., Breedlove, J., Kay, K., & Naselaris, T. (2025). NSD-Imagery: A benchmark dataset for extending fMRI vision decoding methods to mental imagery. CVPR 2025. DOI: 10.1109/CVPR52734.2025.02687. arXiv:2506.06898. https://arxiv.org/abs/2506.06898
  7. Kneeland, R., Torrico Villanueva, C. K., Chen, T., Ojeda, J., Khanna, S., Xu, J., Scotti, P. S., & Naselaris, T. (2026). MIRAGE: Robust multi-modal architectures translate fMRI-to-image models from vision to mental imagery. PLOS Computational Biology. DOI: 10.1371/journal.pcbi.1014263. https://doi.org/10.1371/journal.pcbi.1014263
  8. Caselles-Dupré, H., Mellerio, C., Hérent, P., Lopez-Persem, A., Béranger, B., Soularue, M., Fautrel, P., Vernier, G., & Cord, M. (2024). Mind-to-Image: Projecting Visual Mental Imagination of the Brain from fMRI. arXiv:2404.05468. https://arxiv.org/abs/2404.05468
  9. Horikawa, T., Tamaki, M., Miyawaki, Y., & Kamitani, Y. (2013). Neural Decoding of Visual Imagery During Sleep. Science. DOI: 10.1126/science.1234330. https://doi.org/10.1126/science.1234330
  10. Horikawa, T., & Kamitani, Y. (2022). Attention modulates neural representation to render reconstructions according to subjective appearance. Communications Biology. DOI: 10.1038/s42003-021-02975-5. https://doi.org/10.1038/s42003-021-02975-5
  11. Kronemer, S. I., Bandettini, P. A., & Gonzalez-Castillo, J. (2025). Sleuthing subjectivity: a review of covert measures of consciousness. Nature Reviews Neuroscience, 26(8), 476-496. DOI: 10.1038/s41583-025-00934-1. https://doi.org/10.1038/s41583-025-00934-1
  12. Majima, K. (2025). An Introduction to the Langevin Monte Carlo Method and Its Applications. The Brain & Neural Networks. DOI: 10.3902/jnns.32.99. https://doi.org/10.3902/jnns.32.99
  13. Olza, A., Soto, D., & Santana, R. (2024). Domain Adaptation-Enhanced Searchlight: Enabling classification of brain states from visual perception to mental imagery. arXiv:2408.01163. https://arxiv.org/abs/2408.01163