Horikawa and Kamitani (2017) showed that features of a deep neural network (DNN) trained for object recognition, decoded from fMRI activity during sleep-onset non-rapid eye movement (NREM) dreaming, correlate with the reported dreamed-object categories at mid- to high-level network layers, and that dreamed categories could be identified above chance from the decoded features. Their analyses, however, evaluated category-averaged features on category-averaged brain activity and found an atypical layerwise profile (poor identification at DNN6–7 with relatively high DNN5 and DNN8, versus a mid-layer peak for seen and imagined objects). The unresolved problem is to determine the representational structure of dream content: whether low-level, image-specific visual features (e.g., color, contrast, local edges) of experienced dream content are represented in the sleeping brain at all; whether the atypical layerwise decoding profile reflects dream-specific representation rather than verbal-reporting, memory-retrieval, or category-set confounds; and whether the hierarchical feature-decoding framework generalizes beyond sleep-onset NREM — to other spontaneously generated subjective experiences such as mind wandering or visual hallucination induced by psychedelic drugs. No study has decoded image-specific or low-level features of dream content; EEG-based work decodes only coarse categorical features of reported dreams. The problem remains open.

Background

Brain (neural) decoding uses statistical models trained on measured brain activity to predict properties of a stimulus or mental state. In functional magnetic resonance imaging (fMRI), blood oxygenation level-dependent (BOLD) signals index local neural activity, and pattern-based decoders can predict which of many trained stimuli a person is viewing, or even continuous properties of the stimulus, from activity in visual cortex. A particularly useful proxy for the hierarchy of human visual representations is a feed-forward deep neural network (DNN, here an eight-layer convolutional network of the AlexNet type) trained on object recognition: its layers, from low-level (DNN1, roughly edge and contrast detectors) through mid-level (DNN4–7, texture and object-part combinations) to the top categorical layer (DNN8), encode visual features of increasing abstraction, and their unit activation patterns can be predicted from fMRI patterns by linear regression. 'Generic decoding' means the decoder, trained on feature values of viewed images, can identify arbitrary object categories never used in training, by matching decoded feature vectors to category-average feature vectors computed from a large image database (zero-shot identification). Sleep is divided into non-REM (NREM) stages and rapid eye movement (REM) sleep; 'sleep onset' denotes the transition into NREM sleep, when dreamlike (hypnagogic) experiences are frequent and reportable upon awakening. Dream content itself is unobservable except through verbal reports given after awakening, so reports — typically mapped to word categories (e.g., WordNet synsets) — serve as the only ground-truth labels for dream experiences. Building on their earlier study that decoded dreamed object categories from pre-awakening sleep-onset fMRI (Horikawa, Tamaki, Miyawaki and Kamitani, Science 2013, with about 200 awakening reports per subject), Horikawa and Kamitani (2017, Front. Comput. Neurosci.) trained DNN-feature decoders on waking perception data and applied them to fMRI recorded just before sleep-onset awakenings in two trained subjects. Decoded features correlated with the reported dream categories at mid- to high-level DNN layers, and dream categories could be identified above chance without pre-specifying targets. However, accuracy was poor for low-level features and low-level visual areas, the identification profile across layers was discontinuous and atypical relative to the mid-layer peak seen for perception and imagery (relatively high at DNN5 and at the categorical DNN8 layer, low at DNN6–7, with hints in the time course that report-period memory and verbal processes contributed), and the evaluation used category-averaged features and category-averaged activity, which discards image-specific low-level information. The study was also confined to sleep-onset NREM periods in two subjects. These limitations define the open question. Since then, the same group and others have greatly advanced feature decoding and image reconstruction for waking perception and volitional imagery (deep image reconstruction from fMRI; DNN-layer decodability characterization), but sleep-side decoding progress has been mainly via scalp electroencephalography (EEG): high-density EEG studies have decoded whether a person was dreaming and coarse content classes of reported dreams (faces, locations, verbal thought) in NREM stage 2 and REM, and machine-learning classifiers of dream presence have followed; none of this reaches image-specific or low-level visual features, and no fMRI study has extended the hierarchical feature-decoding paradigm beyond sleep onset. A 2024 review of new strategies in dream science still describes fMRI-based dream decoding as anchored to the sleep-onset studies and limited in REM by scanner noise and discomfort.

Problem Statement

Determine the representational structure of visual dream content in the sleeping brain, by extending the stimulus-trained hierarchical DNN feature-decoding paradigm of Horikawa and Kamitani (2017) so that: (i) it is tested whether low-level, image-specific visual features of experienced dream content (e.g., color, contrast, local edges) are represented in brain activity preceding awakenings — requiring analyses sensitive to feature-level, not category-averaged, information and to activity of lower visual areas; (ii) it is examined whether the atypical layerwise identification profile observed for dreaming (low accuracy at DNN6–7, relatively high at DNN5 and at the categorical DNN8 layer, versus the mid-layer peak for seen and imagined objects) reflects differences in representational content during dreaming itself, or confounds introduced by verbal reporting, memory retrieval, or differences in tested category sets — the source discusses this profile as possibly reflecting higher cognitive functions such as memory and abstract knowledge, and its report-period time-course evidence as possibly reflecting memory retrieval and verbal reporting — for example by designs that dissociate pre-awakening activity from report-period activity and control the category sets; and (iii) it is established whether the same feature-decoding framework generalizes beyond sleep-onset NREM dreaming to other spontaneously generated subjective experiences, such as mind wandering or visual hallucination induced by psychedelic drugs, as proposed by the source. These components jointly support one objective — establishing which levels of the visual feature hierarchy the sleeping brain recruits, and how dream representation differs from perception and imagery. The three components are complementary and none is independently sufficient: the task is resolved only by addressing all three.

References

  1. Tomoyasu Horikawa and Yukiyasu Kamitani (2017). Hierarchical Neural Representation of Dreamed Objects Revealed by Brain Decoding with Deep Neural Network Features. Frontiers in Computational Neuroscience 11:4. DOI: 10.3389/fncom.2017.00004. https://doi.org/10.3389/fncom.2017.00004
  2. T. Horikawa, M. Tamaki, Y. Miyawaki and Y. Kamitani (2013). Neural Decoding of Visual Imagery During Sleep. Science 340:639–642. DOI: 10.1126/science.1234330. https://doi.org/10.1126/science.1234330
  3. Tomoyasu Horikawa and Yukiyasu Kamitani (2017). Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications 8:15037. DOI: 10.1038/ncomms15037. https://doi.org/10.1038/ncomms15037
  4. Guohua Shen, Tomoyasu Horikawa, Kei Majima and Yukiyasu Kamitani (2019). Deep image reconstruction from human brain activity. PLOS Computational Biology 15(1):e1006633. DOI: 10.1371/journal.pcbi.1006633. https://doi.org/10.1371/journal.pcbi.1006633
  5. Tomoyasu Horikawa, Shuntaro C. Aoki, Mitsuaki Tsukamoto and Yukiyasu Kamitani (2019). Characterization of deep neural network features by decodability from human brain activity. Scientific Data 6(1):190012. DOI: 10.1038/sdata.2019.12. https://doi.org/10.1038/sdata.2019.12
  6. Francesca Siclari, Benjamin Baird, Lampros Perogamvros, Giulio Bernardi, Joshua J. LaRocque, Brady Riedner, Melanie Boly, Bradley R. Postle and Giulio Tononi (2017). The neural correlates of dreaming. Nature Neuroscience 20:872–878. DOI: 10.1038/nn.4545. https://doi.org/10.1038/nn.4545
  7. Remington Mallett, Karen R. Konkoly, Tore Nielsen, Michelle Carr and Ken A. Paller (2024). New strategies for the cognitive science of dreaming. Trends in Cognitive Sciences 28(12). DOI: 10.1016/j.tics.2024.10.004. https://doi.org/10.1016/j.tics.2024.10.004
  8. Karen R. Konkoly, Kristoffer Appel, Emma Chabani, Anastasia Mangiaruga, Jarrod Gott, Remington Mallett, Bruce Caughran, Sarah Witkowski, Nathan W. Whitmore, Christopher Y. Mazurek, Jonathan B. Berent, Frederik D. Weber, Başak Türker, Smaranda Leu-Semenescu, Jean-Baptiste Maranci, Gordon Pipa, Isabelle Arnulf, Delphine Oudiette, Martin Dresler and Ken A. Paller (2021). Real-time dialogue between experimenters and dreamers during REM sleep. Current Biology 31:1417–1427.e6. DOI: 10.1016/j.cub.2021.01.026. https://doi.org/10.1016/j.cub.2021.01.026
  9. W. Wong, V. Noreika, L. Móró, A. Revonsuo, J. Windt, K. Valli and N. Tsuchiya (2020). The Dream Catcher experiment: blinded analyses failed to detect markers of dreaming consciousness in EEG spectral power. Neuroscience of Consciousness 2020(1):niaa006. DOI: 10.1093/nc/niaa006. https://doi.org/10.1093/nc/niaa006
  10. L. Moctezuma, M. Molinas and T. Abe (2025). From high- to low-density EEG for automatic classification of dream experiences during stage 2 of NREM. SLEEP Advances 6(4):zpaf066. DOI: 10.1093/sleepadvances/zpaf066. https://doi.org/10.1093/sleepadvances/zpaf066
  11. Pinyao Liu, Keon Ju Lee, Alexander Steinmaurer, Claudia Picard-Deland, Michelle Carr and Alexandra Kitson (2025). DreamLLM-3D: Affective Dream Reliving using Large Language Model and 3D Generative AI. arXiv:2503.16439. https://arxiv.org/abs/2503.16439