AIMA 2026 projects ✦ Project I ✦ Neuro-AI

Brain decoding: from fMRI & EEG to images and video

Human-Centered

In three years, brain decoding has gone from blurry shapes to photo-like images and smooth video. It has also learned that a strong diffusion prior can paint a convincing picture that the brain never saw. Our group will build on the best recent methods and ask how much of each reconstruction truly came from the brain.

In partnership with

  • Duke University
  • Northwestern University

Seeing, recording, decoding

Photograph shown to the viewer

1 What they see

a real photograph

2 The viewer

3 Recorded signal

4 Decoder

brain encoder → shared latent

5 Reconstruction

Try another photo

Photos: CC0, Wikimedia Commons. Steps 3–5 are illustrations of how decoders work, not output from a real recording; the reconstructions are softened versions of the photo made for this page.

Three years of the field, on one map

Thirty-one papers from 2023 to 2026, most at CVPR, ICCV, ECCV, NeurIPS, ICML, ICLR, AAAI and Nature-family journals. Filter by signal and output; hover or tap a paper for its one-line summary and link.

Signal Output

Eight open research gaps

Each gap is backed by the papers above and sized for one student group and one to four GPUs. The group picks one or two with its mentors in month one.

01

Cross-subject decoding with little or no data

+

Shared-subject models cut new-subject data from 40 hours to about one3–6, and zero-shot decoding is emerging7. But zero-shot still trails fine-tuning, and almost all results come from the same 7–8 NSD subjects24.

DirectionA leave-one-subject-out benchmark of alignment methods (linear, transfer matrix, LoRA, adversarial) at 0, 15 and 60 minutes of new-subject data, plus a new hyperalignment-adapter baseline.

02

Faithfulness: is it the brain, or the prior?

+

A 30–50-dimensional bottleneck recovers most reconstruction quality9, and text-guided NSD models behave like category classifiers plus diffusion “hallucination”8. Common metrics reward plausible pictures, not faithful ones.

DirectionA faithfulness audit suite (bottleneck, shuffled-signal and noise controls, out-of-distribution test sets, category-leakage checks) used to re-score 4–5 open-source decoders. Mostly inference, so it needs little compute.

03

The EEG fidelity gap

+

EEG is cheap and portable, but reconstructions are semantically right and low-level wrong11,12. MEG decodes high-level features well over time13. Older EEG datasets carry a block-design confound18.

DirectionEEG-to-image on THINGS-EEG225 with a pretrained EEG foundation encoder23 and a separate low-level branch, reporting low- and high-level metrics separately. Trains on a single RTX 4090.

04

Low-level detail versus meaning

+

Most strong decoders split the problem into a low-level layout path and a semantic path2,16, but the trade-off between them is tuned by hand.

DirectionA controlled study of the low-level/semantic Pareto front per brain region (early visual cortex versus higher areas) on NSD, with region-specific adapters.

05

Temporal consistency in video decoding

+

fMRI is slow and the standard video set has only three subjects27. Motion can be invented by the video generator; permutation tests are needed to show it was decoded10,15–17.

DirectionAn explicit motion-decoding head with motion-shuffled baselines and temporal metrics on CC2017. Needs an A100 for video diffusion.

06

Real-time, portable decoding

+

MEG points toward real-time decoding13, and lightweight models now run on consumer-grade EEG14,28.

DirectionA latency- and parameter-efficient EEG decoder, measuring accuracy against number of channels and time window.

07

Generalizing beyond one dataset

+

NSD’s limited semantic diversity inflates results8. Few papers test on a second dataset.

DirectionTrain on NSD, test on THINGS-fMRI26 and other held-out sets, and report the drop on novel categories.

08

Brain to language, responsibly

+

Brain signals can be decoded into captions and continuous language19–22, which raises privacy questions and lacks hallucination-aware evaluation.

DirectionAn fMRI-to-caption evaluation on NSD with object-hallucination metrics, comparing open models, with an ethics section.

Aim 1 · Months 1–2

Reproduce and audit

Reproduce two open decoders (for example MindEye2 and ATM) and run the faithfulness controls from gap 02 on them.

Aim 2 · Months 3–4

Close one gap

Propose a method for the chosen gap: few-data cross-subject decoding, or low-level fidelity for EEG.

Aim 3 · Months 5–6

Evaluate honestly, then publish

Test on a second dataset, release code, and submit to a NeurIPS/ICLR workshop or a MICCAI-track venue.

Data

  • NSD, 7T fMRI, 8 subjects24
  • THINGS-EEG2 and THINGS-MEG / fMRI25,26
  • CC2017 video fMRI27
  • Alljoined consumer-grade EEG28

You will learn

  • fMRI and EEG preprocessing
  • Contrastive alignment and diffusion priors
  • Cross-subject adaptation (LoRA, alignment)
  • Evaluating generative models honestly

Good fit if you

  • Like neuroscience as much as generative models
  • Are comfortable with PyTorch and GPUs
  • Enjoy asking “is this result real?”
1
Data & signals

Preprocessing pipelines for NSD, THINGS and EEG; subject splits.

2
Baselines

Reproduces open decoders and keeps results reproducible.

3
Method

Designs the alignment, encoder or adapter for the chosen gap.

4
Evaluation

Owns metrics, faithfulness controls and figures.

✦

The plan · 6 months

From first paper to submission.

Six months, from first paper to submission, with weekly group meetings and a monthly review with mentors.

Month 1
Month 2
Month 3
Month 4
Month 5
Month 6
I · Brain decoding
Reading & dataReproduce + auditMethod for the chosen gapCross-dataset testsWrite & submit
Onboarding Baselines Method Experiments & studies Writing
M1
Onboard

Read 10–15 key papers, set up data access and compute, write a one-page proposal.

M2
Reproduce

Match a published baseline. Share a reproducibility report.

M3
Propose

First version of the method, defended in a mentor review.

M4
Iterate

Ablations, failure analysis, fix what the review found.

M5
Validate

External data or pilot study; freeze results.

M6
Submit

Paper, code release and a talk for the AIMA community.

Project I · Neuro-AI

Want to work on
this question?

Apply through the form by November 7, 2026, choosing Project I, with an academic CV and a statement of purpose of up to 300 words. Interviews run November 9 to 13, and groups start around November 18. Mentors from Duke University · Northwestern University.

Apply by November 7 All nine projects

References

  1. Takagi & Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. CVPR 2023. link
  2. Scotti et al. Reconstructing the Mind’s Eye: fMRI-to-image with contrastive learning and diffusion priors. NeurIPS 2023. arXiv
  3. Scotti, Tripathy et al. MindEye2: Shared-subject models enable fMRI-to-image with 1 hour of data. ICML 2024. PMLR
  4. Wang et al. MindBridge: A cross-subject brain decoding framework. CVPR 2024. arXiv
  5. Gong et al. MindTuner: Cross-subject visual decoding with visual fingerprint and semantic correction. AAAI 2025. link
  6. Dai et al. MindAligner: Explicit brain functional alignment for cross-subject visual decoding from limited fMRI data. ICML 2025. PMLR
  7. Wang et al. ZEBRA: Towards zero-shot cross-subject generalization for universal brain visual decoding. NeurIPS 2025. OpenReview
  8. Shirakawa et al. Spurious reconstruction from brain activity. Neural Networks 2025. DOI
  9. Mayo et al. BrainBits: How much of the brain are generative reconstruction methods using? NeurIPS 2024. arXiv
  10. Lu et al. Animate your thoughts: Decoupled reconstruction of dynamic natural vision from slow brain activity (Mind-Animator). ICLR 2025. arXiv
  11. Song et al. Decoding natural images from EEG for object recognition (NICE). ICLR 2024. arXiv
  12. Li et al. Visual decoding and reconstruction via EEG embeddings with guided diffusion (ATM). NeurIPS 2024. arXiv
  13. Benchetrit, Banville & King. Brain decoding: toward real-time reconstruction of visual perception. ICLR 2024. arXiv
  14. Kneeland et al. ENIGMA: A unified lightweight EEG-to-image model for multi-subject visual decoding. NeurIPS 2025 Workshop. link
  15. Chen, Qing & Zhou. Cinematic Mindscapes: High-quality video reconstruction from brain activity (MinD-Video). NeurIPS 2023. arXiv
  16. Gong et al. NeuroClips: Towards high-fidelity and smooth fMRI-to-video reconstruction. NeurIPS 2024. arXiv
  17. Wang et al. NEURONS: Emulating the human visual cortex improves fidelity and interpretability in fMRI-to-video reconstruction. ICCV 2025. CVF
  18. Li et al. The perils and pitfalls of block design for EEG classification experiments. IEEE TPAMI 2021. DOI
  19. Xia et al. UMBRAE: Unified multimodal brain decoding. ECCV 2024. arXiv
  20. Qiu et al. MindLLM: A subject-agnostic and versatile model for fMRI-to-text decoding. ICML 2025. PMLR
  21. Tang et al. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience 2023. DOI
  22. Horikawa. Mind captioning: Evolving descriptive text of mental content from human brain activity. Science Advances 2025. DOI
  23. Jiang et al. Large brain model for learning generic representations with tremendous EEG data in BCI (LaBraM). ICLR 2024. arXiv
  24. Allen et al. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence (NSD). Nature Neuroscience 2022. DOI
  25. Gifford et al. A large and rich EEG dataset for modeling human visual object recognition (THINGS-EEG2). NeuroImage 2022. DOI
  26. Hebart et al. THINGS-data: fMRI, MEG and behavioral data on object representations. eLife 2023. DOI
  27. Wen et al. Neural encoding and decoding with deep learning for dynamic natural vision (CC2017). Cerebral Cortex 2018. DOI
  28. Alljoined: EEG datasets for natural-image decoding, including consumer-grade Alljoined-1.6M. arXiv · arXiv