Real-time MindEye

Real-time reconstruction of human visual perception from fMRI.

This is an active, open-source project being developed in collaboration with the Computational Memory Lab at Princeton University and Sophont. We invite anyone interested to explore the code described in this document, reproduce our analyses, and to connect with us on the MedARC Discord server (https://discord.gg/tVR4TWnRM9), an open-science research forum operated by Sophont.

We are actively seeking collaborators to help extend this framework to new and exciting brain-computer interface applications. Support for this project was provided by NIMH award RF1MH125318 and by Princeton University.

Contact: Rishab Iyer (rsiyer@princeton.edu)

Overview

We present a first-of-its-kind pipeline that can reconstruct seen images from fMRI brain activity in real-time, using MindEye2. This serves as a demonstration that RT-Cloud can support state-of-the-art AI workflows for real-time fMRI at any standard MRI facility. By enabling fine-grained decoding of cognitive representations, we substantially increase the utility of real-time fMRI as a brain-computer interface for clinical and scientific applications.

For example, here is a good reconstruction/retrieval from our first-ever real-time session: The "ground truth" image was seen by the subject in the scanner. Viewing the image elicited a response in brain activity that was measured using fMRI. The "reconstruction" and "retrieval" images were then generated by our model using the subject's measured fMRI brain activity as input to the model. Reconstruction generates an image based solely on the brain activity. Retrieval is akin to a multiple choice question for the model: "based on the brain activity, choose the seen image out of a pool of n images". In this case, the model's choice was correct; its top guess (left-most retrieval image) was the ground truth image viewed by the subject, selected out of a pool of 62 images. All of this was accomplished in real-time, i.e., within seconds of the subject initially viewing the image, while they were still in the scanner.

Prior work relied on extensive processing of the Natural Scenes Dataset (NSD), which collected 30-40 hours of data from each of a few participants. Additionally, NSD used 7 Tesla (7T) MRI which is available at only about 100 sites around the world. Our pipeline involves pre-training MindEye2 on data from NSD and then fine-tuning on just 2-3 hours of data, acquired at 3 Tesla (3T), from a new participant. After this, the model can support reconstruction and retrieval of images viewed by that participant in real-time.

Note that we refer primarily to "reconstruction", but this should be interpreted to mean both reconstruction and retrieval unless specified otherwise.

General pipeline

For a more detailed walkthrough of the pipeline, see Detailed Pipeline in docs/00-pipeline.md.

1. Prior to the real-time session:
   1. Pre-train MindEye2 using data from multiple subjects collected using 7T fMRI from NSD
   2. Collect ~3 hours of data from a new participant using 3T fMRI. Use these data to:
      1. Select voxels in visual cortex that respond reliably to visual stimuli
      2. Fine-tune the model using single-trial response estimates from reliable voxels
2. In real-time:
   1. Stream DICOM images using RT-Cloud as the participant views NSD-like images in 3T fMRI
   2. Accounting for hemodynamic lag, select a target fMRI volume to use for each reconstruction/retrieval. Using the target volume:
   3. Perform motion correction and registration to a reference functional volume from the previous session(s)
   4. Fit a GLM to extract a single-trial response estimate (beta) for each voxel in the pre-defined "reliable voxel" mask
   5. Input this beta pattern into the fine-tuned MindEye2 model to generate image retrievals and/or reconstructions

Prerequisites

You'll need an internet connection, terminal access, and a GPU. This has been primarily tested on Linux (RHEL and Rocky Linux 9.6) with a NVIDIA RTX 6000 Ada Generation GPU (49GB VRAM).

This requires familiarity with Python, Git, and the command line.

We'll install required software (Git, Git LFS, Python, Python packages) along the way.

Specific code snippets that you should run will be formatted like this. Within code snippets, paths that might differ on your computer will be formatted like <this>.

Setup

In this section, we will set up Git, install a uv environment with required packages, and clone repositories containing the analysis code, data, and large files. This should all be installed on a computer with a GPU in order to perform MindEye analysis.

Git and Git LFS

1. Check if the Git command line interface (CLI) is installed: git --version. If it prints a version number, you're good. If not, install it:

   • MacOS: brew install git
   • Ubuntu/Debian Linux: sudo apt install git
   • For other systems and additional details, see documentation.

2. We use Git LFS to handle large files such as model weights. Check if it's installed: git lfs --version. If not, install it:

   • MacOS: brew install git-lfs
   • Ubuntu/Debian Linux: sudo apt install git-lfs
   • For other systems and additional details, see documentation. After installation, run once per user account: git lfs install

Cloning this repository

1. On the command line, navigate to the directory in which you want to set up this repository: cd <path/to/directory>
2. Clone this repository with git clone https://github.com/brainiak/rtcloud-projects.git. This will create a folder rtcloud-projects/, containing MindEye and other projects.

Installing uv environment

We use uv, a fast Python package manager (documentation) to manage Python versions and dependencies. You may be familiar with tools like conda and pip for package management; uv is a much faster and more modern alternative. We have exact versions of Python and all dependencies so you can reproduce the environment exactly. Even if you don't have Python installed on your system, uv will take care of this for you.

1. Install uv
   • MacOS/Linux: curl -LsSf https://astral.sh/uv/install.sh | sh
   • Verify install using uv --version
     • You may need to run source $HOME/.local/bin/env for uv to be a recognized command
2. Create the environment
   1. cd rtcloud-projects/mindeye/conf
   2. Run: uv sync. This will:
      • Install the Python version specified in .python-version
      • Install all dependencies from pyproject.toml pinned by uv.lock
3. Activate the environment
   1. source .venv/bin/activate
   2. Check the Python version: python --version; it should match the version listed in the file .python_version located in the config folder
   3. You can use the command deactivate to deactivate the environment

Installing FSL

We use FSL for real-time compatible preprocessing, using tools such as MCFLIRT and FLIRT for motion correction and registration, respectively.

1. Install FSL
   • MacOS/Linux: curl -Ls https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/releases/getfsl.sh | sh -s
   • For other systems and additional details, see documentation
   • Successful installation should output "FSL successfully installed"
     • You might need to run export FSLDIR=/usr/local/fsl and source $FSLDIR/etc/fslconf/fsl.sh in order to make any of the fsl commands available
   • Verify install using flirt -version

Cloning other necessary repositories

1. Clone the repository containing large model files
   • Navigate to the desired location (for example, this can be inside rtcloud-projects/mindeye): cd <path/to/rtcloud-projects/mindeye>
   • git clone https://huggingface.co/datasets/rishab-iyer1/rt_all_data
2. Clone the repository containing data related to Princeton 3T scans
   • Navigate to the desired location (for example, this can be inside rtcloud-projects/mindeye): cd /path/to/rtcloud-projects/mindeye
   • git clone https://huggingface.co/datasets/rishab-iyer1/3t
3. Create config/config.json with the appropriate file paths. config/config_example.json is provided for reference.
   • project_path should be set to <path/to/rtcloud-projects/mindeye>
   • storage_path should be set to <path/to/rt_all_data>
   • data_path and derivatives_path should be set to <path/to/3t/data> and <path/to/3t/derivatives>, respectively
   • fsl_path should be set to the fsl/bin folder containing the executables of various FSL functions. This is often located in your home directory at ~/fsl/bin
   • The local copy that you update (config/config.json) will be automatically ignored by git (due to .gitignore). It is good practice to not track user-specific file paths with version control, since each user will have different file systems.

Get started!

First, for an overview of the pipeline and description of stimuli, read docs/00-pipeline.md. We provide three guides, described here. The first two are quickstart guides that focus on specific code snippets for you to run and are intended to simplify the setup process. The third document is a procedural guide that provides an overview of the different components that go into the real-time experiment.

If you want to reproduce our analysis or make improvements using existing data, the first guide should suffice. If you are aiming to run your own real-time MindEye experiment, you should plan on building up to the third guide. We strongly recommend completing these in order.

1. docs/01-quickstart_simulation.md: run a simulated real-time MindEye analysis on a GPU using pre-collected data from Princeton, without having to install RT-Cloud. Start here if you're new!
2. docs/02-quickstart_realtime.md: use full RT-Cloud functionality with MindEye, including streaming pre-collected data from Princeton in real-time and analyzing the data as it comes in. Use this to prepare to run your own real-time scan.
3. docs/03-experiment_guide.md: for instructions on how to run a real-time MindEye experiment on a new participant at your MRI facility.

Running your own real-time scan

If you are planning to run your own real-time MindEye scans, here are additional repositories you may find useful. See docs/03-experiment_guide.md for an overview of all the components that go into preparing for and running the real-time scan.

1. mindeye_task: contains all materials required to run NSD-like tasks with PsychoPy
2. mindeye_preproc: contains materials used at Princeton for offline data preprocessing in preparation for the real-time session
3. mindeye_offline: contains materials to fine-tune MindEye on offline-preprocessed data in preparation for the real-time session