Introduction

Brain encoding and decoding

_images/brain-encoding-decoding.png

Fig. 1 To test the consistency of representations in artificial neural networks (ANNs) and the brain, it is possible to encode brain activity based on ANN presented with similar stimuli, or decode brain activity by predicting the expected ANN activity and corresponding annotation of cognitive states. Figure from Schrimpf et al. (2020) [SKH+20], under a CC-BY 4.0 license.

This jupyter book presents an introduction to brain encoding and decoding using fMRI. Brain decoding is a type of model where we try to guess what a subject is doing, based on recordings of brain activity. Brain encoding is the reverse operation, where we use machine learning tools to predict the activity of the brain, either based on annotations of the cognitive states of the subject, or using features learned by an artificial neural network presented with the same stimuli as the subject. The tutorials make heavy use of nilearn manipulate and process fMRI data, as well as scikit-learn and pytorch to apply machine learning techniques on the data.

This resource was developed for use at the Montreal AI and Neuroscience (MAIN) conference in November 2021.

Setup

There are two ways to run the tutorials: local installation and Binder.

Binder

If you wish to run the tutorial in Binder, click on the rocket icon to launch the notebook 🚀

Warning

The computing resource on Binder is limited. Some cells might not execute correctly, or the data download will not be completed. For the full experience, we recommend using the local set up instruction.

Acknowledgements

Parts of the tutorial are directly adapted from a nilearn tutorial on the Haxby dataset.

This tutorial was prepared and presented by Pravish Sainath Shima Rastegarnia, Hao-Ting Wang Loic Tetrel and Pierre Bellec.

Some images and code are used from a previous iteration of this tutorial, prepared by Dr Yu Zhang.

It is rendered here using Jupyter Book,

References

SKH+20

Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, Kailyn Schmidt, Daniel L K Yamins, and James J DiCarlo. Brain-Score: which artificial neural network for object recognition is most Brain-Like? bioRxiv, pages 407007, January 2020.