he first artificial neuron was a simplified real one (McCulloch & Pitts, 1943). For decades afterwards the two fields drifted apart. They met again around 2014, when networks trained to recognise objects turned out to predict the activity of neurons in the monkey brain better than models built for the purpose.

the question

If a network trained on photographs predicts the responses of visual neurons, is it a model of the visual system? Or two different machines that happen to agree on one test?

the classic finding

Hubel and Wiesel found cells in cat visual cortex tuned to edges at particular angles, and more complex cells that pooled them (Hubel & Wiesel, 1962). Fukushima built that hierarchy into the neocognitron (Fukushima, 1980), the ancestor of convolutional networks (Lindsay, 2021). Yamins and colleagues (2014) found that the better a network was at recognising objects, the better its upper layers predicted responses in the monkey's inferior temporal cortex. Representational similarity analysis gave a way to compare a model and a brain region without matching them unit for unit (Kriegeskorte, 2015). The comparison became a public leaderboard, Brain-Score (Schrimpf et al., 2018)(Schrimpf et al., 2020)(Brain-Score, 2018).

what held up, and what did not

Prediction is not explanation. Bowers and colleagues argue that models that score well on neural benchmarks fail many classic tests of human vision [paraphrase, not a quotation]. Feather and colleagues generated images that a model treats as identical to a natural one; people often could not recognise them at all, a sign that the model's invariances are not ours. Learning is another gap: networks learn by backpropagation, and whether brains do anything like it is open (Lillicrap et al., 2020). Even V1, the best-studied area, was far from understood twenty years ago (Olshausen & Field, 2005).

what it looks like in machines

The traffic now runs both ways. Neuroscience supplied convolution and, arguably, attention and replay (Hassabis et al., 2017); networks are now used as hypotheses about what brain areas are optimised for (Richards et al., 2019).

what is still open

What a good score means; whether the right comparison is between representations, behaviour or learning; and the problem from connectomics: full access to a system is not understanding of it (Jonas & Kording, 2017).

in the tour: the room called "The comparison room" has a placard called "Scored against the brain" ···

Reading list

1. McCulloch & Pitts (1943). A logical calculus of the ideas immanent in nervous activity. The Bulletin of Mathematical Biophysics. doi:10.1007/BF02478259
2. Hubel & Wiesel (1959). Receptive fields of single neurones in the cat's striate cortex. The Journal of Physiology. doi:10.1113/jphysiol.1959.sp006308
3. Hubel & Wiesel (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of Physiology. doi:10.1113/jphysiol.1962.sp006837
4. Fukushima (1980). Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics. doi:10.1007/bf00344251
5. Lindsay (2021). Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future. Journal of Cognitive Neuroscience. doi:10.1162/jocn_a_01544
6. Yamins et al. (2014). Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences. doi:10.1073/pnas.1403112111
7. Kriegeskorte (2008). Representational similarity analysis – connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience. doi:10.3389/neuro.06.004.2008
8. Kriegeskorte (2015). Deep Neural Networks: A New Framework for Modeling Biological Vision and Brain Information Processing. Annual Review of Vision Science. doi:10.1146/annurev-vision-082114-035447
9. Schrimpf et al. (2018). Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like?. bioRxiv. doi:10.1101/407007
10. Schrimpf et al. (2020). Integrative Benchmarking to Advance Neurally Mechanistic Models of Human Intelligence. Neuron. doi:10.1016/j.neuron.2020.07.040
11. Brain-Score (2018). Brain-Score: an open benchmark platform scoring models of vision and language against neural and behavioural data. Website, ongoing. www.brain-score.org/
12. Bowers et al. (2023). Deep problems with neural network models of human vision. Behavioral and Brain Sciences. doi:10.1017/s0140525x22002813
13. Feather et al. (2023). Model metamers reveal divergent invariances between biological and artificial neural networks. Nature Neuroscience. doi:10.1038/s41593-023-01442-0
14. Lillicrap et al. (2020). Backpropagation and the brain. Nature Reviews Neuroscience. doi:10.1038/s41583-020-0277-3
15. Olshausen & Field (2005). How Close Are We to Understanding V1?. Neural Computation. doi:10.1162/0899766054026639
16. Hassabis et al. (2017). Neuroscience-Inspired Artificial Intelligence. Neuron. doi:10.1016/j.neuron.2017.06.011
17. Richards et al. (2019). A deep learning framework for neuroscience. Nature Neuroscience. doi:10.1038/s41593-019-0520-2
18. Jonas & Kording (2017). Could a Neuroscientist Understand a Microprocessor?. PLOS Computational Biology. doi:10.1371/journal.pcbi.1005268

see also

	Walter Pitts — the first unit
	archive: the cat
	interpretability