University of Leicester
Browse
1-s2.0-S1571064518301106-main.pdf (3.09 MB)

The unreasonable effectiveness of small neural ensembles in high-dimensional brain.

Download (3.09 MB)
journal contribution
posted on 2019-09-09, 15:45 authored by Alexander N. Gorban, Valeri A. Makarov, Ivan Y. Tyukin
Complexity is an indisputable, well-known, and broadly accepted feature of the brain. Despite the apparently obvious and widely-spread consensus on the brain complexity, sprouts of the single neuron revolution emerged in neuroscience in the 1970s. They brought many unexpected discoveries, including grandmother or concept cells and sparse coding of information in the brain. In machine learning for a long time, the famous curse of dimensionality seemed to be an unsolvable problem. Nevertheless, the idea of the blessing of dimensionality becomes gradually more and more popular. Ensembles of non-interacting or weakly interacting simple units prove to be an effective tool for solving essentially multidimensional and apparently incomprehensible problems. This approach is especially useful for one-shot (non-iterative) correction of errors in large legacy artificial intelligence systems and when the complete re-training is impossible or too expensive. These simplicity revolutions in the era of complexity have deep fundamental reasons grounded in geometry of multidimensional data spaces. To explore and understand these reasons we revisit the background ideas of statistical physics. In the course of the 20th century they were developed into the concentration of measure theory. The Gibbs equivalence of ensembles with further generalizations shows that the data in high-dimensional spaces are concentrated near shells of smaller dimension. New stochastic separation theorems reveal the fine structure of the data clouds. We review and analyse biological, physical, and mathematical problems at the core of the fundamental question: how can high-dimensional brain organise reliable and fast learning in high-dimensional world of data by simple tools? To meet this challenge, we outline and setup a framework based on statistical physics of data. Two critical applications are reviewed to exemplify the approach: one-shot correction of errors in intellectual systems and emergence of static and associative memories in ensembles of single neurons. Error correctors should be simple; not damage the existing skills of the system; allow fast non-iterative learning and correction of new mistakes without destroying the previous fixes. All these demands can be satisfied by new tools based on the concentration of measure phenomena and stochastic separation theory. We show how a simple enough functional neuronal model is capable of explaining: i) the extreme selectivity of single neurons to the information content of high-dimensional data, ii) simultaneous separation of several uncorrelated informational items from a large set of stimuli, and iii) dynamic learning of new items by associating them with already "known" ones. These results constitute a basis for organisation of complex memories in ensembles of single neurons.

Funding

We are grateful to our coauthors B. Grechuck, E.M. Mirkes, D. Prokhorov, K. Sofeykov, I. Romanenko, J. Makarova, C. Calvo, A. Golubkov, and R. Burton. Without our joint work [35], [36], [39], [42], [95], [96], this review would not have been possible. The work was supported by the Ministry of Education and Science of the Russian Federation (Project No. 14.Y26.31.0022). Work of ANG and IVT was also supported by Innovate UK (Knowledge Transfer Partnership grants KTP009890; KTP010522) and University of Leicester. VAM acknowledges support from the Spanish Ministry of Economy, Industry and Competitiveness (grant FIS2017-82900-P).

History

Citation

Physics of Life Reviews, 2019, 29, pp. 55-88

Author affiliation

/Organisation/COLLEGE OF SCIENCE AND ENGINEERING/Department of Mathematics

Version

  • VoR (Version of Record)

Published in

Physics of Life Reviews

Publisher

Elsevier

eissn

1873-1457

Acceptance date

2018-09-20

Copyright date

2018

Available date

2019-09-09

Publisher version

https://www.sciencedirect.com/science/article/pii/S1571064518301106?via=ihub

Language

en

Usage metrics

    University of Leicester Publications

    Categories

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC