Neurocompositional computing in human and machine intelligence: A tutorial - Smolensky, Paul; McCoy, R. Thomas; Fernandez, Roland; Goldrick, Matthew; Gao, Jianfeng (Microsoft, 2022)
The past decade has produced a revolution in Artificial Intelligence (AI), after a half-century of AI repeatedly failing to meet expectations. What explains the dramatic change from 20th-century to 21st-century AI, and how can remaining limitations of current AI be overcome? Until now, the widely accepted narrative has attributed the recent progress in AI to technical engineering advances that have yielded massive increases in the quantity of computational resources and training data available to support statistical learning in deep artificial neural networks. Although these quantitative engineering innovations are important, here we show that the latest advances in AI are not solely due to quantitative increases in computing power but also qualitative changes in how that computing power is deployed. These qualitative changes have brought about a new type of computing that we call neurocompositional computing. In neurocompositional computing, neural networks exploit two scientific principles that contemporary theory in cognitive science maintains are simultaneously necessary to enable human-level cognition. The Compositionality Principle asserts that encodings of complex information are structures that are systematically composed from simpler structured encodings. The Continuity Principle states that the encoding and processing of information is formalized with real numbers that vary continuously. These principles have seemed irreconcilable until the recent mathematical discovery that compositionality can be realized not only through the traditional discrete methods of symbolic computing, well developed in 20th-century AI, but also through novel forms of continuous neural computing—neurocompositional computing. The unprecedented progress of 21st-century AI has resulted from the use of limited—first-generation—forms of neurocompositional computing. We show that the new techniques now being deployed in second-generation neurocompositional computing create AI systems that are not only more robust and accurate than current systems, but also more comprehensible—making it possible to diagnose errors in, and exert human control over, artificial neural networks through interpretation of their internal states and direct intervention upon those states. Note: This tutorial is intended for those new to this topic, and does not assume familiarity with cognitive science, AI, or deep learning. Appendices provide more advanced material. Each figure, and the associated box explaining it, provides an exposition, illustration, or further details of a main point of the paper; in order to make these figures relatively self-contained, it has sometimes been necessary to repeat some material from the text. For a brief introduction and additional development of some of this material see “Neurocompositional computing: From the central paradox of cognition to a new generation of ai systems” (arXiv:2205.01128; to appear, AI Magazine)
52 pages main text, 78 pages total, 11 figures, 2 Appendices, 239 references. For a short presentation of some of this material, see https://arxiv.org/abs/2205.01128 (to appear in AI Magazine).