Instantaneous Neural Networks and AI
In response to my essay on the One Observer Theorem, many readers have written asking about my most fundamantal technical contribution to AI.
But before I respond, here’s some context. Many of the amazing achievements of AI are based on the theory of neural networks and algorithms for training them. These networks are designed to emulate the way the brain works and because of their processing speed and memory advantage they can now perform cognitive tasks that lie beyond the capability of any person.
AI is now everywhere: in decision making and in generative AI (GenAI) that appears to mimic human creativity, and it may be used for music and image generation, game development, for conversation on random topics using LLMs (large language models), and in diverse other applications.
My research on AI has included all kinds of neural networks, which may be feedforward or allow limited or unconstrained feedback. The feedback network is fascinating for it is a model for complex systems in which the component subsystems interact in many ways, and it is also a ground-level model of the brain. The feedforward network is a model of a deliberate, goal-directed computation.
If out of this research field I were to pick just one technical contribution, it is the theory of instantaneously trained neural networks (INNs) [Reference 1].
I made the discovery rather spontaneously in response to a classroom question. I was asked something that I had never thought about, and I blurted out an answer that, on later analysis, turned out to the the theory of INNs. It is remarkable that this was a way of learning instantaneously, without any computation!
Parenthetically, this leads to the question of how discovery is made. My own view is that most creative insights are spontaneous, and one may come by them in dream or trance. The Vedantic literature calls the source of creativity the hiranyagarbha (golden womb).
I received a patent for instantaneous neural networks [Reference 2]. Soon I heard from one of America’s most famous VCs (Venture Captialists) who wished for me to do a startup and work full time to develop the technology. This I didn’t wish to do, and the technology was licensed by the university, to which such inventions are normally assigned, to a company, which developed many applications based on it and resold the technology in different bundles in the areas of pattern recognition, prediction, and gaming.
In the business world, the ability to quickly predict the directionality and value of change is of great value, and INNs have been used in the forecasting of financial time-series as an embedded component of proprietary software.
They were also proposed for use in an AI-based search engine, and IEEE Intelligent Systems invited me to write a story on it a long time ago [References 3 and 4]. I had named this search engine anvish.com (after the Sanskrit term for “search”) but it was reenamed by the company that bought the technology.
I just checked and found that anvish.com is now available for purchase.
I didn’t follow the field closely after this for my research veered into quantum computing for several years and most recently it has focused on e-dimensionality in physics and biology.
I do hear from computer scientists from time to time asking for hints on how to use INNs for specific applications. I tell them that the trick is to create an engine that seamlessly binds to the application domain. In other words, there need to be appropriate coding circuits that map the variables of the application area to the neural network variables. This requires deep understanding of the application domain and an excellent intuition.
A judicious use of INNs within a framework of learning logic using tokens and scores that require many parameters can considerably speedup the training of LLMs.
References
[1] S. Kak, New algorithms for training feedforward neural networks. Pattern Recognition Letters, vol. 15, pp. 295–298, 1994.
[2] S. Kak and J.F. Pastor, Neural networks and methods for training neural networks. U.S. Patent № 5426721, June 20, 1995.
[3] S. Kak, Faster web search and prediction using instantaneously trained neural networks. IEEE Intelligent Systems, vol. 14, pp. 79–82, November/December 1999.
[4] B. Shu and S. Kak, A neural network based intelligent metasearch engine. Information Sciences, vol. 120, pp. 1–11, 1999.
See also The Age of Artificial Intelligence