Ten years ago, when Nick Bostrom’s now-famous book, Superintelligence, was published, it made a bit of a splash. Many were the readers then, and many more are the citations now. I applaud a fellow philosopher for his commercial success, but even good things must sometimes end. My philosophical career has been nowhere near as impactful or success-filled, but I feel confident nonetheless in the critique I intend to venture here. In the discussion to follow, I have the distinct advantage of being not only correct, but possessing empirical evidence that strongly supports my claims–unlike every AGI Doom enthusiast on earth.
For Bostrom’s ideas, the problems started with ChatGPT 3.5, which was a runaway success that invaded the hearts and minds of computer users everywhere. On November 30, 2022, ChatGPT invaded my social circle in the Web3 community and by mid-2023, I was writing fairly heavy articles about it. A longtime student of consciousness and philosophy, I was in hog heaven - but not because AGI or ASI was on the way. I was excited about the humanistic implications of these technologies, and I am happy to say that the interceding time has done everything to reify my initial read of the situation, and nothing to disprove it. This essay will, as briefly as possible, explain what I’ve learned along the way.
Here are the four most important questions for us to answer, each of which will have its own section in the work to follow:
- What is Superintelligence?
- What is an LLM?
- What is consciousness, and why is it required for action?
- What are we building, if not an artificially conscious entity?
Once we are finished, we will better understand what has happened, what is happening today, and what is likely to happen tomorrow. Thinkers from Gary Marcus to Melanie Mitchell will find support for their ideas here, but my intention is to use my own voice to explain the concepts here. One key development we must note, however, is the incredible advance in mechanistic interpretability yielded by a team I applied to join last fall: Anthropic’s reverse engineering team. Though the title seems a bit misleading, “Mapping the Mind of a Large Language Model” is a breakthrough in every sense of the term. I would not be writing this if Anthropic’s work had not revealed a key property of LLMs: determinism. Please, allow me to explain.
What is Superintelligence?
In 2014, Nick Bostrom wrote a book that described a control problem in an unequal power relationship. The problem, for Bostrom, with creating an artificial intelligence that was far greater in power than any or perhaps all human minds, had to do with what that intelligence would get up to. In a notable example, when instructed to create paperclips, the machine optimizes the users of the paperclips out of the equation and ends up killing all humans to make more paperclips.
The concept of superintelligence given voice by Bostrom has a few key attributes: it is very, very smart - smarter than anything else; it is a machine that learns, thinks, and acts; and it is dangerous because it has no feelings or social ties to humanity. Artificial general intelligence, as it is conceived of today, is a step on the path to artificial superintelligence and Wikipedia currently explains it as follows:
Screengrab source: https://en.wikipedia.org/wiki/Artificial_general_intelligence
Despite the goalsetting by OpenAI and DeepMind, as well as Anthropic, AGI is a terribly unclear concept. While LLMs are extremely useful tools for processing gigantic amounts of information, consciousness is related to agency, the ability to make choices and act autonomously.
Intelligence, defined by Oxford Dictionary & accessed via Google Search 6/1/2024
Intelligence, in the sense of the word we mean when we refer to living organisms, relates to an activity motivated by metabolism and entropy. Living organisms decay if they do not take in nutrition and dispose of waste, and this foundation of life generates an autonomous cognitive process. Complex or simple, this process is the part of a living organism which evaluates the surrounding environment and makes choices to suspend entropy and further the metabolic chemical processes that sustain all organisms over time. In the first sense above, the intelligence is the conscious or non-conscious will to self-perpetuation of a particular organism.
Note that, in the sense of ANI, we have a very specific subject upon which an algorithm rests, functioning as a sort of librarian for people who wish to become more familiar with that subject. As AI concepts become more general, they break down, and it is possible that people begin to move from the second definition above, the information-oriented definition, to the first, the living organism-centric definition.
If the conflation between meanings of the word intelligence is to blame for the rise of concepts such as AGI, which push from the second variation to the first, then perhaps this philosophical essay can help to remedy the issues facing the nascent computational AI industry. Perhaps we have discovered a naked emperor at the heart of the AGI/ASI movement.
The Superintelligent Emperor Has No Clothes
The problems with the concept of artificial intelligence beyond the “narrow AI” concept mentioned in the Wikipedia screengrab above start when one begins to wonder what “matches or surpasses human capabilities” means. A camera with a powerful zoom can capture more detail at a farther distance than a human eye, but we wouldn’t even equate the functionality of a camera to an eye because we clearly need eyes as well as cameras on the one hand, and on the other, cameras don’t have any of the other equipment that you need to make use of things like eyes–in other words, cameras lack the property we call integration, which eyes do have. Seeing isn’t everything.
Intelligence in general is a poorly understood concept, even among experts in the literature around it. Currently, there is something of a revolution occurring in the field as biologists and doctors who study the brain and cognitive function in human beings begin to realize that, unless the mind in question cares about some subject, there is little point doing intelligence testing with respect to related matters. Similarly, early intelligence tests unfairly privileged certain ways of thinking and speaking. This clumsy cognitive science led to all manner of horrors, but fortunately seems on the way to being remedied as thinkers and scholars grow to better understand how minds work and how best to help improve them.
To be regarded as intelligent, a machine or algorithm would first need to care about the world - which is impossible, given what we know about machines. Varela & Maturana hypothesized that cognition and metabolism always appear together in nature because they are the same thing, and no evidence to contradict this hypothesis has yet appeared. Artificial intelligence can be called a misnomer if we want to pursue this line of reasoning about cognition, but intelligence agencies gather intelligence and in a sense, intelligence is not only a property of a rational mind but also a thing that minds can produce and share with one another. For this reason, ANI (artificial narrow intelligence) does actually seem well-named. A chatbot with a good algorithm allows a person to gather information and take actions with respect to said information and value can be had on all sides.
The concept breaks down, however, as we move into more general territory. If simply finding information about subjects in a better and faster way than an ordinary human mind can is all that a program needs to qualify as artificial intelligence, the card catalog at your local library is a narrow artificial intelligence. In general, however, programs and algorithms need some sort of an interface that users access via natural language to qualify as artificial intelligence. A typical program that people think of as AI has a repository of information that is queryable via natural language. Even algorithms that exist physically and have high functionality in a narrow band of use-cases just are what they are and are not considered “intelligent” perhaps because they do not adapt in response to natural language queries.
A more general version of an incredibly intelligent system might be Google Search, which may be declining a bit in terms of quality but which nonetheless does an excellent job of indexing content online - and in fact far outperforms any individual person’s ability to do so. However, Google Search is not a candidate for AGI status, and we have to wonder why. The reason ultimately seems to be that it does not refactor content and that it is deterministic–the same search term yields the same results, most of the time. Otherwise, a user inputs text and the very smart algorithm matches that text to information in its system, which it then outputs back to the user. This flow is exactly the same as the modern LLM chatbots, but with one rub: LLMs were thought to be potentially nondeterministic before the recent Anthropic paper proved that mechanistic interpretability was possible. Mechanistic interpretability is only possible if LLMs are deterministic, and hence we once again find that something we thought was incalculably complex has collapsed into a predictable state of affairs.
As so-called tech luminaries wax on and on about the generality of the intelligence of LLMs, a discerning mind might want to know why these algorithms are thought to be intelligent. If merely rearranging the content that the search returns makes these algorithms conscious, then it seems that consciousness must not be all that we thought it was. After all, the basic function of an LLM is not all so different from Google Search - it takes in information, processes it against a database or network of information, and returns the computed result to the user who asked for it.
AGI and ASI both assume, unlike ANI, that a leap forward has been made - that the machine has now developed a will and, indeed, a mind of its own. However, eighteen months post-ChatGPT 3.5, there is absolutely no evidence of any sort of consciousness or autonomous action in any sort of LLM. Further, LLMs have serious limitations including unreliability as well as a dropoff in output quality when LLM-generated text is used as training data for a new LLM.
To conclude this section, we must acknowledge that the question we started with, “What is Superintelligence?” has not been particularly well-answered. The reason for this is that there is no handle by which we can grasp the concept - it is not grounded in reality. Smelling the AGI and smelling the BS are functionally equivalent, here. It is perhaps time for the AI industry to clarify its terms and move from the first definition of intelligence to the second.
What is an LLM?
If not a conscious entity, what is this talking algorithm that we call an LLM? It’s an important question, given that intelligence and care are related to one another. The technical answer is that an LLM is a package of computer programs that processes a massive amount of data and regurgitates it on command. The big leap forward of ChatGPT 3.5 has to do with both the speed and the quality of this regurgitation, but speed and quality of this service do not manage to somehow transform it into something it wasn’t before.
Lacking autonomy, LLMs are only able to respond to the queries of users. This is important because it directly eliminates the “learning” characteristic of AGI or ASI before the algorithm even gets built–what LLMs do is not careful consideration in an analogous process to consciousness, but rather a navigation of an incomprehensibly large graph of speech acts made by human beings. In this sense, an LLM is more of a telescope than a brain. The usefulness of these networks is beyond dispute at this point in time, but the key takeaway is that LLMs absorb a remarkable amount of data and store it in a remarkable way that does not resemble consciousness or intelligence.
When training an LLM, the programmers will feed a massive amount of data into a computer algorithm that compresses this information in a neural net that breaks it down into tiny chunks and analyzes the resulting structure at all levels. Some of these levels are completely ignored by people, but the information that is preserved by an LLM’s neural network is agnostic to whether or not people think of long words as a series of tokens. In popular use, the token is simply an irrelevant attribute that speech acts can have.
LLMs are remarkable because they enable a user to process, using only natural language, a massive amount of information and to search further, wider, and with less effort for information that they need. However, without training data made by real people, the effectiveness of LLMs is greatly diminished. Training LLMs on text generated by other LLMs is ineffective at best because the models develop biases as a result of their training and these biases self-reinforce upon training.
The reason that LLM data is not good for training future LLMs is simple: LLMs are not speakers, they are speech act simulators. If you feed the entire internet into a well-designed neural network, you are essentially using that neural network to index and process the content found on the internet. The trained entity that emerges after this activity is complete is a program that handles the information it was trained upon and cannot generalize beyond the bounds of its training because it quite literally has not yet been fed the lines it needs to recite.
What is Consciousness, and Why is it Required for Action?
Consciousness is still poorly understood at the foundational level, but suffice it to say that, for now, consciousness is a modeling process that runs in large, intelligent brains to create ever-more efficient survival mechanisms in the organisms they are attached to. Action is the modification of the environment of an organism by that organism at any scale to self-perpetuate in the way perceived to be most likely to reduce entropy and ensure continued successful metabolic activity. Consciousness is always related to action because consciousness contains the model which creates action - if the organism has no information about its surroundings and internal states, action is impossible because there can be no preference for one state or another.
Since action is all about living matter’s general preference for reduced entropy as opposed to increased entropy, it makes no sense to speak of machines acting. Instead, machines and other inanimate objects react to forces upon them in accordance with the laws of physics, just like a rock or a glass of water. Consciousness might therefore be said to be a way of interacting with the inanimate objects of the natural world that enables living beings to impact their surroundings, with the general intention to sustain metabolism and arrest/reverse entropy.
Consciousness has a few remarkable properties, which I now study as a Neuroscience Ph.D student at Texas Tech University. My work is focused on the concept of cognitive effort, presently, and though there is not as much information on the subject at this time as I would like, it is rewarding to think through the puzzles around what happens when people think *hard* about particular subjects.
One issue with contemporary consciousness studies is that the physical underpinnings of the process are incredibly complex–unpredictable, nondeterministic, and ultimately I believe we will understand them to be mostly arbitrary with respect to the actual conscious experience that an individual has as they occur. In the fMRI labs, for example, the same process can be repeatedly engaged and neural activation will decrease over time in response to a similar stimulus. Novelty has a disproportionate impact upon the activity of our cortical tissue, and this likely is related to the rapid rate at which we are able to learn to expect the things that happen to us.
Additionally, even things such as emotions, which researchers have hoped to make progress on for decades now by locating them at particular centers in the brain, are unrepeatable. Think of a time when you got angry, and then think of another time you got angry. You were feeling a similar way, but it was not *the same* and this would have held true, had images of the activation patterns in your brain been taken and compared. The reason for this is that brains are incredibly sensitive to the environment and the circumstances under which a particular impression is made, and there is another strange property that brains have: different neurons will fire each time a repetition of a particular stimulus takes place. So you experience something similar, but the actual hardware of your grey matter that interacts to ostensibly produce the sensation you feel is quite different and this can be proven in the lab.
My theory for this is that higher order processing is virtual in nature, that is, it is arbitrary with respect to the underlying hardware. This is similar to the laptop I’m typing this essay into, which does not care what I write and will change about the same over the course of my usage whether I write this advanced paper about consciousness or type a long stream of gibberish into it. The virtual nature of higher order thinking in the brain explains why mindreading is so difficult to do even for the best teams who try the hardest.
And ultimately, if the nature of thinking is not reducible to the physical action of neurons, then we cannot argue that thought is a deterministic system and we will never be successful in reverse engineering it to the point of mechanistic interpretability. This is why the recent breakthrough by Anthropic is so significant. It proves that AGI is impossible because LLMs, despite their massive complexity, are ultimately deterministic. There is no virtual metalayer in which these algorithms develop personalities and decide what to say and do; rather, they are large but simple and we should eventually be able to predict their actions.
Brains are smaller but more complex and less predictable because they are conscious, which ultimately means that a virtual process that cannot be seen is responsible for the actions of many of the components of the physical substrate.
What Are We Building?
All of the firms and people working on AGI/ASI are in fact doing something, even if that thing is not a thing that makes sense on a philosophical level in the way they describe it. And the effectiveness of these teams in pursuit of this end is undeniable. ChatGPT 3.5, Claude3, and the other LLMs that have been designed and built and brought to market are valuable technologies. As we finish our exploration here, it is important to note that something big is going on, even if it isn’t the thing everyone is talking about.
So, backing up a step, an LLM is an algorithm that is trained on a massive amount of data and that can reconfigure, reshape, and regurgitate that data in ways that make it more accessible to users. The LLM is a wonderful tool for searching large amounts of information because it can move so quickly, but much of the time the information that comes back needs to be double and triple checked because artifacts, errors, and other problems occur on a regular basis.
As mechanistic interpretation improves, so too will the quality of the information users get out of LLMs and related technologies. This is significant because these wonderful tools enable researchers such as myself to expand our knowledge to unseen heights, pushing our expertise into novel fields and then drawing upon this breadth of understanding to more effectively evaluate claims.
And it is in fact a claim I would like to make that has motivated me to write this essay, which I could not do the same way without looking back at the original, despite being the author (and not even quite finished yet). Here is my claim:
Instead of AGI, we should be calling it CHI - collective human intelligence.
The name of the technology is extremely important because it is the first handle by which a new mind is able to grasp it. Referring to intelligence as artificial creates a sense that there is agency inside of the black box of the new technology, which is a natural and correct inference because every prior example of something intelligent has co-occurred with the agency of that thing.
In the case of LLMs, especially now that determinism is a known property these machines have, we can safely say that there is no agency involved and that there cannot be, given their properties.
Hence, it is important to go back to the drawing board and come up with a new word for these remarkable, useful, amazing machines (which are nonetheless not intelligent). I suggest CHI, collective human intelligence, because the power of the LLM is its ability to process massive amounts of human-generated intelligence (in the second sense of the word above) and feed it back to individual people in a way that is unparalleled in terms of its digestibility.
What an LLM actually does, rather than think or experience or make decisions, is to function as a sort of looking-glass that feeds in targeted and accurate information to individuals who are looking for that information, thus giving even the layperson something like the access to specialized domains of knowledge that would previously have required decades of study and preparation to understand. Thus, rather than creating an AGI that has a will and conducts actions on its own, we should be focused on creating high-quality CHI tools to help those of us who are living organisms, agents, and students to study and understand on the highest level possible to any of us.