Eight to Late

There is much debate about whether Large Language Models (LLMs) have reasoning capabilities: on the one hand, vendors and some researchers claim LLMs can reason; on the other, there are others who contest these claims. I have discussed several examples of the latter in an earlier article, so I won’t rehash them here. However, the matter is far from settled: the debate will go on because new generations of LLMs will continue to get better at (apparent?) reasoning. 

It seems to me that a better way to shed light on this issue would be to ask a broader question: what purpose does language serve?

More to the point: do humans use language to think or do they use it to communicate their thinking? 

Recent research suggests that language is primarily a tool for communication, not thought (the paper is paywalled, but a summary is available here). Here’s what one of the authors says about this issue:

““Pretty much everything we’ve tested so far, we don’t see any evidence of the engagement of the language mechanisms [in thinking]…Your language system is basically silent when you do all sorts of thinking.”

This is consistent with studies on people who have lost the ability to process words and frame sentences, due to injury. Many of them are still able to do complex reasoning tasks such as play chess or solve puzzles, even though they cannot describe their reasoning in words. Conversely, the researchers find that intellectual disabilities do not necessarily impair the ability to communicate in words.

–x–

The notion that language is required for communicating but not for thinking is far from new. In an essay published in 2017, Cormac McCarthy noted that:

“Problems in general are often well posed in terms of language and language remains a handy tool for explaining them. But the actual process of thinking—in any discipline—is largely an unconscious affair. Language can be used to sum up some point at which one has arrived—a sort of milepost—so as to gain a fresh starting point. But if you believe that you actually use language in the solving of problems I wish that you would write to me and tell me how you go about it.”

So how and why did language arise? In his epic book, The Symbolic Species, the evolutionary biologist and anthropologist Terrence Deacon suggests that it arose out of the necessity to establish and communicate social norms around behaviours, rights and responsibilities as humans began to band into groups about two million years ago. The problem of coordinating work and ensuring that individuals do not behave in disruptive ways in large groups requires a means of communicating with each other about specific instances of these norms (e.g., establishing a relationship, claiming ownership) and, more importantly, resolving disputes around perceived violations.   Deacon’s contention is that language naturally arose out of the need to do this.

Starting with C. S. Pierce’s triad of icons, indexes and symbols, Deacon delves into how humans could have developed the ability to communicate symbolically.  Symbolic communication is based on the powerful idea that a symbol can stand for something else – e.g., the word “cat” is not a cat, but stands for the (class of) cats.  Deacon’s explanation of how humans developed this capability is – in my opinion – quite convincing, but is by no means widely accepted. As echoed by McCarthy in his essay, the mystery remains:

“At some point the mind must grammaticize facts and convert them to narratives. The facts of the world do not for the most part come in narrative form. We have to do that.

So what are we saying here? That some unknown thinker sat up one night in his cave and said: Wow. One thing can be another thing. Yes. Of course that’s what we are saying. Except that he didn’t say it because there was no language for him to say it in [yet]….The simple understanding that one thing can be another thing is at the root of all things of our doing. From using colored pebbles for the trading of goats to art and language and on to using symbolic marks to represent pieces of the world too small to see.”

So, how language originated is still an open question. However, once it takes root, it is easily adapted to purposes other than the social imperatives it was invented for. It is a short evolutionary step from rudimentary communication about social norms in hunter-gatherer groups to Shakespeare and Darwin.

Regardless of its origins, however, it seems clear that language is a vehicle for communicating our thinking, but not for thinking or reasoning.

–x–

So, back to LLMs then:

Based, as they are, on a representative corpus of human language, LLMs mimic how humans communicate their thinking, not how humans think. Yes, they can do useful things, even amazing things, but my guess is that these will turn out to have explanations other than intelligence and / or reasoning. For example, in this paper, Ben Prystawksi and his colleagues conclude that “we can expect Chain of Thought reasoning to help when a model is tasked with making inferences that span different topics or concepts that do not co-occur often in its training data, but can be connected through topics or concepts that do.” This is very different from human reasoning which is a) embodied, and thus uses data that is tightly coupled – i.e., relevant to the problem at hand and b) uses the power of abstraction (e.g. theoretical models).

In time, research aimed at understanding the differences between LLM and human reasoning will help clarify how LLMs do what they do. I suspect it will turn out that that LLMs do sophisticated pattern matching and linking at a scale that humans simply cannot and, therefore, give the impression of being able to think or reason.

Of course, it is possible I’ll turn out to be wrong, but while the jury is out, we should avoid conflating communication about thinking with thinking itself.

–x–x–

Postscript:

I asked Bing Image Creator to generate an image for this post. The first prompt I gave it was:

An LLM thinking

It responded with the following image:

I was flummoxed at first, but then I realised it had interpreted LLM as “Master of Laws” degree. Obviously, I hadn’t communicated my thinking clearly, which is kind of ironic given the topic of this article. Anyway, I tried again, with the following prompt:

A Large Language Model thinking

To which it responded with a much more appropriate image: