Eight to Late

This is the first in a series of reflections on the contrasts between human and machine thinking.  My main aim in the present piece is to highlight how humans make sense of novel situations or phenomena for which we have no existing mental models. The basic argument presented here is that we do this by making analogies to what we already know.

Mental models are simplified representations we develop about how the world works. For everyday matters, these representations work quite well. For example, when called upon to make a decision regarding stopping your car on an icy road, your mental model of the interaction between ice and rubber vs road and rubber tells you to avoid sudden braking. Creative work in physics (or any science) is largely about building mental models that offer footholds towards an understanding of the phenomenon being investigated. 

So, the question is: how do physicists come up with mental models?

Short answer: By making analogies to things they already know.

–x–

In an article published in 2001, Douglas Hofstadter made the following bold claim:

“One should not think of analogy-making as a special variety of reasoning (as in the dull and uninspiring phrase “analogical reasoning and problem-solving,” a long-standing cliché in the cognitive-science world), for that is to do analogy a terrible disservice. After all, reasoning and problem-solving have (at least I dearly hope!) been at long last recognized as lying far indeed from the core of human thought. If analogy were merely a special variety of something that in itself lies way out on the peripheries, then it would be but an itty-bitty blip in the broad blue sky of cognition. To me, however, analogy is anything but a bitty blip — rather, it’s the very blue that fills the whole sky of cognition — analogy is everything, or very nearly so, in my view.”

The key point he makes is that analogy-making comes naturally to us; we do it several times every day, most often without even being aware of it. For example, this morning when describing a strange odour to someone, I remarked, “It smelt like a mix of burnt toast and horse manure.”

Since analogy enables us to understand the unknown in terms of the known, it should be as useful in creative work as it is in everyday conversation. In the remainder of this article, I will discuss a couple of analogies that led to breakthroughs in physics: one well-known, the other less so. 

–x–

Until recently, I had assumed the origin story of the Newton’s theory of gravitation – about a falling apple – to be a myth. I was put right by this very readable history on Newton’s apple tree.  Apart from identifying a likely candidate for the tree that bore the fateful fruit, the author presents reminiscences from Newton’s friends and acquaintances about his encounter with the apple. Here’s an account by William Stukely, a close friend of Sir Isaac:

“… After dinner, the weather being warm we went into the garden and drank thea, under the shade of some apple trees, only he and myself. Amidst other discourses, he told me that he was just in the same situation, as when formally the notion of gravity came into his mind. It was occasioned by the fall of an apple, as he sat in a contemplative mood.”

 It appears that the creative leap to universal theory of gravitation came from an analogy between a falling apple and a “falling” moon – both being drawn to the centre of the earth. Another account of the Stukely story corroborates this:

“…Amidst other discourse, he told me, he was just in the same situation, as when formerly, the notion of gravitation came into his mind.  It was occasioned by the fall of an apple, as he sat in a contemplative mood.  Why should that apple always descend perpendicularly to the ground,  thought he to him self. Why should it not go sideways or upwards, but constantly to the earth’s centre ? Assuredly, the reason is, that the earth  draws it. There must be a drawing power in matter: and the sum of the drawing power in the matter of the earth must be in the earth’s center, not in any side of the earth. Therefore dos this apple fall perpendicularly,  or towards the center. If matter thus draws matter, it must be in proportion of its quantity. Therefore the apple draws the earth, as well as the earth draws the apple. That there is a power, like that we here call gravity, which extends its self thro’ the universe…”

Newton’s great of leap intuition was the realisation that what happens at the surface of the earth, insofar as the effect of matter on matter is concerned, is exactly the same as  what happens elsewhere in the universe.  He  realized that both the apple and the moon tend to fall towards the centre of the earth, but the latter doesn’t fall because it has a tangential velocity which exactly counterbalances the force of gravity. 

The main point I want to make here is well-summarised in this line from the first article I referenced above: “there can be little doubt that it was through the fall of an apple that Newton commenced his speculations upon the behaviour of gravity.”

–x–

The analogical aspect of the Newton story is easy to follow. However, many analogies associated with momentous advances in physics are not so straightforward because the things physicists deal with are hard to visualise. As Richard Feynman once said when talking about quantum mechanics, “we know we have the theory right, but we haven’t got the pictures [visual mental models] that go with the theory. Is that because we haven’t [found] the right pictures or is it because there aren’t any right pictures?“

He then asks a very important question: supposing there aren’t any right pictures [and the consensus is there aren’t], then is it possible to develop mental models of quantum phenomena?

Yes there is!

However, creating these models requires us to give up the requirement of visualisability: atoms and electrons cannot be pictured as balls or clouds or waves or anything else; they can only be understood through the equations of quantum mechanics.  But writing down and solving equations is one thing, understanding their implications is quite another. Many (most?)  physicists focus on the former because it’s easier to “shut up and calculate” than develop a feel for what is actually happening.

So, how does one develop an intuition for what is going on when it is not possible to visualise it?

This question brings me to my second analogy. Although it is considerably more complex than Newton’s apple, I hope to give you a sense for the thinking behind it because it is a tour de force of scientific analogy-making.

–x–

The year 1905 has special significance in physics lore. It was the year in which Einstein published four major scientific papers  that changed the course of physics.

The third and fourth papers in the series are well known because they relate to the special theory of relativity and mass-energy equivalence. The second builds a theoretical model of Brownian motion  – the random jiggling of fine powder scattered on a liquid surface. The first paper of the series provides an intuitive explanation of how light, in certain situations, can be considered to be made up of particles which we now call photons. This paper is not so well-known even though it is the work for which Einstein received the 1921 Nobel Prize for physics. This is the one I’ll focus on as it presents an example of analogy-making par excellence.  

To explain the analogy, I’ll first need to set some context around the state of physics at the turn of the last century. 

(Aside: Although not essential for what follows, if you have some time I  highly recommend you read Feynman’s lecture on the atomic hypothesis, delivered to first year physics students at Caltech in 1962)

–x–

Today, most people do not question the existence of atoms. The situation in the mid1800s was very different. Although there was considerable indirect evidence for the existence of atoms, many influential physicists, such as Ernst Mach, were sceptical. Against this backdrop,  James Clerk Maxwell derived formulas relating microscopic quantities such as pressure and temperature of a gas in a container to microscopic variables such as the number of particles and their speed (and energy). In particular, he derived a formula predicting the probability distribution of particle velocities – that is the proportion of particles that have a given velocity. The key assumption Maxwell made in his derivation is that the gas consists of small, inert spherical particles (atoms!) that keep bouncing off each other elastically in random ways – a so-called ideal gas.

The shape of the probability distribution, commonly called the Maxwell-Boltzmann distribution, is shown below for a couple of temperatures. As you might expect, the average speed increases with temperature.

I should point out that this was one of the earliest attempts to derive a quantitative link between a macroscopic quantity which we can sense directly and microscopic motions which are inaccessible to our senses.  Pause and think about this for a minute. I hope you agree that it is amazing, not the least because it was derived at a time when the atomic hypothesis was not widely accepted as fact. 

It turns out that the Maxwell-Boltzmann distribution played a key role in Einstein’s argument that light could – in certain circumstances – be modelled as a collection of particles. But to before we get to that, we need to discuss some more physics.

–x–

In the late 19th century it was widely accepted that there are two distinct ways to analyse physical phenomena: as particles (using Newton’s Laws) or as  waves (using Maxwell’s Equations for electromagnetic waves, for example).

Particles are localised in space – that is, they are characterised by a position and velocity. Consequently, the energy associated with a particle is localised in space. In contrast, waves are spread out in space and are characterised by a wavelength and frequency as shown in the figure below. Note that the two are inversely related – frequency increases as wavelength decreases and vice versa. The point to note is that, in contrast to particles, the energy associated with a wave is spread out in space.

In the early 1860s, Maxwell established that light is an electromagnetic wave.  However visible light represents a very small part of the electromagnetic spectrum which ranges from highly energetic x-rays to low energy radio waves (see the figure below)

The energy of a wave is directly proportional to its frequency – so, high energy X-rays have higher frequencies (and shorter wavelengths) than visible light.

To summarise then, at the turn of the 20^th  century, the consensus was that light is a wave. This was soon to be challenged from an unexpected direction.

–x–

When an object is heated to a particular temperature, it radiates energy across the entire electromagnetic spectrum. Physically we expect that as the temperature increases, the energy radiated will increase – this is analogous to our earlier discussion of the relationship between the velocity/energy of particles in a gas and the temperature. A practical consequence of this relationship is that blacksmiths can judge the temperature of a workpiece by its colour – red being cooler than white (see chart below).

Keep in mind, though, that visible light is a very small portion of  the electromagnetic spectrum: the radiation that a heated workpiece emits extends well beyond the violet and red ends of the visible part of the spectrum.  

In the 1880s physicists experimentally established that for a fixed temperature, the distribution of energy emitted by a heated body as a function of frequency is unique. That is – the frequency spread of energy absorbed and emitted by a heated body depends on the temperature alone. The composition of the object does not matter as long as it emits all the energy that it absorbs (a so-called blackbody). The figure below shows the distribution of energy radiated by such an idealised object.

Does the shape of this distribution remind you of something we have seen earlier?

Einstein noticed that the blackbody radiation curve strongly resembles the Maxwell-Boltzmann distribution. It is reasonable to assume that others before him would have noticed this too. However, he took the analogy seriously and used it develop a plausibility argument that light could be considered to consist of particles. Although the argument is a little technical, I’ll sketch it out in brief below. Before I do so, I will need to introduce one last physical concept.

–x–

Left to themselves, things tend to move from a state of order to disorder. This applies just as much to atoms as it does to our everyday lives – my workspace tends to move from a state of tidiness to untidiness unless I intervene. In the late 1800s physicists invented a quantitative measure of disorder called entropy. The observation that things tend to become disordered (or messy) if left alone is enshrined in the second law of thermodynamics which states that the entropy of the universe is increasing.

To get a sense for the second law, I urge you to check out this simulation which shows how two gases (red and green) initially separated by a partition tend to mix spontaneously once the partition is removed. The two snapshots below show the initial (unmixed) and equilibrium (mixed) states. 

Snapshot 1: Time=0, ordered state, low entropy

Snapshot 2: Time = 452 seconds, disordered (mixed) state, high entropy

The simulation gives an intuitive feel for why a disordered system will never go back to an ordered state spontaneously. It is the same reason that sugar, once mixed into your coffee will never spontaneously turn into sugar crystals again.

Why does the universe behave this way?

The short answer is that there are overwhelmingly more disordered states in the universe than ordered ones. Hence, if left to themselves, things will end up being more disordered (or messy) than they were initially.  This is as true of my desk as it is of the mixing of two gases.

Incidentally, the logic of entropy applies to our lives in other ways too. For example, it explains why we have far fewer successes than failures in our lives. This is because success typically requires many independent events to line up in favourable ways, and such a confluence is highly unlikely. See my article on the improbability of success for a deeper discussion of this point.

I could go on about entropy as it is fertile topic, but I ‘ll leave that for another time as I need to  get back to my story about Einstein’s photons and finish up this piece.

–x–

Inspired by the similarity between the energy distribution curves of blackbody radiation  and an ideal gas, Einstein made the bold assumption that the light bouncing around inside a heated body consisted of an ideal gas of photons. As he noted in his paper, “According to the assumption to be contemplated here, when a light ray is spreading from a point, the energy is not distributed continuously over ever-increasing spaces, but consists of a finite number of energy quanta that are localized in points in space, move without dividing, and can be absorbed or generated only as a whole.“

In essence he assumed that the energy associated with electromagnetic radiation is absorbed or emitted in discrete packets akin to particles. This assumption enabled him to make an analogy between electromagnetic radiation and an ideal gas. Where did the analogy itself come from. The physicist John Rigden notes the following in this article, “what follows comes from Einstein’s deep well of intuition; specifically, his quantum postulate emerges from an analogy between radiation and an ideal gas.” In other words, we have no idea!

Anyway, with the analogy assumed, Einstein compared the change in entropy when an ideal gas consisting of particles is compressed from a volume to volume at a constant temperature  (or energy ) to the change in entropy when a “gas of electromagnetic radiation” of average frequency undergoes a similar compression.  

Entropy is typically denoted by the letter , and a change in any physical quantity is conventionally denoted by the Greek letter , so the change in entropy is denoted by .  The formulas for the two changes in entropy mentioned in the previous paragraph are:

The first of the two formulas was calculated from physics that was well-known at the time (the same physics that led to the Maxwell-Boltzmann distribution). The second was based on the analogy that Einstein made between an ideal gas and electromagnetic radiation.

Comparing the exponents in the two formulas, we get:

Which basically tells us that the light contained in the heated container consists of discrete particles that have an energy proportional to the frequency . The formula relates a wave characteristic (frequency) to the energy of a photon. In Einstein’s original derivation, the quantity denoted by had a more complicated expression, but he recognised that it was identical to a universal constant identified by Max Planck some years earlier. It is important to note that the connection to Planck’s prior work provided weak evidence for the validity of Einstein’s analogy, but not a rigorous proof.

–x–

As Rigden notes, “Einstein’s “revolutionary” paper has the strange word “heuristic” in the title. This word means that the “point of view” developed – that is, the light particle – is not in itself justified except as it guides thinking in productive ways. Therefore, at the end of his paper, Einstein demonstrated the efficacy of light quanta by applying them to three different phenomena. One of these was the photoelectric effect [which to this day remains] the phenomenon that demonstrated the efficacy of Einstein’s light quantum most compellingly.” (Note: I have not described the other two phenomena here – read Rigden’s article for more about them)

Einstein’s could not justify his analogy theoretically, which is why he resorted to justification by example. Even so many prominent physicists remained sceptical. As Rigden notes, “Einstein’s big idea was universally rejected by contemporary physicists; in fact, Einstein’s light quantum was derisively rejected. When Max Planck, in 1913, nominated Einstein for membership of the Prussian Academy of Science in Berlin, he apologized for Einstein by saying, “That sometimes, as for instance in his hypothesis on light quanta, he may have gone overboard in his speculations should not be held against him.” Moreover, Robert Millikan, whose 1916 experimental data points almost literally fell on top of the straight line predicted for the photoelectric effect by Einstein’s quantum paper, could not accept a corpuscular view of light. He characterized Einstein’s paper as a “bold, not to say reckless, hypothesis of an electro-magnetic light corpuscle of energy hν, which…flies in the face of thoroughly established facts of interference…In his 1922 Nobel address, Niels Bohr rejected Einstein’s light particle. “The hypothesis of light-quanta”, he said, “is not able to throw light on the nature of radiation.” It was not until Arthur Compton’s 1923 X-ray scattering experiment, which showed light bouncing off electrons like colliding billiard balls, that physicists finally accepted Einstein’s idea.”

It took almost twenty years for the implications of Einstein’s bold analogy to be accepted by physicists!

–x–

Perhaps you’re wondering why I’ve taken the time to go through these two analogies in some detail. My reason is simple: I wanted to illustrate the kind of lateral and innovative thinking that humans are uniquely capable of.

I will refrain from making any remarks about whether LLM-based AIs are capable of such thinking. My suspicion – and I’m in good company here – is that the kind of thinking which leads to new insights involves a cognitive realm that has little do with formal reasoning. To put it plainly, although we may describe our ideas using language, the ideas themselves – at least the ones that are truly novel – come from another kind of logic. In the next article in this series, I will speculate on what that logic might be. Then, in a following piece, I will discuss its implications for using AI in ways that augment our capabilities rather than diminish them.

–x—x–

Acknowledgements: the above discussion of Einstein’s analogy are based on this lecture by Douglas Hofstadter and this article by John Rigden. I’d also like to acknowledge Michael Fowler from the University of Virginia for his diffusion simulation (https://galileoandeinstein.phys.virginia.edu/more_stuff/Applets/Diffusion/diffusion.html) which I have used for the entropy explanation in the article.