Eight to Late

A recent article in The Register notes that Microsoft has tweaked its fine print to warn users not to take its AI seriously.  The relevant update to the license terms reads, “AI services are not designed, intended, or to be used as substitutes for professional advice.”  

Aside from the fact that users ought not to believe everything a software vendor claims, Microsoft’s disingenuous marketing is also to blame. For example, their website currently claims that, “Copilot…is unique in its ability to understand … your business data, and your local context.” The truth is that the large language models (LLMs) that underpin Copilot understand nothing.

Microsoft has sold Copilot to organisations by focusing on the benefits of generative AI while massively downplaying its limitations. Although they recite the obligatory lines about having a human in the loop, they neither emphasise nor explain what this entails.

–x–

As Subbarao Kambhampati notes in this article LLMs  should be treated as approximate retrieval engines. This is worth keeping in mind when examining vendor claims.

For example, one of the common use cases that Microsoft touts in its sales spiel is that generative AI can summarise documents.

Sure, it can.

However, given that LLMs are approximate retrieval engines how much would you be willing trust an AI-generated summary?  The output of an LLM might be acceptable for a summary that will go no further than your desktop or may be within your team, but would you be comfortable with it going to your chief executive without thorough verification?

My guess is that you would want to be a (critically thinking!) human in that loop.

–x–

In a 1983 paper entitled, The Ironies of Automation, Lisanne Bainbridge noted that:

“The classic aim of automation is to replace human manual control, planning and problem solving by automatic devices and computers.  However….even highly automated systems, such as electric power networks, need human beings for supervision, adjustment, maintenance, expansion and improvement. Therefore one can draw the paradoxical conclusion that    automated   systems   still   are man-machine systems, for which both technical and human factors are important. This paper suggests that the increased interest in human factors among engineers reflects the irony that the more advanced a control system is, so the more crucial may be the contribution of the human operator.”

Bainbridge’s paper presaged many automation-related failures that could have been avoided with human oversight. Examples range from air disasters to debt recovery and software updates.

–x–

Bainbridge’s prescient warning about the importance of the human in the loop has become even more important in today’s age of AI. Indeed, inspired by Bainbridge’s work, Dr. Mica Endsley recently published a paper on the ironies of AI. In the paper, she lists the following five ironies:

AI is still not that intelligent: despite hype to the contrary, it is still far from clear that current LLMs can reason. See, for example, the papers I have discussed in this piece.

The more intelligent and adaptive the AI, the less able people are to understand the system: the point here being that the more intelligent/adaptive the system, the more complex it is – and therefore harder to understand.

The more capable the AI, the poorer people’s self-adaptive behaviours for compensating for shortcomings: As AIs become better at what they do, humans will tend to offload more and more of their thinking to machines. As a result, when things go wrong, humans will find themselves less and less able to take charge and fix issues.

The more intelligent the AI, the more obscure it is, and the less able people are to determine its limitations and biases and when to use the AI: as AIs become more capable, their shortcomings will become less obvious. There are at least a couple of reasons for this: a) AIs will become better at hiding (or glossing over) their limitations and biases, and b) the complexity of AIs will make it harder for users to understand their workings.

The more natural the AI communications, the less able people are to understand the trustworthiness of the AI: good communicators are often able to trick people into believing or trusting them. It would be exactly the same for a sweet-talking AI.

In summary: the more capable the AI, the harder it is to be a competent human in the loop, but the more critical it is to be one.

–x–

Over the last three years, I’ve been teaching a foundational business analytics course at a local university.  Recently, I’ve been getting an increasing number of questions from students anxious about the implications of generative AI for their future careers. My responses are invariably about the importance of learning how to be a thinking human in the loop.

This brings up the broader issue of how educational practices need to change in response to the increasing ubiquity of generative AI tools. Key questions include:

• How should AI be integrated into tertiary education?
• How can educators create educationally meaningful classroom activities which involve the use AI?
• How should assessments be modified to encourage the use of AI in ways that enhance learning? 

These are early days yet, but some progress has been made on addressing each of the above. For examples see:

• The following paper, based on experiences of introducing AI at a Swiss university  describes how to integrate AI into university curricula: https://link.springer.com/article/10.1186/s41239-024-00448-3
• This paper by Ethan and Lilach Mollick provides some interesting creative examples of AI use in the class: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4391243
• This document, by the Australian Tertiary Education Quality and Standards Agency, provides guidance on how AI can be integrated into university assessments: https://www.teqsa.gov.au/sites/default/files/2023-09/assessment-reform-age-artificial-intelligence-discussion-paper.pdf

However, even expert claims should not be taken at face value.  An example might help illustrate why:

In his bestselling book on AI, Ethan Mollick gives an example of two architects learning their craft after graduation. One begins his journey by creating designs using traditional methods supplemented by study of good designs and feedback from an experienced designer. The other uses an AI-driven assistant that highlights errors and inefficiencies in his designs and suggests improvements. Mollick contends that the second architect’s learning would be more effective and rapid than that of the first.

I’m not so sure.

A large part of human learning is about actively reflecting on one’s own ideas and reasoning. The key word here is “actively” – meaning that thinking is done by the learner. An AI assistant that points out flaws and inefficiencies may save the student time, but it also detracts from learning because the student is also “saved” from the need to reflect on their own thinking.

–x–

I think it is appropriate to end this piece by quoting from a recent critique of AI by the science fiction writer, Ted Chiang:

“The point of writing essays is to strengthen students’ critical-thinking skills; in the same way that lifting weights is useful no matter what sport an athlete plays, writing essays develops skills necessary for whatever job a college student will eventually get. Using ChatGPT to complete assignments is like bringing a forklift into the weight room; you will never improve your cognitive fitness that way.”

So, what role does AI play in your life: assistant or forklift?  

Are you sure??

–x–x–

Acknowledgement: This post was inspired by Sandeep Mehta’s excellent article on the human factors challenge posed by AI systems.