Archive for the ‘Organizations’ Category
Meditations on change
Despite our carefully laid plans, the lives of our projects and the projects of our lives often hinge on events we have little control over. Robert Chia stresses this point in his wonderful book, Strategy without Design:
“Ambitious strategic plans, the ‘big picture’ approach that seeks a lasting solution or competitive advantage through large-scale transformations, often end up undermining their own potential effectiveness because they overlook the fine details of everyday happenings at ‘ground zero’ level.”
At one level we know this, yet we act out our personal and work lives as if it were not so.
–x–
In business (and life!) we are exhorted to think before doing. My boss tells me I need to think about my team’s workplan for next year; my wife tells me I need to think about the future. Thinking is at the centre of our strategies, blueprints, plans etc. – the things that supposedly propel our lives into a imagined, better future.
The exhortation to make detailed plans of what we are going to do is a call to think before acting. As Descartes famously wrote, cogito ergo sum: thinking establishes our being.
But is that really so?
–x–
In his posthumously published book, Angels Fear, Gregory Bateson noted that:
“There is a discrepancy of logical type between “think” and “be”. Descartes is trying to jump from the frying pan of thought, ideas, images, opinions, arguments etc., into the fire of existence and action. But that jump itself is unmapped. Between two such contrasting universes there can be no “ergo” – no totally self-evident link. There is no looking before leaping from “cogito” to “sum”
The gap between our plans and reality is analogous to the gap between thought and action. There is ample advice on how to think, but very little on how to act in difficult situations. This gap is, I think, at the heart of the problem that Chia articulates in his writings on emergent approaches to strategy.
Understanding this at the intellectual level is one thing. Grasping it experientially is quite another. For, as they say, there is no better way to learn than through experience.
–x–
A few weeks ago, I attended a 10-day Vipassana course at the Dhamma Bhumi centre in Blackheath. Late April is a beautiful time in the Blue Mountains, with glorious sunshine and autumn colours just starting to turn. A perfect setting to reflect and meditate.
Vipassana, which means insight in Pali, is a meditation technique that revolves around observing the often-transient sensations one encounters across one’s body without reacting to them (for example, that itch at the back of your head right now).
The objective is to develop a sense of equanimity in the face of an ever-changing world. Strangely, it seems to work: the simple act of observing sensations without reacting to them, if done in the right away and for long enough, has a subtle influence on how one perceives and responds to events in the world.
Now, I would not go so far as to say the experience was life changing, but it has certainly made me more aware of the many little incidents and encounters of everyday life and, more importantly, better mediate my reactions to them. That said, it is still very much work in progress.
Buddhist metaphysics suggests that the practice of Vipassana helps one (gradually) understand that it is futile to force change or attempt to bend the world to one’s will. Such acts invariably end in disappointment and frustration; any planned change aimed at achieving a well-defined, stable end-state will miss the mark because the world is Heraclitean: ever-changing and impermanent.
–x–
Heraclitus famously asserted that everything is in motion all the time or all things in the world are constantly changing. Hidden in that statement is a paradox: if everything is changing all the time, then it is nigh impossible to pinpoint what exactly is changing. Why? Because nothing in the universe is stable – see Bateson’s article, Orders of Change, for more on this.
Lewis Carroll describes this paradox in a conversation between Alice and the Caterpillar in Chapter 5 of Alice in Wonderland:
“Who are you?” said the Caterpillar.
This was not an encouraging opening for a conversation. Alice replied, rather shyly, “I—I hardly know, sir, just at present—at least I know who I was when I got up this morning, but I think I must have been changed several times since then.”
“What do you mean by that?” said the Caterpillar sternly. “Explain yourself!”
“I can’t explain myself, I’m afraid, sir,” said Alice, “because I’m not myself, you see.”
If the world is ever-changing then so is one’s own identity, not to mention the identities of everything else around us. This brings up a raft of interesting philosophical questions that I neither have the time nor expertise to confront.
However, I can write about my lived experiences.
–x–
Anapana is a breathing technique taught as a prelude to learning Vipassana. The technique involves focusing on the sensations caused by breathing – for example, the coolness felt above the upper lip on an incoming breath and the corresponding warmth on exhalation.
After a day or two of intense practice I became reasonable at it. So much so that at times deep in a Anapana session, it felt like the observer and the observed were distinct entities: the “I” who was watching me breathe was no longer the me who was being watched.
This was disconcerting. I asked the teacher what was going on.
His terse reply: “don’t worry about it, just do what you are doing.”
At the time his response felt deeply unsatisfying. It was only later I understood: as Wittgenstein famously noted in the final line of the Tractatus: whereof one cannot speak, thereof one must be silent.
Some things are best experienced and learnt through experience than spoken (or written about) or taught explicitly.
–x–
An organisation is a complex system in which much of the complexity arises from the multiple pathways of interaction between the people who comprise it. The objective of an organisational strategy (of any kind) is to get all those people working purposefully towards a well-defined set of goals. Such a strategy is invariably accompanied by a roadmap that describes what needs to be done to achieve those objectives. Very often the actions are tightly scripted and controlled by those in charge.
But those who wish to control change are no different from those who believe in a chimerical stability: they are today’s Parmenideans. Many years ago, the cybernetician and organisational theorist, Stafford Beer, wrote:
“The most famous of the believers in change was Heraclitus, working in Ephesus, best known for teaching that everything is in constant flux. It was he who wrote that you cannot step into the same river twice. But just down the road the philosophers of Elea were contending that change is impossible. Parmenides, for example, taught that all change is inconceivable – its appearance an illusion. All this in 500 BC. The argument rages on. Today’s management scone is typified in my experience by people fervidly preaching change to people who fervently embrace change – on condition that nothing alters.”
A little later in the same piece, he notes, “Society is Heraclitian; but Parmenides is in charge“
One could say the same for organisations.
But then, the question is: if tightly scripted and controlled approaches to strategy don’t work, what does?
–x–
The technique of Vipassana is simple, straightforward and can be summarised in a few lines:
The basic procedure is to scan (turn one’s attention to) all parts of the body in sequence, objectively observing the sensations one feels on each part. A sensation is anything that comes to your attention. It could be temperature, humidity, itchiness, pressure, strain, pain etc. Although one is immediately aware of relatively intense sensations itches and pains, one is typically not attuned to the subtle, ephemeral sensations experienced across one’s entire body all the time. The technique forces one to focus on the latter in an equanimous manner – i.e., without reacting to them. Instead, one uses these sensations to guide the pace at which one does the scan. See this reddit post for more.
The simplicity is deceptive.
It was sometime in the latter half of the course – may be day 6 or 7 – I realised that a key aspect of the technique is its indirectness. The sense of balance and equanimity I was practising during meditation was, almost imperceptibly, spilling over into other aspects of my life. I found myself being more relaxed about small things I would normally get upset about. Not always, of course, but more often than I used to.
Habits of a lifetime take a while to change, and the trick to changing them seems to centre around taking an oblique or indirect route.
–x–
In a paper published in 2014, Robert Chia noted that:
“Managing change then is more about small, timely and quiet insertions made to release the immanent forces of change always already present in every organizational situation. Change then appears unexceptionally as a naturally occurring phenomenon; it does not attract undue attention and does not generate unnecessary anxieties. Obliqueness of engagement is key to managing sustainable change in a world that is itself ever-changing.”
From personal experience – and more about that in a moment – I can attest that such an indirect approach to change, which leverages latent possibilities within the organisation, really does work. Akin to natural evolution, it is about repurposing or exapting what is at hand to move in a direction that takes one to a better place.
As Chia wrote,
“The Emergent perspective emphasizes a ‘bottoms up’ approach to change and views outcomes as the result of the cumulative and oftentimes ‘piecemeal’ adaptive actions taken in situ by organizational members in learning to cope with the exigencies of organizational situations.”
So, back to the question I hinted at earlier: how does one act in such a manner?
–x–
Strangely, few if any proponents of the Emergent perspective have offered any advice on how to develop and implement strategy in an indirect manner. As Bateson once noted,
“What is lacking is a theory of action within large complex systems, where the active agent is himself a part and a product of the system.”
A couple of sentences later, Bateson offers a route to a possible solution:
“It seems also that great teachers and therapists avoid all direct attempts to influence the action of others and, instead, try to provide the settings or contexts in which some (usually imperfectly specified) change may occur.”
An indirect approach to change must focus on creating a context in which change can happen of its own accord.
–x–
Participants in a Vipassana course are required to abide by a code of discipline for the entire period of the course At first I thought some of the restrictions were over the top – for example, complete silence, no reading or writing. Now I know that is not so: the rules are necessary for creating a context in which an individual can initiate serious changes in his or her outlook and way of life.
By day three I no longer missed having my phone, journals or reading materials at hand. When the weather permitted, I spent the time between meditation sessions walking on the tracks within the centre compound. On rainy days I would just sit and reflect on the things going on in my head.
Practising the technique seems to evoke all kinds of thoughts, memories and emotions. As we were informed in one of the evening discourses these are all expected natural reactions caused by the process of learning Vipassana in the right context. The serene physical environment and the code of discipline provided that context.
–x–
To be clear, creating a context for good things to happen does not guarantee a specific outcomes, let alone positive ones. The Vipassana experience is highly personal: no two people doing it will have the same experience. Yet, the context is the key because creates conditions in which beneficial outcomes are more likely to occur than harmful ones. This is reflected in the overwhelming number of people who speak positively about the experience.
As I have discussed in an earlier piece , there are many actions from which one might reasonably expect positive changes without knowing upfront, in detail, what exactly those changes are. This is exactly what Bateson was getting at when he wrote about good teachers (or change agents) who are somehow able to create “settings or contexts in which some (usually imperfectly specified) change may occur.”
–x–
In the late 1990s, a group from MIT Media worked on a multi- year project to introduce students in rural Thailand to new learning approaches based on computing technologies. In the early stages of the project, it became evident that standard pedagogical approaches would not work for these students, not because of a lack of ability or intelligence but due a lack of relevance. To address this, the group created a context that would motivate the students to learn. They did this by demonstrating how the technology could help address problems the villagers faced – such as building a dam to store water.
The change in approach made all the difference: once students could relate the new technology to issues that they could relate to, learning came for free. They called this approach Emergent Design.
When I came across the MIT work about a dozen years ago, I realised it could be applied to problems of organisational change (indeed, David Cavallo – one of the MIT team – mentions that specifically in his PhD thesis). Since then, I have applied variations of Emergent Design in distinct organisational settings, ranging from multinationals to not for profits and government agencies. Although the broad approach I took was inspired by the MIT work, it gradually took on a life and identity of its own.
I have described my take on Emergent Design in brief in this article and in detail in this book. However, if I were asked to summarise the key to Emergent Design, I would echo Bateson in saying that it is largely about creating a context in which good stuff can happen. Doing this successfully requires the change agent to develop a deep understanding of the organisation and the way it works, and then initiating small changes that enable it to evolve in a positive direction.
Evolution is a slow process, but far more likely to succeed than revolution (see this article for an elaboration of this point).
–x–
In a lecture on Intelligence, Experience and Evolution, delivered at the Naropa Institute in 1975, Bateson started with the remark, “what goes on inside is what goes on outside.” He was referring to the deep analogy between human learning and natural evolution (see Chapter 6 of his book, Mind and Nature, for an elaboration of the analogy). In essence, learning and evolution are processes of change which are context dependent. Both processes are essentially based on gradual improvement through trial and error, and context plays a key role by constraining successive iterations of trial and error to move in productive directions.
Bateson’s analogy between what goes on in our heads and on the outside assumes an even greater significance for me when I view my experiences over ten years doing organisational change via Emergent Design through the lens of the ten days I spent learning Vipassana in Blackheath. The key lesson it brought home to me is that true, lasting change – whether at the societal, organisational or personal level – is best achieved through a gradual, evolutionary process which mirrors what goes on both on the inside and the outside.
–x–x–
On the anticipation of unintended consequences
A couple of weeks ago I bought an anthology of short stories entitled, An Exploration of Unintended Consequences, written by a school friend, Ajit Chaudhuri. I started reading and couldn’t stop until I got to the last page a couple of hours later. I’ll get to the book towards the end of this piece, but first, a rather long ramble about some thoughts which have been going around my head since I read it.
–x–
Many (most?) of the projects we undertake, or even the actions we perform, both at work and in our personal lives have unexpected side-effects or results that overshadow their originally intended outcomes. To be clear, unexpected does not necessarily mean adverse – consider, for example, Adam Smith’s invisible hand. However, it is also true – at least at the level of collectives (organisations or states) – that negative outcomes far outnumber positive ones. The reason this happens can be understood from a simple argument based on the notion of entropy – i.e., the fact that disorder is far more likely than order. Anyway, as interesting as that may be, it is tangential to the question I want to address in this post, which is:
Is it possible to plan and act in a way which anticipates, or even encourages, positive unintended consequences?
Let’s deal with the “unintended” bit first. And you may ask: does the question make sense? Surely, if an outcome is unintended, then it is necessarily unanticipated (let alone positive).
But is that really so? In this paper, Frank DeZwart, suggests it isn’t. In particular, he notes that, ” if unintended effects are anticipated, they are a different phenomenon as they follow from purposive choice and not, like unanticipated effects, from ignorance, error, or ideological blindness.”
As he puts it, “unanticipated consequences can only be unintended, but unintended consequences can be either anticipated or unanticipated.”
So the question posed earlier makes sense, and the key to answering it lies in understanding the difference between purposive and purposeful choice (or action).
–x–
In a classic paper that heralded the birth of cybernetics, Rosenblueth, Wiener and Bigelow noted that, “the term purposeful is meant to denote that the act or behavior may be interpreted as directed to the attainment of a goal-i.e., to a final condition in which the behaving object reaches a definite correlation in time or in space with respect to another object or event.”
Aside 1: The reader will notice that the definition has a decidedly scientific / engineering flavour. So, it is not surprising that philosophers jumped into the fray, and arguments around finer points of the definition ensued (see this sequence of papers, for example). Although interesting, we’ll ignore the debate as it will take us down a rabbit hole from which there is no return.
Aside 2: Interestingly, the Roseblueth-Wiener-Bigelow paper along with this paper by Warren McCulloch and Walter Pitts laid the foundation for cybernetics. A little known fact is that the McCulloch-Pitts paper articulated the basic ideas behind today’s neural networks and Nobel Prize glory, but that’s another story.
Back to our quest: the Rosenblueth-Wiener definition of purposefulness has two assumptions embedded in it:
a) that the goal is well-defined (else, how will an actor know it has been achieved?), and
b) the actor is aware of the goal (else, how will an actor know what to aim for?)
We’ll come back to these in a bit, but let’s continue with the purposeful / purposive distinction first.
As I noted earlier, the cybernetic distinction between purposefulness and purposiveness led to much debate and discussion. Much of the difference of opinion arises from the ways in which diverse disciplines interpret the term. To avoid stumbling into that rabbit hole, I’ll stick to definitions of purposefulness / purposiveness from systems and management domains.
–x–
The first set of definitions is from a 1971 paper by Russell Ackoff in which he attempts to set out clear definitions of systems thinking concepts for management theorists and professionals.
Here are his definitions for purposive and purposeful systems:
“A purposive system is a multi-goal seeking system the different goals of which have a common property. Production of that property is the system’s purpose. These types of systems can pursue different goals, but they do not select the goal to be pursued. The goal is determined by the initiating event. But such a system does choose the means by which to pursue its goals.”
and
“A purposeful system is one which can produce the same outcome in different ways…[and] can change its goals under constant conditions – it selects ends as well as means and thus displays will. Human beings are the most familiar examples of such systems.”
Ackoff’s purpose(!) in making the purposive/purposeful distinction was to clarify the difference between apparent purpose displayed by machines (computers) which he calls purposiveness, and “true” or willed (or human) purpose which he calls purposefulness. Although this seems like a clear cut distinction, it falls apart on closer inspection. The example Ackoff gives for a purposive system is that of a computer which is programmed to play multiple games – say noughts-and-crosses and checkers. The goal differs depending on which game it plays, but the common property is winning. However, this feels like an artificial distinction: surely winning is a goal, albeit a higher-order one.
–x–
The second set of definitions, due to Peter Checkland, is taken from this module from an Open University course on managing complexity:
“Two forms of behaviour in relation to purpose have also been distinguished. One is purposeful behaviour, which [can be described] as behaviour that is willed – there is thus some sense of voluntary action. The other is purposive behaviour – behaviour to which an observer can attribute purpose. Thus, in the example of the government minister, if I described his purpose as meeting some political imperative, I would be attributing purpose to him and describing purposive behaviour. I might possibly say his intention was to deflect the issue for political reasons. Of course, if I were to talk with him I might find out this was not the case at all. He might have been acting in a purposeful manner which was not evident to me.”
This distinction is strange because the definitions of the two terms are framed from two different perspectives – that of an actor and that of an observer. Surely, when one makes a distinction, one should do so from a single perspective.
…and yet, there is something in this perspective shift which I’ll come back to in a bit.
–x–
The third set of definitions is from Robert Chia and Robin Holt’s classic, Strategy Without Design: The Silent Efficacy of Indirect Action:
“Purposive action is action taken to alleviate ourselves from a negative situation we find ourselves in. In everyday engagements, we might act to distance ourselves from an undesirable situation we face, but this does not imply having a pre-established end goal in mind. It is a moving away from rather than a moving towards that constitutes purposive actions. Purposeful actions, on the other hand, presuppose having a desired and clearly articulated end goal that we aspire towards. It is a product of deliberate intention“
Finally, here is a distinction we can work with:
- Purposive actions are aimed at alleviating negative situations (note, this can be framed in a better way, and I’ll get to that shortly)
- Purposeful actions are those aimed at achieving a clearly defined goal.
The interesting thing is that the above definition of purposive action is consistent with the two observations I made earlier regarding the original Rosenbluth-Wiener-Bigelow definition of purposeful systems
a) purposive actions have no well-defined end-state (alleviating a negative situation says nothing about what the end-state will look like). That said, someone observing the situation could attribute purpose to the actor because the behaviour appears to be purposeful (see Checkland’s definition above).
b) as the end-state is undefined, the purposive actor cannot know it. However, this need not stop the actor from envisioning what it ought to look like (and indeed, most purposive actors will).
In a later paper Chia, wrote, , “…[complex transformations require] an implicit awareness that the potentiality inherent in a situation can be exploited to one’s advantage without adverse costs in terms of resources. Instead of setting out a goal for our actions, we could try to discern the underlying factors whose inner configuration is favourable to the task at hand and to then allow ourselves to be carried along by the momentum and propensity of things.”
Inspired by this, I think it is appropriate to reframe the Chia-Holt definition more positively, by rephrasing it as follows:
“Purposive action is action which exploits the inherent potential in a situation so as to increase the likelihood of positive outcomes for those who have a stake in the situation”
The above statement includes the Chia-Holt definition as such an action could be a moving away from a negative situation. However, it could also be an action that comes from recognising an opportunity that would otherwise remain unexploited.
–x–
And now, I can finally answer the question I raised at the start regarding anticipated unintended consequences. In brief:
A purposive action, as I have defined above, is one that invariably leads to anticipated unintended consequences.
Moreover, its consequences are often (usually?) positive, even though the specific outcomes are generally impossible to articulate at the start.
Purposive action is at the heart of emergent design, which is based on doing things that increase the probability of organisational success, but in an unobtrusive manner which avoids drawing attention. Examples of such low-key actions based on recognising the inherent potential of situations are available in the Chia-Holt book referenced above and in the book I wrote with Alex Scriven.
I should also point out that since purposive action involves recognising the potential of an unfolding situation, there is necessarily an improvisational aspect to it. Moreover, since this potential is typically latent and not obvious to all stakeholders, the action should be taken in a way that does not change the dynamics of the situation. This is why oblique or indirect actions tend to work better than highly visible, head-on ones. Developing the ability to act in such a manner is more about cultivating a disposition that tolerates ambiguity than learning to follow prescribed rules, models or practices.
–x–
So much for purposive action at the level of collectives. Does it, can it, play a role in our individual lives?
The short answer is: yes it can. A true story might help clarify:
“I can’t handle failure,” she said. “I’ve always been at the top of my class.”
She was being unduly hard on herself. With little programming experience or background in math, machine learning was always going to be hard going. “Put that aside for now,” I replied. “Just focus on understanding and working your way through it, one step at a time. In four weeks, you’ll see the difference.”
“OK,” she said, “I’ll try.”
She did not sound convinced but to her credit, that’s exactly what she did. Two months later she completed the course with a distinction.
“You did it!” I said when I met her a few weeks after the grades were announced.
“I did,” she grinned. “Do you want to know what made the difference?”
Yes, I nodded.
“Thanks to your advice, I stopped treating it like a game I had to win,” she said, “and that took the pressure right off. I then started to enjoy learning.”
–x–
And this, finally, brings me back to the collection of short stories written by my friend Ajit. The stories are about purposive actions taken by individuals and their unintended consequences. Consistent with my discussion above, the specific outcomes in the stories could not have been foreseen by the protagonists (all women, by the way), but one can well imagine them thinking that their actions would eventually lead to a better place.
That aside, the book is worth picking up because the author is a brilliant raconteur: his stories are not only entertaining, they also give readers interesting insights into everyday life in rural and urban India. The author’s note at the end gives some background information and further reading for those interested in the contexts and settings of the stories.
I found Ajit’s use of inset stories – tales within tales – brilliant. The anthropologist, Mary Catherine Bateson, once wrote, “an inset story is a standard hypnotic device, a trance induction device … at the most obvious level, if we are told that Scheherazade told a tale of fantasy, we are tempted to believe that she, at least, is real.” Ajit uses this device to great effect.
Finally, to support my claim that the stories are hugely entertaining, here are a couple of direct quotes from the book:
The line “Is there anyone here at this table who, deep down, does not think that her husband is a moron?” had me laughing out loud. My dear wife asked me what’s up. I told her; she had a good laugh too, and from the tone of her laughter, it was clear she agreed.
Another one: “…some days I’m the pigeon and some days I’m the statue. It’s just that on the days that I’m the pigeon, I try to remember what it is like to be the statue. And on the days that I’m the statue, I try not to think.” Great advice, which I’ve passed on to my two boys.
–x–
I called Ajit the other day and spoke to him for the first time in over 40 years; another unintended consequence of reading his book.
–x—x–
Selling AI ethically – a customer perspective
Artificial intelligence (AI) applications that can communicate in human language seem to capture our attention whilst simultaneously blunting our critical capabilities. Examples of this abound, ranging from claims of AI sentience to apps that are “always here to listen and talk.” Indeed, a key reason for the huge reach of Large Language Models (LLMs) is that humans can interact with them effortlessly. Quite apart from the contested claims that they can reason, the linguistic capabilities of these tools are truly amazing.
Vendors have been quick to exploit our avidity for AI. Through relentless marketing, backed up by over-the-top hype, they have been able to make inroads into organisations. Their sales pitches tend to focus almost entirely on the benefits of these technologies, with little or no consideration of the downsides. To put it bluntly, this is unethical. Doubly so because customers are so dazzled by the capabilities of the technology that they rarely ask questions that they should.
AI ethics frameworks (such as this one) overlook this point almost entirely. Most of them focus on things such as fairness, privacy, reliability, transparency etc. There is no guidance or advice to vendors on selling AI ethically, by which I mean a) avoiding overblown claims, b) being clear about limitations of their products and c) showing customers how they can engage with AI tools meaningfully – i.e., in ways that augment human capabilities rather than replacing them.
In this article, I offer some suggestions on how vendors can help their customers develop a balanced perspective on what AI can do for them. To set the scene, I will begin by recounting the public demo of an AI product in the 1950s which was accompanied by much media noise and public expectations.
Some things, it seems, do not change.
–x–
The modern history of Natural Language Processing (NLP) – the subfield of computer science that deals with enabling computers to “understand” and communicate in human language – can be traced back to the Georgetown-IBM research experiment that was publicly demonstrated in 1954. The demonstration is trivial by today’s standards. However, as noted by John Hutchins’’ in this paper, “…Although a small-scale experiment of just 250 words and six ‘grammar’ rules it raised expectations of automatic systems capable of high quality translation in the near future…” Here’s how Hutchins describes the hype that followed the public demo:
“On the 8th January 1954, the front page of the New York Times carried a report of a demonstration the previous day at the headquarters of International Business Machines (IBM) in New York under the headline “Russian is turned into English by a fast electronic translator”: A public demonstration of what is believed to be the first successful use of a machine to translate meaningful texts from one language to another took place here yesterday afternoon. This may be the cumulation of centuries of search by scholars for “a mechanical translator.” Similar reports appeared the same day in many other American newspapers (New York Herald Tribune, Christian Science Monitor, Washington Herald Tribune, Los Angeles Times) and in the following months in popular magazines (Newsweek, Time, Science, Science News Letter, Discovery, Chemical Week, Chemical Engineering News, Electrical Engineering, Mechanical World, Computers and Automation, etc.) It was probably the most widespread and influential publicity that MT (Machine Translation – or NLP by another name) has ever received.”
It has taken about 60 years, but here we are: present day LLMs go well beyond the grail of machine translation. Among other “corporately-useful” things, LLM-based AI products such as Microsoft Copilot can draft documents, create presentations, and even analyse data. As these technologies requires no training whatsoever, it is unsurprising that they have captured corporate imagination like never before.
Organisations are avid for AI and vendors are keen to cash in.
Unfortunately, there is a huge information asymmetry around AI that favours vendors: organisations are typically not fully aware of the potential downsides of the technology and vendors tend to exploit this lack of knowledge. In a previous article, I discussed how non-specialists can develop a more balanced perspective by turning to the research literature. However, this requires some effort and unfairly puts the onus entirely on the buyer.
Surely, vendors have a responsibility too.
–x–
I recently sat through a vendor demo of an LLM-based “enterprise” product. As the presentation unfolded, I made some notes on what the vendor could have said or done to help my colleagues and I make a more informed decision on the technology. I summarise them below in the hope that a vendor or two may consider incorporating them in their sales spiel. OK, here we go:
Draw attention to how LLMs do what they do: it is important that users understand how LLMs do what they do. Vendors should demystify LLM capabilities by giving users an overview of how they do their magic. If users understand how these technologies work, they are less likely to treat their outputs as error-free or oracular truths. Indeed, a recent paper claims that LLM hallucinations (aka erroneous outputs) are inevitable – see this article for a simple overview of the paper.
Demo examples of LLM failures: The research literature has several examples of the failure of LLMs in reasoning tasks – see this article for a summary of some. Demonstrating these failures is important, particularly in view of Open AI’s claim that its new GPT-4o tool can reason. Another point worth highlighting is the bias present in LLM (and more generally Generative AI) models. For an example, see the image created by the Bing Image Creator – the prompt I used was “large language model capturing a user’s attention.”
Discourage users from outsourcing their thinking: Human nature being what it is, many users will be tempted to use these technologies to do their thinking for them. Vendors need to highlight the dangers of doing so. If users do not think a task through before handing it to an LLM, they will not be able to evaluate its output. Thinking task through includes mapping out the steps and content (where relevant), and having an idea of what a reasonable output should look like.
Avoid anthropomorphising LLMs: Marketing will often attribute agency to LLMs by saying things such as “the AI is thinking” or “it thinks you are asking for…”. Such language suggests that LLMs can think or reason as humans do, and biases users towards attributing agency to these tools.
Highlight potential dangers of use in enterprise settings: Vendors spend a lot of time assuring corporate customers that their organisational data will be held securely. However, exposing organisational data (such as data on corporate OneDrive folders) even within the confines of the corporate network can open the possibility of employees being able to query information that they should not have access. Moreover, formulating such queries is super simple because they can be asked in plain English. Vendors claim that this is not an issue if file permissions are implemented properly in the organisation. However, in my experience, people always tend to overshare files within their organisations. Another danger is that the technology opens the possibility of spying on employees. For example, a manager who wants to know what an employee is up to can ask the LLM about which documents an employee has been working on.
Granted, highlighting the above might make some corporate customers wary of rushing in to implement LLM technologies within their organisations. However, I would argue that this is a good thing for vendors in the long run, as it demonstrates a commitment to implementing AI ethically.
–x–
It is appropriate to end this piece by making a final point via another historical note.
The breakthrough that led to the development LLMs was first reported in a highly cited 2017 paper entitled. “Attention is all you need”. The paper describes an architecture (called transformer) that enables neural networks to accurately learn the multiple contexts in which words occur in a large volume of text. If the volume of text is large enough – say a representative chunk of the internet – then a big enough neural network with billions of nodes can be trained to encode the entire vocabulary of the English language in all possible contexts.
The authors’ choice of the “attention” metaphor is inspired because it suggests that the network “learns to attend to” what is important. In the context of humans, however, the word “attention” means much more than just attending to what is important. It also refers to the deep sense of engagement with what we are attending to. The machines we use should help us deepen that engagement, not reduce (let alone eliminate) it. And therein lies the ethical challenge for AI vendors.
–x–x–
Eight to Late 