Archive for the ‘Bias’ Category
The role of cognitive biases in project failure
Introduction
There are two distinct views of project management practice: the rational view which focuses on management tools and techniques such as those espoused by frameworks and methodologies, and the social/behavioural view which looks at the social aspect of projects – i.e. how people behave and interact in the context of a project and the wider organisation. The difference between the two is significant: one looks at how projects should be managed, it prescribes tools, techniques and practices; the other at what actually happens on projects, how people interact and how managers make decisions. The gap between the two can sometimes spell the difference between project success and failure. In many failed projects, the failure can be traced back to poor decisions, and the decisions themselves to cognitive biases: i.e. errors in judgement based on perceptions. A paper entitled, Systematic Biases and Culture in Project Failure, by Barry Shore looks at the role played by selected cognitive biases in the failure of some high profile projects. The paper also draws some general conclusions on the relationship between organisational culture and cognitive bias. This post presents a summary and review of the paper.
The paper begins with a brief discussion of the difference in the rational and social/behavioural view of project management. The rational view is prescriptive – it describes management procedures and techniques which claim to increase the chances of success if followed. Further, it emphasises causal effects (if you follow X procedure then Y happens). The social/behavioural view is less well developed because it looks at human behaviour which is hard to study in controlled conditions, let alone in projects. Yet, developments in behavioural economics – mostly based on the pioneering work of Kahnemann and Tversky – can be directly applied to project management (see my post on biases in project estimation, for instance). In the paper, Shore looks at eight case studies of failed projects and attempts to attribute their failure to selected cognitive biases. He also looks into the relationship between (project and organisational) culture and the prevalence of the selected biases. Following Hofstede, he defines organisational culture as shared perceptions of organisational work practices and, analogously, project culture as shared perceptions of project work practices. Since projects take place within organisations, project culture is obviously influenced by the organisational culture.
Scope and Methodology
In this section I present a brief discussion of the biases that the paper focuses on and the study methodology.
There are a large number of cognitive biases in the literature. The author selects the following for his study:
Available data: Restricting oneself to using data that is readily or conveniently available. Note that “Available data” is a non-standard term: it is normally referred to as a sampling bias, which in turn is a type of selection bias.
Conservatism (Semmelweis reflex): Failing to consider new information or negative feedback.
Escalation of commitment: Allocating additional resources to a project that is unlikely to succeed.
Groupthink: Members of a project group under pressure to think alike, ignoring evidence that may threaten their views.
Illusion of control: Management believing they have more control over a situation than an objective evaluation would suggest.
Overconfidence: Having a level of confidence that is unsupported by evidence or performance.
Recency (serial position effect): Undue emphasis being placed on most recent data (ignoring older data)
Selective perception: Viewing a situation subjectively; perceiving only certain (convenient) aspects of a situation.
Sunk cost: Not accepting that costs already incurred cannot be recovered and should not be considered as criteria for future decisions. This bias is closely related to loss aversion.
The author acknowledges that there is a significant overlap between some of these effects: for example, illusion of control has much in common with overconfidence. This implies a certain degree of subjectivity in assigning these as causes for project failures.
The failed projects studied in the paper are high profile efforts that failed in one or more ways. The author obtained data for the projects from public and government sources. He then presented the data and case studies to five independent groups of business professionals (constituted from a class he was teaching) and asked them to reach a consensus on which biases could have played a role in causing the failures. The groups presented their results to the entire class, then through discussions, reached agreement on which of the biases may have lead to the failures.
The case studies
This section describes the failed project studied and the biases that the group identified as being relevant.
Airbus 380: Airbus was founded as a consortium of independent aerospace companies. The A380 project which was started in 2000 – was aimed at creating the A380 superjumbo jet with a capacity of 800 passengers. The project involved coordination between many sites. Six years into the project, when the aircraft was being assembled in Toulouse, it was found that a wiring harness produced in Hamburg failed to fit the airframe.
The group identified the following biases as being relevant to the failure of the Airbus project:
Selective perception: Managers acted to guard their own interests and constituencies.
Groupthink: Each participating organisation worked in isolation from the others, creating an environment in which groupthink would thrive.
Illusion of control: Corporate management assumed they had control over participating organisations.
Availability bias: Management in each of the facilities did not have access to data in other facilities, and thus made decisions based on limited data.
Coast Guard Maritime Domain Awareness Project: This project, initated in 2001, was aimed at creating the maritime equivalent of an air traffic control system. It was to use a range of technologies, and involved coordination between many US government agencies. The goal of the first phase of the project was to create a surveillance system that would be able to track boats as small as jet skis. The surveillance data was to be run through a software system that would flag potential threats. In 2006 – during the testing phase – the surveillance system failed to meet quality criteria. Further, the analysis software was not ready for testing.
The group identified the following biases as being relevant to the failure of the Maritime Awareness project:
Illusion of control: Coordinating several federal agencies is a complex task. This suggests that project managers may have thought they had more control than they actually did.
Selective perception: Separate agencies worked only on their portions of the project, failing to see the larger picture. This suggests that project groups may have unwittingly been victims of selective perception.
Columbia Shuttle: The Columbia Shuttle disaster was caused by a piece of foam insulation breaking off the propellant tank and damaging the wing. The problem with the foam sections was known, but management had assumed that it posed no risk.
In their analysis, the group found the following biases to be relevant to the failure of this project:
Conservatism: Management failed to take into account negative data.
Overconfidence: Management was confident there were no safety issues.
Recency: Although foam insulation had broken off on previous flights, it had not caused any problems.
Denver Airport Baggage Handling System: The Denver airport project, which was scheduled for completion in 1993, was to feature a completely automated baggage handling system. The technical challenges were enormous because the proposed system was an order of magnitude more complex than those that existed at the time. The system was completed in 1995, but was riddled with problems. After almost a decade of struggling to fix the problems, not to mention being billions over-budget, the project was abandoned in 2005.
The group identified the following biases as playing a role in the failure of this project:
Overconfidence: Although the project was technically very ambitious, the contractor (BAE systems) assumed that all technical obstacles could be overcome within the project timeframes.
Sunk cost: The customers (United Airlines) did not pull out of the project even when other customers pulled out, suggesting that they were reluctant to write off already incurred costs.
Illusion of control: Despite evidence to the contrary, management assumed that problems could be solved and that the project remained under control.
Mars Climate Orbiter and Mars Polar Lander: Telemetry signals from the Mars climate orbiter ceased when the spacecraft approached its destination. The root cause of the problem was found to be a failure to convert between metric and British units: apparently the contractor, Lockheed, had used British units in the engine design but NASA scientists who were responsible for operations and flight assumed the data was in metric units. A few months after the climate orbiter disaster, another spacecraft, the Mars polar lander fell silent just short of landing on the surface of Mars. The failure was attributed to a software problem that caused the engines to shutdown prematurely, thereby causing the spacecraft to crash.
The group attributed the above project failures to the following biases:
Conservatism: Project engineers failed to take action when they noticed that the spacecraft was off-trajectory early in the flight.
Sunk cost: Managers were under pressure to launch the spacecraft on time – waiting until the next launch window would have entailed a wait of many months thus “wasting” the effort up to that point. (Note: In my opinion this is an incorrect interpretation of sunk cost)
Selective perception: The spacecraft modules were constructed by several different teams. It is very likely that teams worked with a very limited view of the project (one which was relevant to their module).
Merck Vioxx: Vioxx was a very successful anti-inflammatory medication developed and marketed by Merck. An article published in 2000 suggested that Merck misrepresented clinical trial data, and another paper published in 2001 suggested that those who took Vioxx were subject to a significantly increased risk of assorted cardiac events. Under pressure, Merck put a warning label on the product in 2002. Finally, the drug was withdrawn from the market in 2004 after over 80 million people had taken it.
The group found the following biases to be relevant to the failure of this project:
Conservatism: The company ignored early warning signs about the toxicity of the drug.
Sunk cost: By the time concerns were raised, the company had already spent a large amount of money in developing the drug. It is therefore likely that there was a reluctance to write off the costs incurred to that point.
Microsoft Xbox 360: The Microsoft Xbox console was released to market in 2005, a year before comparable offerings from its competitors. The product was plagued with problems from the start; some of them include: internet connectivity issues, damage caused to game disks, faulty power cords and assorted operational issues. The volume of problems and complaints prompted Microsoft to extend the product warranty from one to three years at an expected cost of $1 billion.
The group thought that the following biases were significant in this case:
Conservatism: Despite the early negative feedback (complaints and product returns), the development group seemed to acknowledge that there were problems with the product.
Groupthink: It is possible that the project team ignored data that threatened their views on the product. The group reached this conclusion because Microsoft seemed reluctant to comment publicly on the causes of problems.
Sunk cost: By the time problems were identified, Microsoft had invested a considerable sum of money on product development. This suggests that the sunk cost trap may have played a role in this project failure.
NYC Police Communications System: (Note: I couldn’t find any pertinent links to this project). In brief: the project was aimed at developing a communications system that would enable officers working in the subway system to communicate with those on the streets. The project was initiated in 1999 and scheduled for completion in 2004 with a budgeted cost of $115 million. A potential interference problem was identified in 2001 but the contractors ignored it. The project was completed in 2007, but during trials it became apparent that interference was indeed a problem. Fixing the issue was expected to increase the cost by $95 million.
The group thought that the following biases may have contributed to the failure of this project:
Conservatism: Project managers failed to take early data on intereference account.
Illusion of control: The project team believed – until very late in the project – that the interference issue could be fixed.
Overconfidence: Project managers believed that the design was sound, despite evidence to the contrary.
Analysis and discussion
The following four biases appeared more often than others: Conservatism, illusion of control, selective perception and sunk cost.
The following biases appeared less often: groupthink and overconfidence.
Recency and availability were mentioned only once.
Based on the small data sample and the somewhat informal means of analysis, the author concludes that the first four biases may be dominant in project management. In my opinion this conclusion is shaky because the study has a few shortcomings, which I list below:
- The sample size is small
- The sample covers a range of domains.
- No checks were done to verify the group members’ understanding of all the biases.
- The data on which the conclusions are based is incomplete – based only on publicly available data. (perhaps is this an example of the available data bias at work?)
- A limited set of biases is used – there could be other biases at work.
- The conclusions themselves are subject to group-level biases such as groupthink. This is a particular concern because the group was specifically instructed to look at the case studies through the lens of the selected cognitive biases.
- The analysis is far from exhaustive or objective; it was done as a part of classroom exercise.
For the above reasons, the analysis is at best suggestive: it indicates that biases may play a role in the decisions that lead to project failures.
The author also draws a link between organisational culture and environments in which biases might thrive. To do this, he maps the biases on to the competing values framework of organisational culture, which views organisations along two dimensions:
- The focus of the organisation – internal or external.
- The level of management control in the organisation – controlling (stable) or discretionary (flexible).
According to the author, all nine biases are more likely in a stability (or control) focused environment than a flexible one, and all barring sunk cost are more likely to thrive in a internal focused organisation than an externally focused one. This conclusion makes sense: project teams are more likely to avoid biases when empowered to make decisions, free from management and organisational pressures. Furthermore, biases are also less likely to play a role when external input – such as customer feedback – is taken seriously.
That said, the negative effects of internally focused, high control organisations can be countered. The author quotes two examples:
- When designing the 777 aircraft, Boeing introduced a new approach to project management wherein teams were required to include representatives from all groups of stakeholders. The team was encouraged to air differences in opinion and to deal with these in an open manner. This approach has been partly credit for the success of the 777 project.
- Since the Vioxx debacle, Merck rewards research scientists who terminate projects that do not look promising.
Conclusions
Despite my misgivings about the research sample and methodology, the study does suggest that standard project management practices could benefit by incorporating insights from behavioural studies. Further, the analysis indicates that cognitive biases may have indeed played a role in the failure of some high profile projects. My biggest concern here, as stated earlier, is that the groups were required to associate the decisions with specific biases – i.e. there was an assumption that one or more of the biases from the (arbitrarily chosen) list was responsible for the failure. In reality, however, there may have been other more important factors at work.
The connections with organisational culture are interesting too, but hardly surprising: people are more likely to do the right thing when management empowers them with responsibility and authority.
In closing: I found the paper interesting because it deals with an area that isn’t very well represented in the project management literature. Further, I believe these biases play a significant role in project decision making, especially in internally focussed / controlled organisations (project managers are human, and hence not immune…). However, although the paper supports this view, it doesn’t make a wholly convincing case for it.
Further Reading
For more on cognitive biases in organisations, see Chapter 2 of my book, The Heretic’s Guide to Best Practices.
Measuring the unmeasurable: a note on the pitfalls of performance metrics
Many organisations measure performance – of people, projects processes or whatever – using quantitative metrics, or KPIs as they are often called. Some examples of these include: calls answered / hour (for a person working in a contact centre); % complete (for a project task) and orders processed / hour (for an order handling process). The rationale for measuring performance quantitatively is rooted in Taylorism or scientific management. The early successes of Taylorism in improving efficiencies on the shopfloor lead to its adoption in other areas of management. The scientific approach to management underlies the assumption that metrics are a Good Thing, echoing the words of the 19th century master physicist, Lord Kelvin:
When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge of it is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced it to the stage of science.
This is a fine sentiment for science: precise measurement is a keystone of physics and other natural sciences. So much so, that scientists expend a great deal of effort in refining and perfecting certain measurements. However, it can be misleading and sometimes downright counterproductive to attempt such quantification in management. This post explains why I think so.
Firstly, there are basically two categories of things (indicators, characteristics or whatever) that management attempts to quantify when defining performance metrics– tangible (such as number of calls per unit time) and intangible (for example, employee performance on a five point scale). Although people attach numerical scores to both kinds of things, I’m sure most people would agree that any quantification of employee performance is way more subjective than number of calls per unit time. Now, it is possible to reduce this subjectivity by associating the intangible characteristic to a tangible one – for example, employee performance can be tied to sales (for a sales rep), r number of projects successfully completed (for a project manager) or customer satisfaction as measured by surveys (for a customer service representative). However, all such attempts result in a limited view of the characteristic being measured. Such associated tangible metrics cannot measure all aspects of the intangible metric in question. In the case at hand – employee performance – factors such as enthusiasm, motivation, doing things beyond the call of duty etc., all of which are important aspects of employee performance, remain unmeasurable. So as a first point we have the following: attaching a numerical score to intangible quantities is fraught with subjectivity and ambiguity.
But even measures of tangible characteristics can have issues. An example that comes to mind is the infamous % complete metric for tasks in a project management. Many project managers record a progress by noting that a task – say data migration – is 70% complete. But, what does this figure mean? Does it mean that 70% of the data has been migrated (and what does that mean anyway?), or is it that 70% of the total effort required (as measured against days allocated to the task) has been expended. Most often, the figure quoted has no explanation as to what it means – and everyone interprets it in a way that best suits their agenda. My point here is: a well designed metric should include an unambiguous statement as to what is being measured, how it is to be measured and how it is to be interpreted. Many seemingly well defined metrics do not satisfy this criterion – the % complete metric being a sterling example. These give the illusion of precision, which can be more harmful than having no measurement at all. My second point is thus summarised as follows: it is hard to design unambiguous metrics, even for tangible performance characteristics. Of course, speaking of the % complete metric, many project managers now understand its shortcomings and use an “all or nothing” approach – a task is either 0% complete (not started or in progress) or 100% complete (truly complete).
Another danger of quantification of performance is highlighted by Eliyahu Goldratt in his book The Haystack Syndrome. To quote from the book:
…Tell me how you measure me and I will tell you how I will behave. If you measure me in an illogical way…do not complain about illogical behaviour…
A case in point is the customer contact centre employee who is measured by calls handled per hour. The employee knows he has to maximise calls taken, so he ends up trying to keep conversations short – even if it means upsetting customers. By trying to improve call throughput, the company ends up reducing quality of service. Fortunately, some service companies are beginning to understand this – read about Repco‘s experience in this article from MIS Australia, for example. The take-home point here is: performance measurements that focus on the wrong metric have the potential to distort employee behaviour to the detriment of the organisation.
Finally, metrics that rely on human judgements are subject to cognitive bias. Specifically, it is well known that biases such as anchoring and framing can play a big role in determining the response received to a question such as, “How would you rate X’s performance on a scale of 1 to 5 (best performance being 5)?” In earlier posts, I’ve written about the role of cognitive biases in project task estimation and project management research. The effect of these biases on performance metrics can be summarised as follows: since many performance metrics rely on subjective judgements made by humans, these metrics are subject to cognitive biases. It is difficult, if not impossible, to correct for these biases.
To conclude: it is difficult to design performance metrics that are unambiguous, unbiased and do not distort behaviour. Use them if you must – or are required to do so by your organisation – but design and interpret them with care because, if used unthinkingly, they can cause terminal damage to employee morale.
Anchored and over-optimistic: why quick and dirty estimates are almost always incorrect
Some time ago, a sales manager barged into my office. “I’m sorry for the short notice,” she said, “but you’ll need to make some modifications to the consolidated sales report by tomorrow evening.”
I could see she was stressed and I wanted to help, but there was an obvious question that needed to be asked. “What do you need done? I’ll have to get some details before I can tell you if it can be done within the time,” I replied.
She pulled up a chair and proceeded to explain what was needed. Within a minute or two I knew there was no way I could get it finished by the next day. I told her so.
“Oh no…this is really important. How long will it take?”
I thought about it for a minute or so. “OK how about I try to get it to you by day after?”
“Tomorrow would be better, but I can wait till day after.” She didn’t look very happy about it though. “Thanks,” she said and rushed away, not giving me a chance to reconsider my off-the-cuff estimate.
—
After she left, I had a closer look at what needed to be done. Soon I realised it would take me at least twice as long if I wanted to do it right. As it was, I’d have to work late to get it done in the agreed time, and may even have to cut a corner or two ( or three) in the process.
So why was I so wide off the mark?
I had been railroaded into giving the manager an unrealistic estimate without even realising it. When the manager quoted her timeline, my subconscious latched on to it as an initial value for my estimate. Although I revised the initial estimate upwards, I was “pressured” – albeit unknowingly – into quoting an estimate that was biased towards the timeline she’d mentioned. I was a victim of what psychologists call anchoring bias – a human tendency to base judgements on a single piece of information or data, ignoring all other relevant factors. In arriving at my estimate, I had focused on one piece of data (her timeline) to the exclusion of all other potentially significant information (the complexity of the task, other things on my plate etc.).
Anchoring bias was first described by Amos Tversky and Daniel Kahnemann in their pioneering paper entitled, Judgement under Uncertainty: Heuristics and Biases. Tversky and Kahnemann found that people often make quick judgements based on initial (or anchor) values that are suggested to them. As the incident above illustrates, the anchor value (the manager’s timeline) may have nothing to do with the point in question (how long it would actually take me to do the work). To be sure, folks generally adjust the anchor values based on other information. These adjustments, however, are generally inadequate. The final estimates arrived at are incorrect because they remain biased towards the initial value. As Tversky and Kahnemann state in their paper:
In many situations, people make estimates by starting from an initial value that is adjusted to yield the final answer. The initial value, or starting point, may be suggested by the formulation of the problem, or it may be the result of a partial computation. In either case, adjustments are typically insufficient. That is, different starting points yield different estimates, which are biased toward the initial values. We call this phenomenon anchoring.
Although the above quote may sound somewhat academic, be assured that anchoring is very real. It affects even day-to-day decisions that people make. For example, in this paper Neil Stewart presents evidence that credit card holders repay their debt more slowly when their statements suggest a minimum payment. In other words the minimum payment works as an anchor, causing the card holder to pay a smaller amount than they would have been prepared to (in the absence of an anchor).
Anchoring, however, is only part of the story. Things get much worse for complex tasks because another bias comes into play. Tversky and Kahnemann found that subjects tended to be over optimistic when asked to make predictions regarding complex matters. Again, quoting from their paper:
Biases in the evaluation of compound events are particularly significant in the context of planning. The successful completion of an undertaking, such as the development of a new product, typically has a conjunctive character: for the undertaking to succeed, each of a series of events must occur. Even when each of these events is very likely, the overall probability of success can be quite low if the number of events is large. The general tendency to overestimate the probability of conjunctive events leads to unwarranted optimism in the evaluation of the likelihood that a plan will succeed or that a project will be completed on time.
Such over-optimism in the face of complex tasks is sometimes referred to as the planning fallacy.1
Of course, as discussed by Kahnemann and Fredrick in this paper, biases such as anchoring and the planning fallacy can be avoided by a careful, reflective approach to estimation – as opposed to a “quick and dirty” or intuitive one. Basically, a reflective approach seeks to eliminate bias by reducing the effect of individual judgements. This is why project management texts advise us (among other things) to:
- Base estimates on historical data for similar tasks. This is the basis of reference class forecasting which I have written about in an earlier post.
- Draft independent experts to do the estimation.
- Use multipoint estimates (best and worst case scenarios)
In big-bang approaches to project management, one has to make a conscious effort to eliminate bias because there are fewer chances to get it right. On the other hand, iterative / incremental methodologies have bias elimination built-in because one starts with initial estimates, which include inaccuracies due to bias, and subsequently refine these as one progresses. The estimates get better as one goes along because every refinement is based on an improved knowledge of the task.
Anchoring and the planning fallacy are examples cognitive biases – patterns of deviations of judgement that humans display in a variety of situations. Since the pioneering work of Tversky and Kahnemann, these biases have been widely studied by psychologists. It is important to note that these biases come into play whenever quick and dirty judgements are involved. They occur even when subjects are motivated to make accurate judgements. As Tversky and Kahnemann state towards the end of their paper:
These biases are not attributable to motivational effects such as wishful thinking or the distortion of judgments by payoffs and penalties. Indeed, several of the severe errors of judgment reported earlier (in the paper) occurred despite the fact that subjects were encouraged to be accurate and were rewarded for the correct answers.
The only way to avoid cognitive biases in estimating is to proceed with care and consideration. Yes, that’s a time consuming, effort-laden process, but that’s the price one pays for doing it right. To paraphrase Euclid, there is no royal road to estimation.
1 The planning fallacy is related to optimism bias which I have discussed in my post on reference class forecasting.
Eight to Late 