Archive for the ‘IT Management’ Category
Symptoms, not causes: a systems perspective on project failure
Introduction
A quick search reveals that the topic of project failure gets a fair bit of attention on the Internet. Many of the articles identify factors such as lack of executive support or incomplete/misunderstood requirements as the prime causes of failure. In this post I use concepts from systems theory to argue that commonly identified “causes” of project failure – such as the ones noted above – are symptoms rather than causes. I then surface the real causes of project failure by taking a systems perspective – a viewpoint that considers the project and the hosting organisation as a whole.
As I will argue, project failures can often be traced back to dysfunctional structures and processes within the organisation. These factors usually have little to do with the project directly and are therefore not always obvious at first sight.
Setting the scene
Readers who have waded through the vast literature on failed projects will have noted that there are diverse opinions on the prime reasons for failure. It would take me too long to wade through these articles, so I’ll just pick a source that is well known even if not entirely credible: The Chaos Report by the Standish Group. As per this summary the Chaos Report 2009 claimed the following were the top three causes of project failure:
- Lack of user input.
- Incomplete or changing requirements/specifications.
- Lack of executive support.
Although the report lists the top ten factors, in the interests of space I’ll focus on just these three.
I should mention that the report refers to the above as “Project Challenged Factors.” Grammar issues aside, this is a somewhat strange way of putting it. Anyway, I interpret this phrase to mean reasons (or driving factors) for failure.
Systems theory and projects
First up, what exactly is a system?
Here is a jargon-free definition from the wonderful book by Donella Meadows entitled, Thinking in Systems: A Primer. Incidentally, I highly recommend the book as an easy-to-read and engaging introduction to systems theory:
A system is a set of things interconnected in such a way that they produce their own pattern of behaviour over time. The system may be buffeted, constricted , triggered or driven by outside forces. But its response to these is characteristic of itself and is seldom simple in the real world. (italics mine)
The word interconnected is important because it tells us that when we study something from a systems perspective, we must identify all the important connections it has with its environment. In the case of projects, the important connections are fairly obvious. A project is usually carried out within an organisational setting so it would have many connections to the hosting organisation. Chief amongst these is the fact that a project is staffed and resourced by the hosting organisation (or by an organisation designated by it). Another important connection is that a project will affect different stakeholder groups within the hosting organisation in different ways. However, since all stakeholders have ongoing organisational roles that go beyond the project, the project is not their only interest. This is a key point to which we will return later.
The phrases “own pattern of behaviour over time” and “characteristic of itself” tell us that systems have a unique pattern of behaviour that is characteristic to them. The important point to note is that characteristic behaviour implies that different systems that have the same signature – i.e. identifying features – will tend to behave in the same way. From the perspective of projects this tells us that projects within similar organisations will evolve in similar ways.
A systems view of the causes of project failure
With only this very brief look at systems theory we are now in a position to get some insights into the real causes of project failure. As mentioned earlier, we will focus on the top three reasons ala Standish.
Lack of user input
First up, consider lack of user input. Systems theory tells us that we need to look at the issue from the perspective of the project and the organisation. From this point of view it is clear that users who have been asked to work on the project in addition to their normal duties will view the project as a burden rather than something that may benefit them in the future.
This by itself is not a new insight. In fact, project management gurus have talked themselves hoarse about the need to free up resources etc. However, the point is that organisational structures typically work against this. Firstly, people who are asked to work on a project know that it is an initiative that will end in a finite (usually, reasonably short) time. Therefore, their long terms interests lies in their ongoing roles rather than in projects. Secondly, most organisations are still structured along functional lines and people’s identities are anchored within their functional reporting lines rather than ephemeral project hierarchies.
Changing requirements
The issue of changing requirements and specifications can also be understood from a systems point of view. A characteristic of many systems is that they are stable – i.e. they resist change. Organisations are typically stable systems – they tend to retain their identity despite changes in their environment. One of the characteristics of such organisations is that people in them tend to think and act in set ways – that is, their actions and thinking processes follow well worn patterns to the point where they do not need to think about what they do too deeply.
One of the consequences of this is that when they are asked for requirements users often provide incomplete description of what they do, leaving out significant items that are obvious to them (but not to the analysts who are gathering requirements!). Although I don’t have the figures to back this – I speculate that a fair proportion of changes in requirements are the result of inadequate detail or thought put into developing initial requirements. The point is users and sponsors don’t necessarily see these as changes, but project teams do.
Lack of executive support
Finally, let’s look at the the problem of lack of executive support. Project sponsors usually hold important executive-level positions within the hosting organisation. By virtue of their positions, they have a number of important things that compete for their attention. A project – even an important one – is only one of many things going on in an organisation at any one time. Moreover, organisational priorities do shift, perhaps more often than executives may want to admit. So a project that was the key focus yesterday may be superseded by other priorities today.
There are of course many other ways in which project sponsors can be distracted, but I think I’ve made my point which is that lack of executive support is due to features that are inherent in organisations. So no amount of forcing executives to pay attention to their projects is going to work unless the entire system (project + organisation) changes. And this is difficult if not impossible to achieve because stable systems such as organisations tend to resist change, and therefore continue to display their characteristic patterns of behaviour.
Discussion
So we see that the causes of project failure can be traced back to the organisations in which they are embedded. Specifically, they lie in unwritten norms and formal policies that dictate how the hierarchy operates and how things are done within the organisation. The most important consequence of this is that standard fixes (of encouraging user input and executive support, or instituting change management, say) will not cure the problem of project failure because they do not address the dysfunctional norms and policies that are the root cause of failure.
The above is not news. In fact, the matrix organisation structure was proposed as a response to the need for “project friendly” organisations. I’m no expert on matrix organisations so I will leave it to others to comment on how successful they are. The only point I would make is that in my experience multiple reporting lines (even if dotted and solid) do not work too well. There are always conflicting interests that cause divided loyalties and conflicting interests.
So, the natural question is : what – if anything – can we do about this? The answer is implicit in the foregoing paragraphs. One has to align the project with the organisation, not just at the level of objectives or structure, but also in operational matters such as timelines, budget and resources – the very things that make up the so-called iron triangle. The best time to do this is at the front-end of the project, i.e. the start. At this time, the person(s) driving the project have to engage all stakeholders who will be affected by the initiative and find out their motivations, interests and – most importantly – their concerns regarding the project. If this discussion happens in an open and frank manner, it should surface the issues highlighted In the previous section. Since the discussion takes place even before the project starts, there is at least some hope of addressing these concerns.
There are many ways to structure and facilitate such discussions. Check out this post for an introduction to one and have a look at my book co-authored with Paul Culmsee for much more. That said, one doesn’t need any particular technique – the willingness to discuss difficult matters openly and an openness to other points of view is all that’s needed. That, however, is not always easy to come by…
Conclusion
We have seen that the top causes of project failure can be traced back to the hierarchies and incentive systems of the hosting organisation. Therefore, superficial attempts to fix the problem at the level of individual projects (or even a PMO) will not work. The only hope of addressing the root causes of project failure is to focus on the systemic dysfunctions that cause them.
A visit from the methodology police – a PMO satire
They came for me at 11:00 am.
I was just settling down to finishing that damned business case when I heard the rat-a-tat-tat on my office door. “Come in,” I said, with a touch of irritation in my voice.
The door opened and there they were. They looked at me as though I was something that had crawled out from under a rock. “Mr. Hersey, I presume,” said the taller, uglier one.
“Yes, that’s me.”
“Joe Hersey?” He asked, wanting to make sure before unloading on me.
“Yes, the one and only,” I said, forcing a smile. I had a deep sense of foreboding now: they looked like trouble; I knew they couldn’t be enquiring after my welfare.
“You need to come with us,” said the shorter one. I did imply he was handsomer of the two, but I should clarify that it was a rather close call.
“I have better things to do than follow impolite summons from people I don’t know. I think you should talk to my manager. In fact, I will take you to him,” I replied, rising from my chair. “He won’t be happy that you’ve interrupted my business case. He wants it done by lunchtime,” I added, a tad smugly.
“We’ve already seen him. He knows. I would advise you to come with us. It would make life easier for everyone concerned,” I forget which one of the two said this.
“What is going on?” I asked, toning down my irritation. To be honest, I had no clue what they were on about.
“We’re the methodology police,” they said in unison. I guess they’d had a fair bit of practice scaring the crap out of hapless project managers. “We’re from the PMO,” they added unnecessarily – I mean, where else could they be from.
“Holy s**t,” I said to myself. I was in big trouble.
—
“Well, Hersey,” said the short one, “I think you owe the PMO an explanation.” Ah, I loved his use of the third person– not “us” but “the PMO.”
We were seated at a table in a meeting room deep in the bowels of the PMO: windowless, with low wattage lighting sponsored by one of those new-fangled, energy-saving, greenie bulbs . The three chairs were arranged in interrogation mode , with the two goons on one side and me – Joseph M. Hersey, Project Manager Extraordinaire – on the other.
I was in trouble alright, but I have this perverse streak in me, “I don’t know what you are talking about,” I said, feeling a bit like a hero from a Raymond Chandler novel. I knew what I had done, of course. But I also knew that I was one of the good guys. The clowns sitting opposite me were the forces of evil…such thoughts, though perverse, lifted my spirits.
I must have smiled because the tall one said, “You think this is funny, do you? We have a direct line to the board and we could make life really unpleasant for you if you continue this uncooperative attitude.”
That was bad. I did not want to be hauled up in front of the big cheese. If I was branded a troublemaker at that level, there would be no future for me in the company. And to be absolutely honest, I actually enjoyed working here – visits from the methodology police excepted, of course.
“OK, tell me what you want to know,” I said resignedly.
“No, you tell us, Hersey. We want to hear the whole story of your subversion of process in your own words. We’ll stop you if we need any clarification.” Again, I forget which one of the two said this. Understandable, I think – I was pretty stressed by then.
—
Anyway, there is no sense in boring you with all the PMO and process stuff. Suffice to say, I told them how I partitioned my big project into five little ones, so that each mini project would fall below the threshold criteria for major projects and thus be exempt from following the excruciating methodology that our PMO had instituted.
Process thus subverted, I ran each of the mini projects separately, with deliverables from one feeding into the next. I’d got away with it; with no onerous procedures to follow I was free to devise my own methodology, involving nothing more complicated than a spreadsheet updated daily following informal conversations with team members and stakeholders. All this held together – and, sorry, this is going to sound corny – by trust.
The methodology cops’ ears perked up when they heard that word, “Trust!” they exclaimed, “What do you mean by trust?”
“That’s when you believe people will do as they say they will,” I said. Then added, “A concept that may be foreign to you.” I regretted that snide aside as soon as I said it.
“Look, “ said the uglier guy, “I suggest you save the wisecracks for an audience that may appreciate them. “You are beginning to annoy me and a report to the board is looking like a distinct possibility if you continue in this vein.”
I have to say, if this guy had a lot of patience if he was only just “beginning to get annoyed.” I was aware that I had been baiting him for a while. Yes, I do know when I do that. My wife keeps telling me it will get me into trouble one day. May be today’s the day.
“…I do know what trust is,” the man continued, “but I also know that you cannot run a project on warm and fuzzy notions such as trust, sincerity, commitment etc. The only thing I will trust are written signed off project documents.”
Ah, the folly, the folly. “Tell me this, what would you prefer – project documentation as per the requirements of your methodology or a successful project.”
“The two are not mutually exclusive. In fact, methodology improves the chance of success.”
“No it doesn’t,” I retorted.
“It does,” he lobbed back.
Jeez, this was beginning to sound like recess in the local kindergarten. “Prove it,” I said, staking my claim to the title of King of Kindergarten Debates.
“There are several studies that prove the methodologies efficacy,” said the short one, “but that is not the point.”
“All those studies are sponsored by the Institute,” I said, referring to the August Body that maintains the standard. “so there is a small matter of vested interest….anyway, you say that isn’t the point. So what is your point then?.”
“The methodology is an internal requirement, so you have to follow It regardless. We could have a lot of fun debating it, but that is neither here no there. Compliance is mandatory, you have no choice.”
“I did comply,” I said, “none of my projects were over the threshold, so I did not need to follow the methodology.”
“That was subterfuge – it was one project that you deliberately divided into five so that you could bypass our processes.”
I was getting tired and it was close to my lunchtime. “OK, fair point“ I said, “I should not have done that. I will not do it again. Can I go now?”
“Hmm,” they said in unison. I don’t think either of them believed me. “That’s not good enough.”
I sighed. “What do you want then?” I asked, weary of this pointless drama.
“You will read and sign this form,” said the short one, “declaring you have been trained in the PMO processes – which you were last year, as you well know – and that you will follow the processes henceforth. I particularly urge you to read and digest the bit about the consequences of non-compliance.” He flicked the form in my direction.
I was not surprised to see that the form was a multi-page affair, written in 8pt bureaucratese, utterly incomprehensible to mere mortals such as I. I knew I would continue to bypass or subvert processes that made no sense to me, but I also knew that they needed me to sign that form – their boss would be very unhappy with them if I didn’t. Besides, I didn’t want to stay in that room a second longer than necessary.
“OK, where do I sign,” I said, picking up a pen that lay on the table.
“Don’t you want to read it.”
“Nah,” I said, “I have a pretty fair idea of what it’s about.”
I signed.
“We’re done, Hersey. You can go back to your business case now. But you can be sure that you are on our radar now. We are watching you.”
“Well Gents, enjoy the show. I promise, to lead a faultless life henceforth. I will be a model project manager,” I said as I rose to leave.
“We’re counting on it Hersey. One more violation and you are in deep trouble.”
I refrained from responding with a wisecrack as I exited, leaving them to the paperwork that is their raison d’etre.
On the evolution of corporate information infrastructures
Introduction
In the last few decades two technology trends have changed much of the thinking about corporate IT infrastructures: commoditisation and the cloud. As far as the first trend is concerned, the availability of relatively cheap hardware and packaged “enterprise” software has enabled organisations to create their own IT infrastructures. Yet, despite best efforts of IT executives and planners, most of these infrastructures take on lives of their own, often increasing in complexity to the point where they become unmanageable.
The maturing of cloud technologies in the last few years appears to offer IT decision makers an attractive solution to this problem: that of outsourcing their infrastructure headaches. Notwithstanding the wide variety of mix-and-match options of commodity and cloud offerings, the basic problem still remains: one can create as much of a mess in the cloud as one can in an in-house data center. Moreover, the advertised advantages of cloud-based enterprise solutions can be illusory: customers often find that solutions are inflexible and major changes can cost substantial sums of money.
Conventional wisdom tells us that these problems can be tackled by proper planning and control. In this post I draw on Claudio Ciborra’s book, From Control to Drift: The Dynamics of Corporate Information Infrastructures, to show why such a view is simplistic and essentially untenable.
The effects of globalisation and modernity
The basic point made by Ciborra and Co. is that initiatives to plan and control IT infrastructures via centrally-driven, standards-based governance structures are essentially misguided reactions to the unsettling effects of globalisation and modernity, terms that I elaborate on below.
Globalisation
Globalisation refers to the processes of interaction and integration between people of different cultures across geographical boundaries. The increasing number of corporations with a global presence is one of the manifestations of globalisation. For such organisations, IT infrastructures systems are seen as a means to facilitate globalisation and also control it.
There are four strategies that an organisation can choose from when establishing a global presence. These are:
- Multinational: Where individual subsidiaries are operated autonomously.
- International: Where work practices from the parent company diffuse through the subsidiaries (in a non-formal way).
- Global: Where local business activities are closely controlled by the parent corporation.
- Transnational: This (ideal) model balances central control and local autonomy in a way that meets the needs of the corporation while taking into account the uniqueness of local conditions.
These four business strategies map to two corporate IT strategies:
- Autonomous: where individual subsidiaries have their own IT strategies, loosely governed by corporate.
- Headquarters-driven: where IT operations are tightly controlled by the parent corporation.
Neither is perfect; both have downsides that start to become evident only after a particular strategy is implemented. Given this, it is no surprise that organisations tend to cycle between the two strategies, with cycle times varying from five to ten years; a trend that corporate IT minions are all too familiar with. Typically, though, executive management tends to favour the centrally-driven approach since it holds the promise of higher control and reduced costs.
Another consequence of globalisation is the trend towards outsourcing IT infrastructure and services. This is particularly popular for operational IT – things like infrastructure and support. In view of this, it is no surprise that organisations often choose to outsource IT development and support to external vendors. Equally unsurprising, perhaps, is that the quality of service often does not match expectations and there’s little that can be done about it. The reason is simple: complex contracts are hard to manage and perhaps more importantly, not everything can be contractualised. See my post on the transaction cost economics of outsourcing for more on this point.
The effect of modernity
The phenomenon of modernity forms an essential part of the backdrop against which IT systems are implemented. According to a sociological definition due to Anthony Giddens, modernity is “associated with (1) a certain set of attitudes towards the world, the idea of the world as open to transformation, by human intervention; (2) a complex of economic institutions, especially industrial production and a market economy; (3) a certain range of political institutions, including the nation-state and mass democracy”
Modernity is characterised by the following three “forces” that have a direct impact on information infrastructures:
- The separation of space and time: This refers to the ways in which technology enables us reconfigure our notions of geographical space and time. For instance, coordinating activities in distant locations is now possible – global supply chains and distributed project teams being good examples. The important consequence of this ability, relevant to IT infrastructures such as ERP and CRM systems, is that it makes it possible (at least in principle) for organisations to increase their level of surveillance and control of key business processes across the globe.
- The development of disembedding mechanisms: As I have discussed at length in this post, organisations often “import” procedures that have worked well in organisations. The assumption underlying this practice is that the procedures can be lifted out of their original context and implemented in another one without change. This, in turn, tacitly assumes that those responsible for implementing the procedure in the new context understand the underlying cause-effect relationships completely. This world-view, where organisational processes and procedures are elevated to the status of universal “best practices” is an example of a disembedding mechanism at work. Disembedding mechanisms are essentially processes via which certain facts are abstracted from their context and ascribed a universal meaning. Indeed, most “enterprise” class systems claim to implement such “best practices.”
- The reflexivity of knowledge and practice: Reflexive phenomena are those for which cause-effect relationships are bi-directional – i.e. causes determine effects which in turn modify the causes. Such phenomena are unstable in the sense that they are continually evolving – in potentially unpredictable ways. Organisational practices (which are based on organisational knowledge) are reflexive in the sense that they are continually modified in the light of their results or effects. This conflicts with the main rationale for IT infrastructures such as ERP systems, which is to rationalise and automate organisational processes and procedures in a relatively inflexible manner.
Implications for organisations
One of the main implications of globalisation and modernity is that the world is now more interconnected than ever before. This is illustrated by the global repercussions of the financial crises that have occurred in recent times. For globalised organisations this manifests itself in not-so-obvious dependencies of the organisation’s well-being on events within the organisation and outside it. These events are usually not within the organisation’s control So they have to be managed as risks.
A standard response to risk is to increase control. Arguably, this may well be the most common executive-level rationale behind decisions to impose stringent controls and governance structures around IT infrastructures. Yet, paradoxically, the imposition of controls often lead to undesirable outcomes because of unforeseen side effects and the inability to respond to changing business needs in a timely manner.
A bit about standards
Planners of IT infrastructures spend a great deal of time worrying about which standards they should follow. This makes sense if for no other reason than the fact that corporate IT infrastructures are embedded in a larger (external) ecosystem that is made up of diverse organisations, each with their own infrastructures. Standards ease the problem of communication between interconnected organisations. For example, organisations often have to exchange information electronically in various formats. Without (imposed or de-facto) standards, this would be very difficult as IT staff would have to write custom programs to convert files from one format to another.
The example of file formats illustrates why those who plan and implement IT infrastructures prefer to go with well established technologies and standards rather than with promising (but unproven) new ones. The latter often cause headaches because of compatibility problems with preexisting technologies. There are other reasons, of course, for staying with older technologies and established standards – acceptance, maturity and reliability being a few important ones.
Although the rationale for adopting standards seems like a sound one, there are a few downsides too. Consider the following:
- Lock in: This refers to the fact that once a technology is widely adopted, it is very difficult for competing technologies to develop. The main reason for this is that dominant technology will attract a large number of complementary products. These make it more attractive to stick with the dominant standard. Additionally, contractual commitments, availability of expertise, switching costs make it unviable for customers to move to competitor products.
- Inefficiency: This refers to the fact that a dominant standard is not necessarily the best. There are many examples of cases where a dominant standard is demonstrably inferior to a less popular competitor. My favourite example is the Waterfall project management methodology which became a standard for reasons other than its efficacy. See this paper for details of this fascinating story.
- Incompatibility: In recent years, consumer devices such as smartphones and tablets have made their way into corporate computing environments, primarily because of pressures and demands from technology savvy end-users. These devices pose problems for infrastructure planners and administrators because they are typically incompatible with existing corporate technology standards and procedures. As an example, organisations that have standardised on a particular platform such as Microsoft Windows may face major challenges when introducing devices such as iPads in their environments.
Finally, and most importantly, the evolution of standards causes major headaches for corporate IT infrastructure planners. Anyone who has been through a major upgrade of an operating system at an organisation-wide level will have lived this pain. Indeed, it is such experiences that have driven IT decision-makers to cloud offerings. The cloud brings with it a different set of problems, but that’s another story. Suffice to say that the above highlights, once again, the main theme of the book: that infrastructure planning is well and good, but planners have to be aware that the choices they make constrain them in ways that they will not have foreseen.
Summing up
The main argument that Ciborra and his associates make is that corporate information infrastructures drift because they are subject to unpredictable forces within and outside the hosting organisation. Standards and processes may slow the drift (if at all) but they cannot arrest it entirely. Infrastructures are therefore best seen as ever-evolving constructs made up systems, people and processes that interact with each other in (often) unforeseen ways. As Ciborra so elegantly puts it:
Corporate information infrastructures are puzzles, or better collages, and so are the design and implementation processes that lead to their construction and operation. They are embedded in larger, contextual puzzles and collages. Interdependence, intricacy, and interweaving of people, systems, and processes are the culture bed of infrastructure. Patching, alignment of heterogeneous actors and making do are the most frequent approaches…irrespective of whether management [is] planning or strategy oriented, or inclined to react to contingencies.
And therein lies an important message for those who plan and oversee information infrastructures.
Notes:
Sections of this post are drawn from my article entitled, The ERP Paradox.
Eight to Late 