Eight to Late

Sensemaking and Analytics for Organizations

Archive for the ‘Organizational Culture’ Category

On the relationship between projects and organisations

with 3 comments

Introduction

Most of the research and practice literature on project management tends to view projects as being isolated from their environment.  It is obvious to anyone who has worked on a project that this isn’t so. In view of this,  it is useful to look at the relationship between projects and the organisations that host them.  This post looks at this issue, drawing on a paper by Gernot Grabher entitled, Cool Projects, Boring Institutions: Temporary Collaboration in Social Context.

The emergence of projects

Grabher begins his discussion with a sketch of the how projects emerged as a distinct work form. Projects  – i.e. time bound, goal focused activities – have always been around. The modern notion of a project, however,   arose from a development philosophy that came out of the US Department of Defense in the 1950s.  He states,

…Instead of fragmenting and pre-specifying the development of military technologies along functional disciplines, these technologies were described in relation to their objectives, i.e. the military parameters of these weapons. The pacing of these concentrated efforts was crucial: parameters had to be met, goals had to be accomplished according to a grand scheme (program?) to win the armament race. Development processes that earlier were seen as separate activities were now conceptualized as an integrated entity called a program, system or project. The overwhelming scale of these projects in terms of financial and scientific resources as well as their ambitious timing created formidable problems of coordination and control. Experiments with various forms of organizational control ultimately lead to the professionalization of the role of the project manager…

From thereon the concepts of projects and project management were taken up (with much enthusiasm and optimism) by business and industry. The formalization of various project management methodologies, standards , qualifications and trade journals can be seen a culmination of this process.

Given the military-industrial origins of the profession, it is easy to see why a “command and control” philosophy dominates much of project management thought and practice. Many of the early projects that are paraded as textbook examples of successful projects operated outside normal organizational oversight. They were, to a large extent, deliberately shielded from external influences.  I believe this is why isolation from the environment is seen as a Good Thing by project managers –  problems of coordination and control become so  much simpler when one does not have to manage relationships and politics that are (perceived as being) external to a project. This practice may be necessary and workable for classified projects that run on billion dollar budgets, but it doesn’t work so well in   environments that most project managers work in. Projects don’t take place in a   vacuum; they are born, live and die in real-world organizations. To forget that is to see the “tree of the project” and miss the “forest of the organization.” This is particularly so because,  unlike those near-mythical mega-projects of the 1950s,  the efforts that you and I work on are deeply entwined with their hosting organizations.

Organisation-related characteristics of projects

Grabher then notes some characteristics of projects. I summarize these in the next few paragraphs.

First, it is interesting that the original meaning of the word “project” referred to a “proposal” or “idea”, rather than a “directed, time-bound effort.” Grabher points out that this  shift in meaning was  accompanied by a shift in focus: from project as idea (or goal) to project as process (or means to achieving a goal).   Projects are thus seen as vehicles for achieving organisational goals.

Second, Grabher notes that projects are often hard to decompose into constituent tasks, and that such a (commonly agreed) decomposition is only possible when stakeholders interrelate with each other continually.  This underscores the importance of communication in projects.

Third, Grabher highlights the importance of the project manager (he uses the term contractor) as the “lynchpin on whom trust is focused.” The role of the manager is particularly important in projects on which team members do not have the time to get to know each other well.

Fourth, the project manager / contractor is also the wielder of organizational authority as far as the project is concerned. He or she is, in this sense, a representative of the organization – a person whose presence underlines the fact that the project exists to achieve specified organizational goals.

Finally, deadlines are a defining aspect of projects. They serve several functions. For example, they ensure that a sense of urgency for action and progress remains through the duration of the project. They also might serve to legitimize execution of project work without external interference (this argument was frequently used in the military-industrial projects of the 1950s). But above all, the final deadline,  which culminates in the termination of the project, also serves as a connector to the rest of the organization. It is a time in which handoffs, documentation, team disbanding etc. occurs, thus enabling the results and  experiences from the project disperse into the wider organization.

Projects in organisations

The characteristics noted above highlight the dual nature of projects: on the one hand, as noted earlier, projects are seen as semi-autonomous temporary organisations, but on the other they are also firmly embedded within the hosting organisation. An effect of the latter is particularly evident  in consulting and software services firms (or even corporate IT shops), which tend to do similar projects over and over.  As Grabher notes,

[projects] apparently operate in a milieu of recurrent collaboration that, after several project cycles, fills a pool of resources and gels into latent networks. Project organising is mostly directed towards the actual realization of a potential that is generated and reproduced by the practice of drawing on core members of (successful) prior projects to serve on derivative successor projects. Such chains of repeated co-operation are held together (or cut off ) by the reputation members gain (or lose) in previous collaborations…

Another aspect of embedded-ness is the co-location of team members within a larger organizational milieu. The standard benefits of co-location are well known. These are:

  1. Savings of transactional costs such as those incurred in communication, supervision of staff at remote locations etc. See my post on a transaction cost view of outsourcing for more on this.
  2. Co-location improves the efficacy of communication by encouraging face-to-face interactions.
  3. It enables “near real-time” monitoring of the health of the project and its environment.

There’s more though. Grabher notes that  in addition to the above “intentional” or “strategic” benefits, co-location also ensures that  team members are exposed to the same organizational noise – which consists of  a “concoction of rumours, impressions, recommendations, trade folklore and strategic misinformation (falsehoods!).”  Co-location enables project teams to make collective sense of organisational noise – this shared understanding of the environment can contribute significantly to the creation of a team spirit.

A related notionis that of enculturation: that is, the process of becoming an accepted member of the group, or an insider. This has less to do with expertise and knowledge than learning the unspoken rules and norms of a community.  Although becoming a member of  a community has much to do with social interactions within the workplace, there is more: a lot of essential know-how and know-what is transferred through informal interactions between senior members of the team (who are often senior members of the organisation) and others.

Projects generally need to draw upon a range of organizational resources: people and physical infrastructure being the most obvious ones.   Grabher notes that the increasing projectisation of organizations can be attributed to a perception that project-based management is an efficient way to allocate productive resources in a flexible manner   (…whether this perception is correct, is another matter altogether). However, there are other less obvious influences that organisations exert too.  For example,  Grabher points out that organizational norms and rules provide the basis for the emergence of swift trust, which is trust based on roles and professional ability rather than individuals and personalities.  Further, at a higher level,  organizational culture plays a role in determining how a project is governed, managed and run. These explicit and implicit norms have a stabilising influence on projects.

In addition to the stabilizing influence of the hosting organisation, projects also offer opportunities to build and enhance links between organisations – for instance, strategic partnerships.  This is, in effect, institution building aimed at leveraging the strengths of the participating organisations for a greater joint benefit. In such situations the participating organisations take on the role of “lynchpins” on whom trust is focused.

Grabher makes the point that firms (and institutions comprised of firms) not only provide resources that make projects possible, but also host a range of processes that are needed to organize and run projects. For one, projects are usually preceded by several organisational processes involving deliberation, selection and preparation. These activities have to occur for a project to happen, but they normally fall outside the purview of the project.

A somewhat paradoxical aspect of projects is although they offer the opportunity for enhancing organizational knowledge, this rarely happens in practice. The high pressure environment in projects leaves little time for formal training or informal learning, or even to capture knowledge in documents. To  a large extent the hosting organisations are to blame: Grabher suggests that this paradox is a consequence of the lack of organizational redundancy in project-based organizing.

I’ll end this section with the observation that the social dimension of projects is often neglected.  Projects are often hindered by organizational politics and inertia.  Further, a large number of  projects fail because of varying perceptions of project goals and the rationale behind them. Although it seems obvious that a project should not proceed unless all stakeholders have a shared understanding of objectives and the reasons for them, it is surprising how many projects drift along without it.   Many project planners neglect this issue, and it invariably comes back to bite them.

Conclusions

In the conclusion to the paper, Grabher states:

The formation and operation of projects essentially relies on a societal infrastructure which is built on and around networks, localities, institutions and firms. Relations between temporary and permanent systems are not a matter of straightforward substitution but have to be regarded in terms of interdependence. ‘Cool’ projects, indeed, rely on ‘boring’ institutions

This is unarguable, but it should also be kept in mind that projects are often subject to negative organisational influences which can slow them down, or even kill them altogether (which is perhaps why those early defence projects were set up as near-autonomous initiatives). So  although  it is true that projects  are made possible and sustained by the organisations they’re embedded in,  they are sometimes  hindered by those very organisations .

To sum up in a line:    Projects depend on organisations not only for material and human resources, but also draw sustenance from (and are affected by)  the  social environment and culture that exists within those organisations.

Redefining project success

with 16 comments

Introduction

When can one say that a project has  failed?  Although the answer to this question depends on the criteria used to measure project success, most project managers would not think the question (or its answer) could raise much controversy.  In this post I discuss why the issue of project success is more ambiguous (and contentious!) than seems at first sight.

The crux of the issue lies in the observation that managers and programmers often use different criteria to judge project outcomes. As Robert Glass states in his wonderful book on facts and fallacies of  software engineering:

There is a disconnect between management and their programmers. In one research study of a project that failed to meet its estimates and was seen by its management as a failure, the technical participants saw it as the most successful project they had ever worked on.

I’ll expand on this statement, drawing from Glass’ book, the research study he mentions, and personal experience.

The disconnect

Much of  Glass’ discussion is based on a paper by Kurt Linberg entitled, Software Developer Perceptions about Software Project Failures: A Case Study.  The  paper details a lot of information about the project, its context, reporting structures and the composition of the team.  Although this is important from the perspective of a research study, it is more relevant to look at how the various project stakeholders perceived the project.

First a few words about the project that Linberg studied. The project was aimed at creating a software-driven medical device. The management-approved schedule for the project ran over 14 months, but the project actually took 27 months to complete.  Further, the software development portion of the product was originally budgeted at a little over a million dollar but ended up costing more than four million! In view of this there’s little surprise that management deemed the project a failure.

What about the programmers? Linberg interviewed/surveyed many of the developers who were a part of the team. One of the questions he asked was: what was the most successful project that you have worked on, and why? Surprisingly, more than 50% of the team answered that the project described above was the most successful one they had ever worked on! The reasons they gave were revealing:

  1. The project was a technical challenge.
  2. The product worked as intended.
  3. The team was small and high performing.

The following quote from one of the team members says it all:

[the project] was the most successful because of what we accomplished. The code was more solid than any I have ever written. The verification testing was extensive. All the other places that I’ve worked relied on the software developer doing the testing. I always thought that I could have done more testing. I also really enjoyed the people I worked with. I also think that it was the best-managed project of any that I’ve experienced. Also, the product was well designed. I did not see the scope creep that is so prevalent on other projects. I never felt pressure from schedule. I would ask the project manager and technical lead if we were in trouble. They would be cool, calm and collected. They emphasized the importance of doing the best job we could do. We were able to focus our energy on the tasks at hand!

So, we see a complete disconnect: according to (majority of) the developers the project was success because it delivered a quality product, but by management metrics of budget and time it was a failure!

Bridging the gap

To bridge the disconnect between management and developers, Linberg asked the team members why the project was late and over-budget. The following reasons were offered:

  1. Unrealistic schedule and expectations.
  2. Poor understanding of scope (underestimating the effort required)
  3. Lack of resources

One of the team members summarised the schedule situation as follows:

Unfortunately, I believe program management established over-ambitious and incorrect expectations during early meetings with the executives. In reality, we had no idea how long it would take to develop this system. It is a real tribute to this team that they pulled together and developed an excellent product.

Another had this to say about the poor understanding of scope / lack of resources:

We ignored history. Although we didn’t have experience estimating this type of development, there are groups of people in our building that have been working these types of applications for years. They typically have 6 to 10 people working on their applications. How could management not see that assigning one or two people and constraining the schedule would be unrealistic?

Yet, the team thought that the project was managed quite well. Quoting from the paper:

Compared with other projects, the software developers believed the project management on the project was between average and best. In two cases, the software developers said it was the best that they had experienced.  One of the software developers that gave high marks for the project management stated that the project manager had good technical skills and could manage time lines.

In fact, one of the developers said:

The project manager, team lead, and program manager each provided leadership. The leadership worked very well together. The program manager knew his strengths and his boundaries. He left the software and firmware management to the software project manager. The software project manager was not threatened and left technical decisions to the team lead. We were always told of changes before they happened. The leadership was the best that I have experienced in my 14 years.

The overall impression that Linberg conveys is that the project was well-managed.   So where is the problem?

The programmers felt that upper management was largely responsible for the disconnect. This makes sense: front line managers rarely have the clout to set overall allocations of time and resources. They have to work within the parameters set by executives.

Based on this, Linberg proposes a new definition of software project success, which is based on industry averages rather than arbitrary metrics.

Completed Cancelled
Failure Does not meet customer expectations Not learning anything that can be applied to future projects
Low Success Below average cost, effort, and schedule performance compared to industry AND meeting quality expectations Some learning can be applied to future projects
Success Average cost, effort, and schedule performance compared to industry AND meeting quality expectations Some learning can be applied to future projects and some artifacts can be used on future projects.
High Success Better than average cost, effort, and schedule performance compared to industry AND meeting quality expectations Substantial learning can be applied to future projects and a large number of artifacts used.
Exceptional Success Meeting all quality, cost, effort and schedule expectations. A Cancelled project cannot be considered an exceptional success.

This definition begins to bridge the chasm between developer and senior management perceptions of project success. It does so by broadening the definition of success. The current project management-driven definition is too narrow because it takes only one perspective into account – the perspective of those who hold the purse-strings.  Such a definition cannot work because it completely ignores the viewpoint of those who know software development best: the developers.

Conclusion

Linberg’s case study highlights the fact that there can be a huge gap between how developers and managers view project outcomes. Interestingly, the factors that cause the gap are usually present from the start of the project: unrealistic schedules, poorly understood scope and lack of adequate resources. One could hypothesise that most of the projects that “fail” do so for one or more of these reasons. Also interesting is the fact that even such “failed” projects can be deemed successful in the eyes of developers because they see things from a different perspective: what was learnt and what can be used again. Seen in this light, a project that goes over-time and over-budget can be successful because the experience gained can be used on future projects.

I’ll end with a story from experience. Some years ago I was a part of a team developing a product for a telecom software company. The product was ambitious in scope, and the timeline and budget tight. Nevertheless,  the team looked forward to the challenge of designing and building a product from scratch. The architect, project manager  and several members of the development team (excluding me) had years of experience in building software for the telecom industry. The product specs were developed by domain experts.  Management’s initial intentions were good: they assured the dev team that they (management) knew that it would take time to develop a solid product from the ground up. To reduce risk it was decided to build the product using incremental approach with monthly deliverables.

Progress was slow but steady.  Unfortunately, as the product started taking shape, business reality intruded: the sales guys who had been working hard  had not been able to convince anyone to sign up.  Funds began to dry up and management responded by cracking the whip.  Predictably, the project started to unravel and was canned a few months later.  Officially the project was deemed a failure, yet many  on the dev team believed that  a) it was one of the best projects they had worked on b) they had learnt a lot and  c) what they had learnt wouldn’t go waste.

Was the project a failure? I suppose the answer is that it failed from a purely  commercial standpoint. From a broader perspective, however, it fits quite neatly into Linberg’s definition of a successful cancelled project.

Perhaps it is time to rethink the definition of project success.

Written by K

April 16, 2010 at 5:50 am

On the relationship between systems and the organisations that build them

with one comment

Introduction

System design is a creative activity, but one that is subject to a variety of constraints. Many of these constraints are obvious: for example, when tasked with designing a new software product, a team might be asked to work within a budget or use a particular technology. These constraints place boundaries on the design activity; they force designers to work within parameters specified by the constraints. But there are other less obvious  constraints too. In a paper entitled, How Do Committees Invent, published in 1968, Melvin Conway described a notion that is now called Conway’s Law: An organisation which designs a system will inevitably produce a design that mirrors the organisation’s communication structure. This post is a summary of the key points of the paper.

Conway begins the paper with the observation that the system design is an activity that involves specifying how a system will be built using a number of diverse parts.  Many elements of the act of design are similar, regardless of the nature of the system –be it software or a shopping mall. The objective of a design team or organisation is to produce a specification or blueprint based on which the system can be built.

Much of design work is about making choices. Conway points out that these choices may be more than design decisions:

Most design activity requires continually making choices. Many of these choices may be more than design decisions; they may also be personal decisions the designer makes about his own future. As we shall see later, the incentives which exist in a conventional management environment can motivate choices which subvert the intent of the sponsor.

The paper is essentially an elaboration and justification of this claim.

Pre-design work

The preliminary stages of design work are more about organizing than design itself.  First, the boundaries have to be understood so that the solution space can be defined. Second, the high-level structure of the system has to be explored so that work can be subdivided in a sensible way within the organisation that’s doing the design. This latter point is the crux of Conway’s argument:

…the very act of organizing a design team means that certain design decisions have already been made, explicitly or otherwise. Given any design team organization, there is a class of design alternatives which cannot be effectively pursued by such an organization because the necessary communication paths do not exist. Therefore, there is no such thing as a design group which is both organized and unbiased.

There are a couple of important points here:

  1. The act of delegating design tasks narrows the scope of design options that can be pursued.
  2. Once tasks are delegated to groups, coordination (via communication) between these groups is the only way that the work can be integrated.

Further, once established, it is very hard to change design idea (or a project team, for that matter).

The system mirrors the organisation

Most systems of any significance are composed of several subsystems that communicate with each other via interfaces. According to Conway, these elements (italicized in the previous sentence) have a correspondence with the organisation that designs the system.  How so? Well, every subsystem is designed by a group within the organisation (call it a design group).  If two subsystems are to communicate interact with each other, the two groups responsible for their design must communicate with each other (to negotiate the interface design).  If subsystems don’t interact, no communication is necessary.  What we see from this argument is that the communication between subsystems roughly mirrors the communication paths within the organisation.

As any system designer knows: given a set of requirements, there are a number of designs that can satisfy them.  If the argument of the previous paragraph is true then the structure of the design organisation (or project team)  influences the choice that is made.

Managing systems design

Conway points out that large system design efforts spin out of control more often than those for small systems. He surmises that this happens when the design becomes too complex for one person (or a small, tightly-knit group of people). A standard management reaction to such a situation is to delegate the design of component to sub-teams. Why? Well here’s what Conway says:

A manager knows that he will be vulnerable to the charge of mismanagement if he misses his schedule without having applied all his resources. This knowledge creates a strong pressure on the initial designer who might prefer to wrestle with the design rather than fragment it by delegation, but he is made to feel that the cost of risk is too high to take the chance. Therefore, he is forced to delegate in order to bring more resources to bear.

A major fallacy in this line of thinking is that more resources means that work gets done faster. It is well known that this isn’t so – at least as far as software systems development is concerned. Conway points out that politics also contributes to this effect. In most organisations, managerial status is tied to team size and project budgets. This provides an incentive to managers to expand their organisations (i.e. project teams), making design delegation almost inevitable.

Large teams have a large number of communication paths between their members. Specifically, in a team consisting of N people, there are N(N-1)/2 possible communication paths – each person can communicate with N-1 people making N(N-1), but this has to be halved because paths between every two individuals are counted twice. Organisations deal with this by restricting communication paths to hierarchical management structures.  Because communication paths mirror organizational structures, it is almost inevitable that system designs will mirror them.

Conclusion

The main implication of Conway’s thesis is that a project team (or any organisation) charged with designing a system should be structured in a way that suits the communication needs of the system. For example, sub-teams involved in designing related subsystems should have many more communication channels than those that design independent components.  Further, system design is inherently complex, and the first design is almost never the final one.  A consequence that flows from this is that design organisations should be flexible because they’ll almost always need to be reorganized.

In the end it is less about the number of people on a team than the communication between them. As Conway mentions in the last two lines of his paper:

There is need for a philosophy of system design management which is not based on the assumption that adding manpower simply adds to productivity. The development of such a philosophy promises to unearth basic questions about value of resources and techniques of communication which will need to be answered before our system-building technology can proceed with confidence.

This is as true now as it was forty-odd years ago.

Written by K

March 23, 2010 at 10:24 pm