Archive for the ‘Organizations’ Category
From the coalface: an essay on the early history of sociotechnical systems
The story of sociotechnical systems began a little over half a century ago, in a somewhat unlikely setting: the coalfields of Yorkshire.
The British coal industry had just been nationalised and new mechanised mining methods were being introduced in the mines. It was thought that nationalisation would sort out the chronic labour-management issues and mechanisation would address the issue of falling productivity.
But things weren’t going as planned. In the words of Eric Trist, one of the founders of the Tavistock Institute:
…the newly nationalized industry was not doing well. Productivity failed to increase in step with increases in mechanization. Men were leaving the mines in large numbers for more attractive opportunities in the factory world. Among those who remained, absenteeism averaged 20%. Labour disputes were frequent despite improved conditions of employment. – excerpted from, The evolution of Socio-technical systems – a conceptual framework and an action research program, E. Trist (1980)
Trist and his colleagues were asked by the National Coal Board to come in and help. To this end, they did a comparative study of two mines that were similar except that one had high productivity and morale whereas the other suffered from low performance and had major labour issues.
Their job was far from easy: they were not welcome at the coalface because workers associated them with management and the Board.
Trist recounts that around the time the study started, there were a number of postgraduate fellows at the Tavistock Institute. One of them, Ken Bamforth, knew the coal industry well as he had been a miner himself. Postgraduate fellows who had worked in the mines were encouraged to visit their old workplaces after a year and write up their impressions, focusing on things that had changed since they had worked there. After one such visit, Bamforth reported back with news of a workplace innovation that had occurred at a newly opened seam at Haighmoor. Among other things, morale and productivity at this particular seam was high compared to other similar ones. The team’s way of working was entirely novel, a world away from the hierarchically organised set up that was standard in most mechanised mines at the time. In Trist’s words:
The work organization of the new seam was, to us, a novel phenomenon consisting of a set of relatively autonomous groups interchanging roles and shifts and regulating their affairs with a minimum of supervision. Cooperation between task groups was everywhere in evidence; personal commitment was obvious, absenteeism low, accidents infrequent, productivity high. The contrast was large between the atmosphere and arrangements on these faces and those in the conventional areas of the pit, where the negative features characteristic of the industry were glaringly apparent. Excerpted from the paper referenced above.
To appreciate the radical nature of practices at this seam, one needs to understand the backdrop against which they occurred. To this end, it is helpful to compare the mechanised work practices introduced in the post-war years with the older ones from the pre-mechanised era of mining.
In the days before mines were mechanised, miners would typically organise themselves into workgroups of six miners, who would cover three work shifts in teams of two. Each miner was able to do pretty much any job at the seam and so could pick up where his work-mates from the previous shift had left off. This was necessary in order to ensure continuity of work between shifts. The group negotiated the price of their mined coal directly with management and the amount received was shared equally amongst all members of the group.
This mode of working required strong cooperation and trust within the group, of course. However, as workgroups were reorganised from time to time due to attrition or other reasons, individual miners understood the importance of maintaining their individual reputations as reliable and trustworthy workmates. It was important to get into a good workgroup because such groups were more likely to get more productive seams to work on. Seams were assigned by bargaining, which was typically the job of the senior miner on the group. There was considerable competition for the best seams, but this was generally kept within bounds of civility via informal rules and rituals.
This traditional way of working could not survive mechanisation. For one, mechanised mines encouraged specialisation because they were organised like assembly lines, with clearly defined job roles each with different responsibilities and pay scales. Moreover, workers in a shift would perform only part of the extraction process leaving those from subsequent shifts to continue where work was left off.
As miners were paid by the job they did rather than the amount of coal they produced, no single group had end-to-end responsibility for the product. Delays due to unexpected events tended to get compounded as no one felt the need to make up time. As a result, it would often happen that work that was planned for a shift would not be completed. This meant that the next shift (which could well be composed of a group with completely different skills) could not or would not start their work because they did not see it as their job to finish the work of the earlier shift. Unsurprisingly, blame shifting and scapegoating was rife.
From a supervisor’s point of view, it was difficult to maintain the same level of oversight and control in underground mining work as was possible in an assembly line. The environment underground is simply not conducive to close supervision and is also more uncertain in that it is prone to unexpected events. Bureaucratic organisational structures are completely unsuited to dealing with these because decision-makers are too far removed from the coalface (literally!). This is perhaps the most important insight to come out of the Tavistock coal mining studies.
As Claudio Ciborra puts it in his classic book on teams:
Since the production process at any seam was much more prone to disorganisation than due to uncertainty and complexity of underground conditions, any ‘bureaucratic’ allocation of jobs could be easily disrupted. Coping with emergencies and coping with coping became part of worker’s and supervisors’ everyday activities. These activities would lead to stress, conflict and low productivity because they continually clashed with the technological arrangements and the way they were planned and subdivided around them.
Thus we see that the new assembly-line bureaucracy inspired work organisation was totally unsuited to the work environment because there was no end-to-end responsibility, and decision making was far removed from the action. In contrast, the traditional workgroup of six was able to deal with uncertainties and complexities of underground work because team members had a strong sense of responsibility for the performance of the team as a whole. Moreover, teams were uniquely placed to deal with unexpected events because they were actually living them as they occurred and could therefore decide on the best way to deal with them.
What Bamforth found at the Haighmoor seam was that it was possible to recapture the spirit of the old ways of working by adapting these to the larger specialised groups that were necessary in the mechanised mines. As Ciborra describes it in his book:
The new form of work organisation features forty one men who allocate themselves to tasks and shifts. Although tasks and shifts those of the conventional mechanised system, management and supervisors do not monitor, enforce and reward single task executions. The composite group takes over some of the managerial tasks, as it had in the pre-mechanised marrow group, such as the selection of group members and the informal monitoring of work…Cycle completion, not task execution becomes a common goal that allows for mutual learning and support…There is basic wage and a bonus linked to the overall productivity of the group throughout the whole cycle rather than a shift. The competition between shifts that plagued the conventional mechanised method is effectively eliminated…
Bamforth and Trist’s studies on Haighmoor convinced them that there were viable (and better!) alternatives to those that were typical of mid to late 20th century work places. Their work led them to the insight that the best work arrangements come out of seeking a match between technical and social elements of the modern day workplace, and thus was born the notion of sociotechnical systems.
Ever since the assembly-line management philosophies of Taylor and Ford, there has been an increasing trend towards division of labour, bureaucratisation and mechanisation / automation of work processes. Despite the early work of the Tavistock school and others who followed, this trend continues to dominate management practice, arguably even more so in recent years. The Haighmoor innovation described above was one of the earliest demonstrations that there is a better way. This message has since been echoed by many academics and thinkers, but remains largely under-appreciated or ignored by professional managers who have little idea – or have completely forgotten – what it is like to work at the coalface.
TOGAF or not TOGAF… but is that the question?
“The ‘Holy Grail’ of effective collaboration is creating shared understanding, which is a precursor to shared commitment.” – Jeff Conklin.
“Without context, words and actions have no meaning at all.” – Gregory Bateson.
I spent much of last week attending a class on the TOGAF Enterprise Architecture (EA) framework. Prior experience with IT frameworks such as PMBOK and ITIL had taught me that much depends on the instructor – a good one can make the material come alive whereas a not-so-good one can make it an experience akin to watching grass grow. I needn’t have worried: the instructor was superb, and my classmates, all of whom are experienced IT professionals / architects, livened up the proceedings through comments and discussions both in class and outside it. All in all, it was a thoroughly enjoyable and educative experience, something I cannot say for many of the professional courses I have attended.
One of the things about that struck me about TOGAF is the way in which the components of the framework hang together to make a coherent whole (see the introductory chapter of the framework for an overview). To be sure, there is a lot of detail within those components, but there is a certain abstract elegance – dare I say, beauty – to the framework.
That said TOGAF is (almost) entirely silent on the following question which I addressed in a post late last year:
Why is Enterprise Architecture so hard to get right?
Many answers have been offered. Here are some, extracted from articles published by IT vendors and consultancies:
- Lack of sponsorship
- Not engaging the business
- Inadequate communication
- Insensitivity to culture / policing mentality
- Clinging to a particular tool or framework
- Building an ivory tower
- Wrong choice of architect
(Note: the above points are taken from this article and this one)
It is interesting that the first four issues listed are related to the fact that different stakeholders in an organization have vastly different perspectives on what an enterprise architecture initiative should achieve. This lack of shared understanding is what makes enterprise architecture a socially complex problem rather than a technically difficult one. As Jeff Conklin points out in this article, problems that are technically complex will usually have a solution that will be acceptable to all stakeholders, whereas socially complex problems will not. Sending a spacecraft to Mars is an example of the former whereas an organization-wide ERP (or EA!) project or (on a global scale) climate change are instances of the latter.
Interestingly, even the fifth and sixth points in the list above – framework dogma and retreating to an ivory tower – are usually consequences of the inability to manage social complexity. Indeed, that is precisely the point made in the final item in the list: enterprise architects are usually selected for their technical skills rather than their ability to deal with ambiguities that are characteristic of social complexity.
TOGAF offers enterprise architects a wealth of tools to manage technical complexity. These need to be complemented by a suite of techniques to reconcile worldviews of different stakeholder groups. Some examples of such techniques are Soft Systems Methodology, Polarity Management, and Dialogue Mapping. I won’t go into details of these here, but if you’re interested, please have a look at my posts entitled, The Approach – a dialogue mapping story and The dilemmas of enterprise IT for brief introductions to the latter two techniques via IT-based examples.
<Advertisement > Better yet, you could check out Chapter 9 of my book for a crash course on Soft Systems Methodology and Polarity Management and Dialogue Mapping, and the chapters thereafter for a deep dive into Dialogue Mapping </Advertisement>.
Apart from social complexity, there is the problem of context – the circumstances that shape the unique culture and features of an organization. As I mentioned in my introductory remarks, the framework is abstract – it applies to an ideal organization in which things can be done by the book. But such an organization does not exist! Aside from unique people-related and political issues, all organisations have their own quirks and unique features that distinguish them from other organisations, even within the same domain. Despite superficial resemblances, no two pharmaceutical companies are alike. Indeed, the differences are the whole point because they are what make a particular organization what it is. To paraphrase the words of the anthropologist, Gregory Bateson, the differences are what make a difference.
Some may argue that the framework acknowledges this and encourages, even exhorts, people to tailor the framework to their needs. Sure, the word “tailor” and its variants appear almost 700 times in the version 9.1 of the standard but, once again, there is no advice offered on how this tailoring should be done. And one can well understand why: it is impossible to offer any sensible advice if one doesn’t know the specifics of the organization, which includes its context.
On a related note, the TOGAF framework acknowledges that there is a hierarchy of architectures ranging from the general (foundation) to the specific (organization). However despite the acknowledgement of diversity, in practice TOGAF tends to focus on similarities between organisations. Most of the prescribed building blocks and processes are based on assumed commonalities between the structures and processes in different organisations. My point is that, although similarities are important, architects need to focus on differences. These could be differences between the organization they are working in and the TOGAF ideal, or even between their current organization and others that they have worked with in the past (and this is where experience comes in really handy). Cataloguing and understanding these unique features – the differences that make a difference – draws attention to precisely those issues that can cause heartburn and sleepless nights later.
I have often heard arguments along the lines of “80% of what we do follows a standard process, so it should be easy for us to standardize on a framework.” These are famous last words, because some of the 20% that is different is what makes your organization unique, and is therefore worthy of attention. You might as well accept this upfront so that you get a realistic picture of the challenges early in the game.
To sum up, frameworks like TOGAF are abstractions based on an ideal organization; they gloss over social complexity and the unique context of individual organisations. So, questions such as the one posed in the title of this post are akin to the pseudo-choice between Coke and Pepsi, for the real issue is something else altogether. As Tom Graves tells us in his wonderful blog and book, the enterprise is a story rather than a structure, and its architecture an ongoing sociotechnical drama.
Conditions over causes: towards an emergent approach to building high-performance teams
Introduction
Much of the work that goes on in organisations is done by groups of people who work together in order to achieve shared objectives. Given this, it is no surprise that researchers have expended a great deal of effort in building theories about how teams work. However, as Richard Hackman noted in this paper, more than 70 years of research (of ever-increasing sophistication) has not resulted in a true understanding of the factors that give rise to high-performing teams. The main reason for this failure is that:
“…groups are social systems. They redefine objective reality, they create new realities (both for their members and in their system contexts), and they evolve their own purposes and strategies for pursuing those purposes. Groups are not mere assemblies of multiple cause–effect relationships; instead, they exhibit emergent and dynamic properties that are not well captured by standard causal models.”
Hackman had a particular interest in leadership as a causal factor in team performance. One of the things he established is that leadership matters a whole lot less than is believed…or, more correctly, it matters for reasons that are not immediately obvious. As he noted:
“…60 per cent of the difference in how well a group eventually does is determined by the quality of the condition-setting pre-work the leader does. 30 per cent is determined by how the initial launch of the group goes. And only 10 per cent is determined by what the leader does after the group is already underway with its work. This view stands in stark contrast to popular images of group leadership—the conductor waving a baton throughout a musical performance or an athletic coach shouting instructions from the sidelines during a game.”
Although the numbers quoted above can be contested, the fact is that as far as team performance is concerned, conditions matter more than the quality of leadership. In this post, I draw on Hackman’s paper as well as my work (done in collaboration with Paul Culmsee) to argue that the real work of leaders is not to lead (in the conventional sense of the word) but to create the conditions in which teams can thrive.
The fundamental attribution error
Poor performance of teams is often attributed to a failure of leadership. A common example of this is when the coach of a sporting team is fired after a below par season. On the flip side, CxOs can earn big-buck dollar bonuses when their companies make or exceed their financial targets because they are seen as being directly responsible for the result.
Attributing the blame or credit for the failure or success of a team to a specific individual is called the leadership attribution error. Hackman suggested that this error is a manifestation of a human tendency to assign greater causal priority to factors that are more visible than those that are not: leaders tend to be in the limelight more than their teams and are therefore seen as being responsible for their teams’ successes and failures.
This leader-as-hero (or villain!) perspective has fueled major research efforts aimed at pinning down those elusive leadership skills and qualities that can magically transform teams into super-performing ensembles. This has been accompanied by a burgeoning industry of executive training programmes to impart these “scientifically proven” skills to masses of managers. These programmes, often clothed in the doublespeak of organisation culture, are but subtle methods of control that serve to establish directive approaches to leadership. Such methods rarely (if ever) result in high-performing organisations or teams.
An alternate approach to understanding team performance
The failure to find direct causal relationships between such factors and team performance led Hackman to propose a perspective that focuses on structural conditions instead. The basic idea in this alternate approach is to focus on the organisational and social conditions that enable the team to perform well.
This notion of conditions over causes is relevant in other related areas too. Here are a couple of examples:
- Innovation: Most attempts to foster innovation focus on exhorting people to be creative and/or instituting innovation training programmes (causal approach). Such approaches usually result in innovation of an incremental kind at best. Instead, establishing a low pressure environment that enables people to think for themselves and follow-up on their ideas without fear of failure generally meets with more success (structural approach).
- Collaboration: Organisations generally recognise the importance of collaboration. Yet, they attempt to foster in the worst possible way: via the establishment of cross-functional teams without clear mandates or goals and/or forced team-building exercises that have the opposite effect to the one intended (causal approach). The alternate approach is to simplify reporting lines, encourage open communication across departments and generally make it easy for people from different specialisations to work together in informal groups (structural approach). A particularly vexing intra-departmental separation that I have come across recently is the artificial division of responsibilities between information systems development and delivery. Such a separation results in reduced collaboration and increased finger pointing.
That said, let’s take a look at Hackman’s advice on how to create an environment conducive to teamwork. Hackman identified the following five conditions that tend to correlate well with improved team performance:
- The group must be a real team– i.e. it must have clear boundaries (clarity as to who is a member and who isn’t), interdependence (the performance of every individual in the team must in some way depend on others in the team) and stability (membership of the team should be stable over time).
- Compelling direction– the team must have a goal that is clear and worth pursuing. Moreover, and this is important, the team must be allowed to determine how the goal is to be achieved – the end should be prescribed, not the means.
- The structure must enable teamwork– The team should be structured in a way that allows members to work together. This consists of a couple of factors: 1) The team must be of the right size – as small and diverse as possible (large, homogenous teams are found to be ineffective), and 2) There must be clear norms of conduct. Note that Hackman lists these two as separate points in his paper.
- Supportive organizational context– the team must have the organisational resources that enable it to carry out its work. For example, access to the information needed for the team to carry out its work and access to technical and subject matter experts. In addition, there should be a transparent reward system that provides recognition for good work.
- Coaching– the team must have access to a mentor or coach who understands and has the confidence of the team. Apart from helping team members tide over difficult situations, a good coach should be able to help them navigate organizational politics and identify emerging threats and opportunities that may not be obvious to them.
To reiterate, these are structural rather than causal factors in that they do not enhance team performance directly. Instead, when present, they tend to encourage behaviours that enhance team performance and suppress those that don’t.
Another interesting point is that some of these factors are more important than others. For example, Ruth Wageman found that team design (the constitution and structure of the team) is about four times more important than coaching in affecting the team’s ability to manage itself and forty times as powerful in affecting team performance (see this paper for details). Although the numbers should not be taken at face value, Wageman’s claim reiterates the main theme of this article: that structural factors matter more than causal ones.
The notion of a holding environment
One of the things I noticed when I first read Hackman’s approach is that it has some similarities to the one that Paul and I advocated in our book, The Heretic’s Guide to Best Practices.
The Heretic’s Guide is largely about collaborative approaches to managing (as opposed to solving!) complex problems in organisations. Our claim is that the most intractable problems in organisations are consequences of social rather than technical issues. For example, the problem of determining the “right” strategy for an organisation cannot be settled on objective grounds because the individuals involved will have diverse opinions on what the organisation’s focus should be. The process of arriving at a consensual strategy is, therefore, more a matter of dealing with this diversity than reaching an objectively right outcome. In other words, it is largely about achieving a common view of what the strategy should be and then building a shared commitment to executing it.
The key point is that there is no set process for achieving a shared understanding of a problem. Rather, one needs to have the right environment (structure!) in which contentious issues can be discussed openly without fear. In our book we used the term holding environment to describe a safe space in which such open dialogue can take place.
The theory of communicative rationality formulated by the German philosopher, Juergen Habermas, outlines the norms that operate within a holding environment. It would be too long a detour to discuss Habermas’ work in any detail – see this paper or chapter 7 of our book to find out more. What is important to note is that an ideal holding environment has the following norms:
- Inclusion
- Autonomy
- Empathy
- Power neutrality
- Transparency
Problem is, some of these are easier to achieve than others. Inclusion, autonomy and power neutrality can be encouraged by putting in place appropriate organisational structures and rules. Empathy and transparency, however, are typically up to the individual. Nevertheless, conditions that enable the former will also encourage (though not guarantee) the latter.
In our book we discuss how such a holding environment can be approximated in multi-organisational settings such as large projects. It would take me too far afield to get into specifics of the approach here. The point I wish to make, however, is that the notion of a holding environment is in line with Hackman’s thoughts on the importance of environmental or structural factors.
In closing
Some will argue that this article merely sets up and tears down a straw man, and that modern managers are well aware of the pitfalls of a directive approach to leading teams. Granted, much has been written about the importance of setting the right conditions (such as autonomy)…and it is possible that many managers are aware of it too. The point I would make is that this awareness, if it exists at all, has not been translated into action often enough. As a result, the gap between the rhetoric and reality of leadership remains as wide as ever – managers talk the talk of leadership, but do not walk it.
Perhaps this is because many (most?) managers are reluctant let go the reins of control when they know they will be held responsible if things were to go belly-up. The few who manage to overcome their fears know that it requires the ability to trust others, as well as the courage and integrity to absorb the blame when things go wrong (as they inevitably will from time to time). These all too rare qualities are essential for the approach described here to truly take root and flourish. In conclusion, I think it is fair to say that the biggest challenges associated with building high-performance teams are ethical rather than technical ones.
Further Reading
Don’t miss Paul Culmsee’s entertaining and informative posts on the conditions over causes approach in enterprise IT and project management.
Eight to Late 
