Archive for the ‘Decision Making’ Category
Six heresies for business intelligence
What is business intelligence?
I recently asked a few acquaintances to answer this question without referring to that great single point of truth in the cloud. They duly came up with a variety of responses ranging from data warehousing and the names of specific business intelligence tools to particular functions such as reporting or decision support.
After receiving their responses, I did what I asked my respondents not to: I googled the term. Here are a few samples of what I found:
According to CIO magazine, Business intelligence is an umbrella term that refers to a variety of software applications used to analyze an organization’s raw data.
Wikipedia, on the other hand, tells us that BI is a set of theories, methodologies, architectures, and technologies that transform raw data into meaningful and useful information for business purposes.
Finally, Webopedia, tell us that BI [refers to] the tools and systems that play a key role in the strategic planning process of the corporation.
What’s interesting about the above responses and definitions is that they focus largely on processes and methodologies or tools and techniques. Now, without downplaying the importance of either, I think that many of the problems of business intelligence practice come from taking a perspective that is overly focused on methodology and technique. In this post, I attempt to broaden this perspective by making some potentially controversial statements –or heresies – that challenge this view. My aim is not so much to criticize current practice as to encourage – or provoke – business intelligence professionals to take a closer look at some of the assumptions underlie their practices.
The heresies
Without further ado, here are my six heresies for business intelligence practice (in no particular order).
A single point of truth is a mirage
Many organisations embark on ambitious programs to build enterprise data warehouses – unified data repositories that serve as a single source of truth for all business-relevant data. Leaving aside the technical and business issues associated with establishing definitive data sources and harmonizing data, there is the more fundamental question of what is meant by truth.
The most commonly accepted notion of truth is that information (or data in a particular context) is true if it describes something as it actually is. A major issue with this viewpoint is that data (or information) can never fully describe a real-world object or event. For example, when a sales rep records a customer call, he or she notes down only what is required by the customer management system. Other data that may well be more important is not captured or is relegated to a “Notes” or “Comments” field that is rarely if ever searched or accessed. Indeed, data represents only a fraction of the truth, however one chooses to define it – more on this below.
Some might say that it is naïve to expect our databases to capture all aspects of reality, and that what is needed is a broad consensus between all relevant stakeholders as to what constitutes the truth. The problem with this is that such a consensus is often achieved by means that are not democratic. For example, a KPI definition chosen by a manager may be hotly contested by an employee. Nevertheless, the employee has to accept it because that is the way (many) organisations work. Another significant issue is that the notion of relevant stakeholders is itself problematic because it is often difficult to come up with clear criterion by which to define relevance.
There are other ways to approach the notion of truth: for example, one might say that a piece of data is true as long as it is practically useful to deem it so. Such a viewpoint, though common, is flawed because utility is in the eye of the beholder: a sales manager may think it useful to believe a particular KPI whereas a sales rep might disagree (particularly if the KPI portrays the rep in a bad light!).
These varied interpretations of what constitute a truth have implications for the notion of a single point of truth. For one, the various interpretations are incommensurate – they cannot be judged by the same standard. Further, different people may interpret the same piece of data differently. This is something that BI professionals have likely come across – say when attempting to come up with a harmonized definition for a customer record.
In short: the notion of a single point of truth is problematic because there is a great deal of ambiguity about what constitutes a truth.
There is no such thing as raw data
In his book, Memory Practices in the Sciences, Geoffrey Bowker wrote, “Raw data is both an oxymoron and a bad idea; to the contrary, data should be cooked with care.” I love this quote because it tells a great truth (!) about so-called “raw” data.
To elaborate: raw data is never unprocessed. Firstly, the data collector always makes a choice as to what data will be collected and what will not. So in this sense, data already has meaning imposed on it. Second, and perhaps more important, the method of collection affects the data. For example, responses to a survey depend on how the questions are framed and how the survey itself is carried out (anonymous, face-to-face etc.). This is also true for more “objective” data such as costs and expenses. In both cases, the actual numbers depend on specific accounting practices used in the organization. So, raw data is an oxymoron because data is never raw, and as Bowker tells us, we need to ensure that the filters we apply and the methods of collection we use are such that the resulting data is “cooked with care.”
In short: data is never raw, it is always “cooked.”
There are no best practices for business intelligence, only appropriate ones
Many software shops and consultancies devise frameworks and methodologies for business intelligence which they claim are based on best or proven practices. However, those who swallow that line and attempt to implement the practices often find that the results obtained are far from best.
I have discussed the shortcomings of best practices in a general context in an earlier article, and (at greater length) in my book. A problem with best practice approaches is that they assume a universal yardstick of what is best. As a corollary, this also suggest that practices can be transplanted from one organization to another in a wholesale manner, without extensive customisation. This overlooks the fact that organisations are unique, and what works in one may not work in another.
A deeper issue is that much of the knowledge pertaining to best practices is tacit – that is, it cannot be codified in written form. Indeed, what differentiates good business intelligence developers or architects from great ones is not what they learnt from a textbook (or in a training course), but how they actually practice their craft. These consist of things that they do instinctively and would find hard to put into words.
So, instead of looking to import best practices from your favourite vendor, it is better to focus on understanding what goes on in your environment. A critical examination of your environment and processes will reveal opportunities for improvement. These incremental improvements will cumulatively add up to your very own, customized “best practices.”
In short: develop your own business intelligence best practices rather than copying those peddled by “experts.”
Business intelligence does not support strategic decision-making
One of the stated aims of business intelligence systems is to support better business decision making in organisations (see the Wikipedia article, for example). It is true that business intelligence systems are perfectly adequate – even indispensable – for certain decision-making situations. Examples of these include, financial reporting (when done right!) and other operational reporting (inventory, logistics etc). These generally tend to be routine situations with clear cut decision criteria and well-defined processes – i.e. decisions that can be programmed.
In contrast, decisions pertaining to strategic matters cannot be programmed. Examples of such decisions include: dealing with an uncertain business environment, responding to a new competitor etc. The reason such decisions cannot be programmed is that they depend on a host of factors other than data and are generally made in situations that are ambiguous. Typically people use deliberative methods – i.e. methods based on argumentation – to arrive at decisions on such matters. The sad fact is that all the major business tools in the market lack support for deliberative decision-making. Check out this post for more on what can be done about this.
In short: business intelligence does not support strategic decision-making .
Big data is not the panacea it is trumpeted to be
One of the more recent trends in business intelligence is the move towards analyzing increasingly large, diverse, rapidly changing datasets – what goes under the umbrella term big data. Analysing these datasets entails the use of new technologies (e.g. Hadoop and NoSQL) as well as statistical techniques that are not familiar to many mainstream business intelligence professionals.
Much has been claimed for big data; in fact, one might say too much. In this article Tim Harford (aka the Undercover Economist) summarises the four main claims of “big data cheerleaders” as follows (the four phrases below are quoted directly from the article):
- Data analysis produces uncannily accurate results.
- Every single data point can be captured, making old statistical sampling techniques obsolete.
- It is passé to fret about what causes what, because statistical correlation tells us what we need to know.
- Scientific or statistical models aren’t needed.
The problem, as Harford points out, is that all of these claims are incorrect.
Firstly, the accuracy of the results that come out of a big data analysis depend critically on how the analysis is formulated. However, even analyses based on well-founded assumptions can get it wrong, as is illustrated in this article about Google Flu Trends.
Secondly, it is pretty obvious that it is impossible to capture every single data point (also relevant here is the discussion on raw data above – i.e. how data is selected for inclusion).
The third claim is simply absurd. The fact is detecting a correlation is not the same as understanding what is going on – a point made rather nicely by Dilbert. Enough said, I think.
Fourthly, the claim that scientific or statistical models aren’t needed is simply ill-informed. As any big data practitioner will tell you, big data analysis relies on statistics. Moreover, as mentioned earlier, a correlation-based understanding is no understanding at all – it cannot be reliably extrapolated to related situations without the help of hypotheses and (possibly tentative) models of how the phenomenon under study works.
Finally, as Danah Boyd and Kate Crawford point out in this paper , big data changes the meaning of what it means to know something….and it is highly debatable as to whether these changes are for the better. See the paper for more on this point. (Acknowledgement: the title of this post is inspired by the title of the Boyd-Crawford paper).
In short: business intelligence practitioners should not uncritically accept the pronouncements of big data evangelists and vendors.
Business intelligence has ethical implications
This heresy applies to much more than business intelligence: any human activity that affects other people has an ethical dimension. Many IT professionals tend to overlook this facet of their work because they are unaware of it – and sometimes prefer to remain so. Fact is, the decisions business intelligence professionals make with respect to usability, display, testing etc. have a potential impact on the people who use their applications. The impact may be as trivial as having to click a button or filter too many before they get their report, to something more significant, like a data error that leads to a poor business decision.
In short: business intelligence professionals ought to consider how their artefacts and applications affect their users.
In closing
This brings me to the end of my heresies for business intelligence. I suspect there will be a few practitioners who agree with me and (possibly many) others who don’t…and some of the latter may even find specific statements provocative. If so, I consider my job done, for my intent was to get business intelligence practitioners to question a few unquestioned tenets of their profession.
The consultant’s dilemma – a business fable
It felt like a homecoming. That characteristic university smell (books, spearmint gum and a hint of cologne) permeated the hallway. It brought back memories of his student days: the cut and thrust of classroom debates, all-nighters before exams and near-all-nighters at Harry’s Bar on the weekends. He was amazed at how evocative that smell was.
Rich checked the directory near the noticeboard and found that the prof was still in the same shoe-box office that he was ten years ago. He headed down the hallway wondering why the best teachers seemed to get the least desirable offices. Perhaps it was inevitable in a university system that rated grantsmanship over teaching.
It was good of the prof to see him at short notice. He had taken a chance really, calling on impulse because he had a few hours to kill before his flight home. There was too much travel in this job, but he couldn’t complain: he knew what he was getting into when he signed up. No, his problem was deeper. He no longer believed in what he did. The advice he gave and the impressive, highly polished reports he wrote for clients were useless…no, worse, they were dangerous.
He knew he was at a crossroad. Maybe, just maybe, the prof would be able to point him in the right direction.
Nevertheless, he was assailed by doubt as he approached the prof’s office. He didn’t have any right to burden the prof with his problems …he could still call and make an excuse for not showing up. Should he leave?
He shook his head. No, now that he was here he might as well at least say hello. He knocked on the door.
“Come in,” said the familiar voice.
He went in.
“Ah, Rich, it is good to see you after all these years. You’re looking well,” said the prof, getting up and shaking his hand warmly.
After a brief exchange of pleasantries, he asked Rich to take a seat.
“Just give me a minute, I’m down to the last paper in this pile,” said the prof, gesturing at a heap of term papers in front of him. “If I don’t do it now, I never will.”
“Take your time prof,” said Rich, as he sat down.
Rich cast his eye over the bookshelf behind the prof’s desk. The titles on the shelf reflected the prof’s main interest: twentieth century philosophy. A title by Habermas caught his eye.
Habermas!
Rich recalled a class in which the prof had talked about Habermas’ work on communicative rationality and its utility in making sense of ambiguous issues in management. It was in that lecture that the prof had introduced them to the evocative term that captured ambiguity in management (and other fields) so well, wicked problems.
There were many things the prof spoke of, but ambiguity and uncertainty were his overarching themes. His lectures stood in stark contrast to those of his more illustrious peers: the prof dealt with reality in all its messiness, the other guys lived in a fantasy world in which their neat models worked and things went according to plan.
Rich had learnt from the prof that philosophy was not an arcane subject, but one that held important lessons for everyone (including hotshot managers!). Much of what he learnt in that single term of philosophy had stayed with him. Indeed, it was what had brought him back to the prof’s door after all these years.
“All done,” said the prof, putting his pen down and flicking the marked paper into the pile in front of him. He looked up at Rich: “Tell you what, let’s go to the café. The air-conditioning there is so much better,” he added, somewhat apologetically.
As they walked out of the prof’s office, Rich couldn’t help but wonder why the prof stuck around in a place where he remained unrecognized and unappreciated.
—
The café was busy. Though it was only mid-afternoon, the crowd was already in Friday evening mode. Rich and the prof ordered their coffees and found a spot at the quieter end of the cafe.
After some small talk, the prof looked him and said, “Pardon my saying so, Rich, but you seem preoccupied. Is there something you want to talk about?”
“Yes, there is…well, there was, but I’m not so sure now.”
“You might as well ask,” said the prof. “My time is not billable….unlike yours.” His face crinkled into a smile that said, no offence intended.
“Well, as I mentioned when I called you this morning, I’m a management consultant with Big Consulting. By all measures, I’m doing quite well: excellent pay, good ratings from my managers and clients, promotions etc. The problem is, over the last month or so I’ve been feeling like a faker who plays on clients’ insecurities, selling them advice and solutions that are simplistic and cause more problems than they solve,” said Rich.
“Hmmm,” said the prof, “I’m curious. What triggered these thoughts after a decade in the game?”
“Well, I reckon it was an engagement that I completed a couple of months ago. I was the principal consultant for a big change management initiative at a multinational. It was my first gig as a lead consultant for a change program this size. I was responsible for managing all aspects of the engagement – right from the initial discussions with the client, to advising them on the change process and finally implementing it.” He folded his hands behind his head and leaned back in his chair as he continued, “In theory I’m supposed to offer independent advice. In reality, though, there is considerable pressure to use our standard, trademarked solutions. Have you heard of our 5 X Model of Change Management?”
“Yes, I have,” nodded the prof.
“Well, I could see that the prescriptions of 5 X would not work for that organization. But, as I said, I had no choice in the matter.”
“Uh-huh, and then?”
“As I had foreseen,” said Rich, “the change was a painful, messy one for the organization. It even hit their bottom line significantly. They are trying to cover it up, but everyone in the organization knows that the change is the real reason for the drop in earnings. Despite this, Big Consulting has emerged unscathed. A bunch of middle managers on the client’s side have taken the rap.” He shook his head ruefully. “They were asked to leave,” he said.
“That’s terrible,” said the prof, “I can well understand how you feel.”
“Yes, I should not have prescribed 5 X. It is a lemon. The question is: what should I do now?” queried Rich.
“That’s for you to decide. You can’t change the past, but you might be able to influence the future,” said the prof with a smile.
“I was hoping you could advise me.”
“I have no doubt that you have reflected on the experience. What did you conclude?”
“That I should get out of this line of work,” said Rich vehemently.
“What would that achieve?” asked the prof gently.
“Well, at least I won’t be put into such situations again. I’m not worried about finding work, I’m sure I can find a job with the Big Consulting name on my resume,” said Rich.
“That’s true,” said the prof, “but is that all there is to it? There are other things to consider. For instance, Big Consulting will continue selling snake oil. How would you feel about that?”
“Yeah, that is a problem – damned if I do, damned if I don’t,” replied Rich. “You know, when I was sitting in your office, I recalled that you had spoken about such dilemmas in one of your classes. You said that the difficulty with such wicked issues is that they cannot be decided based on facts alone, because the facts themselves are either scarce or contested…or both!”
“That’s right,” said the prof, “and this is a wicked problem of a kind that is very common, not just in professional work but also in life. Even relatively mundane issues such as whether or not to switch jobs have wicked elements. What we forget sometimes, though, is that our decisions on such matters or rather, our consequent actions, might also affect others.”
“So you’re saying I’m not the only stakeholder (if I can use that term) in my problem. Is that right?”
“That’s right, there are other people to consider,” said the prof, “but the problem is you don’t know who they are .They are all the people who will be affected in the future by the decision you make now. If you quit, Big Consulting will go on selling this solution and many more people might be adversely affected. On the other hand, if you stay, you could try to influence the future direction of Big Consulting, but that might involve some discomfort for yourself. This makes your wicked problem an ethical one. I suspect this is why you’re having a hard time going with the “quit” option.”
There was a brief silence. The prof could see that Rich was thinking things through.
“Prof, I’ve got to hand it to you,” said Rich shaking his head with a smile, “I was so absorbed by the quit/don’t quit dilemma from my personal perspective that I didn’t realize there are other angles to consider. Thanks, you’ve helped immensely. I’m not sure what I will do, but I do know that what you have just said will help me make a more considered choice. Thank you!”
“You’re welcome, Rich”
…And as he boarded his flight later that evening, Rich finally understood why the prof continued to teach at a place where he remained unrecognized and unappreciated
Objectivity and the ethical dimension of organisational decision-making
When making decisions, some people follow a structured process that involves gathering data, identifying options and analysing them to arrive at a decision. Others prefer an approach in which they seek to understand the diversity of perspectives on the issue and then attempt to synthesise a decision based on their understanding. To be sure these are stereotypes and, like all stereotypes, are somewhat exaggerated. Nevertheless, most decision-makers in organisations fall into one of these categories, at least as far as their preferred decision-making mode is concerned. Of course, people may change from one mode to another, depending on the situation. For example, a person who is predominantly an objective decision maker in his or her professional life might not be so objective when it comes to making personal decisions.
The differences between the two approaches roughly mirrors the divide between those who believe in an objective reality and those who believe that reality, or at least our perception of it, is a subjective matter. This is akin to the difference between CP Snow’s Two Cultures: the scientists and the artists. At the risk of making a gross generalisation, those who have a scientific or technical background tend to fall into the first category whereas those who lean towards the arts or humanities tend to fall into the other. Like all generalisations this one is, again, not strictly correct, but I think it is fair to say that a person’s training does have an influence on what they deem as the right way to make decisions.
The physicist and polymath Heinz von Foerster summed this up nicely when he noted that the difference between the two types of decision-makers is akin to the differences between discoverers and inventors. The objective decision-maker (the discoverer) attempts to discover the objectively correct decision based on what he or she believes to be true. On the other hand, the subjective decision-maker (the inventor) constructs or creates the decision based on facts and opinion (or even emotion) rather than facts alone.
The conventional view of decision making in organisations – that decisions should be made on the basis of facts – does not recognize this difference. To be sure, matters that can be decided based on facts should be made on the basis of facts. For example, a decision regarding the purchase of equipment can (often) be made based on predetermined criteria.The problem, however, is that most important decisions in organisations do not fall into this category – they have wicked elements that cannot be resolved by facts because the “facts” themselves are ambiguous. Unfortunately, decision-makers often do not understand the difference between the two types of decision problems. A common symptom of this lack of understanding is that when confronted with a wicked decision problem, many decision-makers feel compelled to clothe their reasoning and choices in a garb of (false) objectivity.
The above is not news to observers and scholars of organisational life – see this post, for example. However, a not-so-well-appreciated dimension to the objective/subjective debate on decision-making is that wicked decision problems invariably have an ethical dimension. I elaborate on this briefly below.
In a paper on ethics and cybernetics, von Foerster noted that the objective approach to decision making is but a means of avoiding responsibility. In his words:
…objectivity requires that the properties of an observer be left out of any descriptions of his (sic) observations. With the essence of observing (namely the processes of cognition) having been removed, the observer is reduced to a copying machine with the notion of responsibility successfully juggled away.
Objectivity…and other devices [such as rules and processes] are all derivations of a choice between a pair of in-principle undecidable questions [See Note 1] which are:
“Am I apart from the universe?”
or
“Am I a part of the universe?”
Although von Foerster may be accused overblown rhetoric in the quote, he raises a critical question that we all ought to ask ourselves when confronted with an undecidable issue:
When making this decision, am I going to avoid involvement (and responsibility) by hiding behind rules or processes, or am I going to take full responsibility for it regardless of the outcome?
An honest answer will reveal that such decisions are invariably made on ethical grounds rather than objective ones. Indeed, the decisions we make in our professional lives tell us more about ourselves than we might be willing to admit.
An undecidable question is one that cannot be decided on logical grounds alone – a wicked problem by another name. See my post on wickedness, undecidability and the metaphysics of organizational decision making for more on this point.
Eight to Late 