Tips for Executives – How to Create a Culture of Evidence

We’re often asked how do we create a Culture of Evidence? Most leaders know that they should be more evidence-based in how they work, but don’t know how they can go about doing it.

We’ve all heard the phrase “Culture eats strategy for breakfast” and anyone who’s attempted to drive change in a complex organization knows how true that statement can be. And, many seasoned leaders know that culture change doesn’t happen overnight, but here are some tips that you can use to get started.

Culture of Evidence

Tip 1: Paint a picture of “What a Culture of Evidence looks like”
If you want to make meaningful progress towards creating a culture of evidence, there’s no better place to start than envisioning your future state. Things to consider include:

  • How will life be better? For you, your team and for the company?
  • What opportunities will you be able to access?
  • What risks will you be able to avoid?
  • What decisions will be smarter?
  • What time will be saved?

If you can create a compelling vision of your organization in the future that thrives in a Culture of Evidence, then you can use this to win supporters.

Tip 2: Set the standard for “What counts as evidence?”
In the spirit of “crawl, walk, run”, getting started with using evidence doesn’t have to begin with hiring a team of scientists, researchers and lawyers. To begin with it may be as simple as using data to support your decision-making, carrying out basic research, or using spreadsheets to do “what if” analysis. Most leaders do this already, but many others still rely on their intuition to make their decisions.

The following is an illustrative example of “what counts as evidence?”:

  • A declarative statement of your position such as “I believe that we should launch a social media awareness campaign for our red widgets”
  • Some form of objective proof that shows how you formed your position, such as “According to our market data 85% of our target customers have never heard of our red widgets, and 57% of them use social media. The campaign would be cost effective even if it only generated a 5% increase in our market share.”
  • A disclosure of what you don’t know, such as “Admittedly our market data is one year old, so we’re assuming that the patterns still hold.”
  • An action statement, such as “I’d like to update our market data but the delays and costs outweigh the risk of missing an opportunity … I recommend that we launch the campaign and track performance.”

The ultimate goal of evidence is that it holds up to the review process, meaning that another leader could review the evidence and arrive at the same conclusions. Along those lines, “what counts as evidence?” could be just that … an objective analysis that has been peer reviewed.

Manager Reading Data

Tip 3: Put the tools in place
To set your team up for success, you will want to make sure that the basic tools are available for evidence-based thinking. Some questions to consider include:

  • Are the right investments being made to collect the right data?
  • Does your team have access to the data they need? Is the data being collected at the source, but it’s not being stored in the data warehouse? Or is the data there, but the privacy levels are too restrictive?
  • Do they have the skills for working with the data, or alternatively, is the right information available in insightful reports or visual dashboards?
  • Do they have the right technical and human resources perform deeper analyses, in response to important business questions that arise?

Tip 4: Lead by example
If you want to convince your team and your peers that you are fully behind this idea of a Culture of Evidence, then you’ll need to walk the talk. This will require effort at the beginning, but after a while it will become just “the way things are done around here”. Leading by example can include shifting your own language from “I think this is what we should do …” into “The evidence tells me that this is what we should do …”

It can also include making a concerted effort to not do things the old way because “that’s the way we’ve always done it” but instead doing things in ways that are proven to generate the right outcomes. This relates to everyday decision-making and operations, as well as longer-term strategy and planning.

Tip 5: Reward the adopters
It is often said that “you get what you reward”. This is an easy concept to apply to building a Culture of Evidence. For example you can reward your team for using evidence in situations like:

  • Decision-making on special projects: Projects that have proposals that have supporting evidence are often approved, whereas other projects often don’t.
  • Decision-making on budget: Budget increases (or exemptions from budget cuts) are generally provided to those departments that can prove that they need it, whereas departments that can’t prove their value miss out.
  • Decision-making on promotions: Team members that demonstrate the effective use of evidence are generally promoted to higher positions, whereas other team members don’t.

By taking this approach it won’t take long for people in your organization to learn that the way to win is by embracing an evidence-based approach. Team members will either adopt the new direction or self-select themselves out of your organization. Over time this will increase the momentum of the culture change, and gradually you will find that your organization attracts talent that values a Culture of Evidence.

Tips for Executives – How to Get the Data You Need

One of the most common complaints that we hear from leaders and executives is that they have “too much data” and “not enough information”. Some examples of what they mean by “too much data” include:

  • Reports that consist of pages and pages of numbers
  • Tables of figures with no overall summary number
  • Charts that are cluttered and confusing
  • Analyses that show a lot of numbers but no “so what” message

It doesn’t have to be that way. Here are a few tips that executives can use to get the data they need:

Tip 1: Ask yourself “What information would help me be more effective?”
It may sound selfish, but you should ask yourself “What information would help me be more effective in my job?” This might be information that helps you save your own time, make better decisions, or seize big opportunities.

The Data Thinker

Another way to approach this question is to review the data that you already have available and ask yourself “What isn’t this telling me?” or “Why is this not useful to me?”

Based on this thought process, prepare a simple table with two columns. In the first column include a description of what you want, and in the second column identify why you want it. Then choose your top 3 to 5 items on the list. Now you’re ready to start the next step – following up with your Data Team and/or your Business Intelligence people to have a first conversation about your top-ranked items.

Tip 2: When people say data isn’t available, use the “5 Whys”
Many data people have difficultly seeing the world beyond the standard data that they use every day. So, when you meet with them and tell them about the data that you need, chances are that they will reply by saying “that just isn’t available”.

When it comes to data – almost anything is available – it’s just a matter of how much you’re willing to fight to get what you need.

The “5 Whys” is a simple process of getting to the root of an issue. When your data people tell you that getting the data you need is impossible, ask “why”. They will give you a list of reasons such as “it’s not in the data warehouse”, or “we don’t measure that”, or “the system doesn’t allow that type of reporting”. Pick any of the reasons, and then ask “why” again, which will generate a new list of reasons. Continue this until you’ve reached the root of the issue (hopefully in 5 or less “whys”). The root issue is often one or more of the following:

  • Nobody thought to ask for this before
  • At some point in the past, somebody decided that it was too hard to collect the data
  • The people running the analysis and reporting are limiting themselves based on the capabilities of their reporting tools
  • Nobody has thought of taking a prospective data collection approach, and/or nobody has thought of doing a sampling approach (to reduce data collection costs)

Through a few meetings, you now should have the real reasons why you don’t currently have the information you need. You may even have a sense of how much it would cost.

Tip 3: Estimate the cost of not having the information you need
The last step is where you can make your convincing argument. For each of your top-ranked ideas, you can think about what it’s costing you to not have access to that information.

Does it translate to productivity? Lost time? Missed opportunities? Lost revenue? Customer loyalty? Employee turn-over? If so, then you can translate these consequences into real tangible costs. This isn’t an exercise of doing high-precision activity based costing – instead this is just getting the cost estimates roughly right.

These figures give you an idea of how much your organization could potentially invest into better data and reporting. If you’re business-minded and you could work out the actual investment amounts that would still generate a positive return on investment.

Armed with this analysis, now you’re in a position to convince others what this information is worth. Which brings us to our last step.

Tip 4: Gain the support of the leadership team
Chances are that the information that will help you be more effective in your role, will also be useful to others in the leadership team and throughout your organization. If you can gain the support of the rest of the leadership team then you can increase the chances of getting what you want.

Each team dynamic is different, but a one-on-one approach often works well. These can be quick conversations with each leader with a real focus on “what’s in it for them”. You may be surprised with how many of your peers are equally frustrated by the lack of good information.

With the support of the team, the cost of not having the information and some return on investment estimates, you’ll be able to drive to get the information you need to be successful.

These are just a few tips, but I’m sure there are many of leaders out there who have many more great ideas and experiences. If you have suggestions, or alternate points of view, please weigh in.

 

Note: What is a Data Team?
When we refer to “Data Teams” it’s a catch all for groups of technical, statistical, and subject-matter domain experts that are involved in providing information to support their organization. These teams are sometimes called “Business Intelligence”, “Decision Support”, or “Information Management”, but they can also be internal consultants such as “Operations Analysts”, “Strategic Information” or “Research”. Many of these concepts equally apply to teams of Data Scientists.


Tips for Data Teams – The Consistency Check

Have you ever delivered an analysis, only to hear from your client that “these numbers can’t be right”? It’s hard to convince someone that your results are credible when they don’t even pass the first 5 seconds of review. As much as we may not want to admit it, sometimes the numbers are indeed wrong, so how do we avoid these situations from happening? One type of check that a Data Team can adopt is the “Consistency Check”. Here are some questions that you can ask yourself when doing a consistency check:

Consistent numbers

Question 1) Are the numbers consistent with themselves?
When building complicated analyses, different sections of the analysis can fall “out of sync” with each other if they are not all updated in the same way. When this happens it can produce inconsistent summary results (i.e. the cover page reports 255 conversions per hour, but the supporting details on other pages show 237 conversions per hour). Sometimes we place too much faith on our reporting tools and assume that they will report exactly as intended. In other situations it’s just a matter of being too close to the work. After a while the numbers are burned into your short term memory and you lose your ability to critically review them with an objective eye. Suggested work-arounds include:

  • Have another member of your team do a consistency check on the results, preferably someone who hasn’t been involved in the work.
  • Take an old school approach. Print out the results, and use different colored highlighters for each type of metric. Highlight the summary numbers that represent the same result, and confirm that they are indeed consistent. Continue until you’ve highlighted all summary numbers.
  • Take another old school approach. Get your calculator out or use a separate spreadsheet, and confirm that you can replicate the summary numbers just based on the results that are being presented. You may be surprised with how many of your clients are doing this with your results already.

Question 2) Are the numbers consistent with your previous analyses?
When a client receives a new set of results they often pull up the previous results that you gave them. They are asking the question “how much have things changed?” You can beat them to the punch by doing this consistency check yourself. To be more specific:

  • Start with the previous result that was presented or released. Compare the summary numbers from the previous results to your current summary numbers.
  • Assess if the changes are interpretable. If they are, then this interpretation will likely be part of what you communicate when you release the new results. If the changes are not interpretable, then it’s time to go back into your current results, or your previous results to diagnose why the changes aren’t explainable.

Question 3) Are the numbers consistent with other reports?
Stepping into the shoes of your audience, you can think about the other reports that they are referring to on an on-going basis. It doesn’t matter if the other reports that they use came from a completely different source – from their perspective all data from all sources is supposed to tell the same story. In a similar manner to Question 2, you can do some additional homework so that your results are valuable to your audience as possible. For example you could:

  • Ask your clients if they have any other reports that they use frequently, and if they would be willing to share them with you. You can frame it honestly – you want to make sure that your results are valid, and if they are different from other sources, you want to be able to explain why.
  • Do a little research on your own, in particular, reviewing any routine corporate reporting, or industry reporting. Sometimes, a skeptic can be won over by proving that you did your homework. Again if the numbers line up from other sources, it becomes something you can report as proof of consistency. If the numbers don’t line up and you can’t explain the difference, then it may be an indication that you need to review your analysis.

Question 4) Are you telling the right story?
Taking all of the above into account, you should be able to deliver your results confidently. You should now know that the numbers in the report are consistent amongst themselves, that the analysis is consistent with previous analyses, and that the results are interpretable in comparison to other sources. This now can become part of your summary and presentation of your stunning new work. Or at least it can form as an addendum to the email, or the presentation that shows your audience the efforts that you went through to ensure that the numbers are the right numbers. Then you have the foundation to begin telling the actual story of the analysis (the “so what” message).

These are just a few tips, but I’m sure there are many of experts out there who have many more great ideas. If you have suggestions, or alternate points of view, please weigh in.

Note: What is a Data Team?
When we refer to “Data Teams” it’s a catch all for groups of technical, statistical, and subject-matter domain experts that are involved in providing information to support their organization. These teams are sometimes called “Business Intelligence”, “Decision Support”, or “Information Management”, but they can also be internal consultants such as “Operations Analysts”, “Strategic Information” or “Research”. Many of these concepts equally apply to teams of Data Scientists.