The universe of digital analytics is massive and can seem as complex as the cosmic universe.
With such big, complicated subjects, we can get lost in the vast wilderness or become trapped in a silo. We can wander aimlessly, or feel a false sense of either accomplishment or frustration. Consequently, we lose sight of where we are, how we are doing and which direction is true north.
I have experienced these challenges on numerous occasions myself. Even simple questions like “How effective is our analytics strategy?” elicit a complicated set of answers, instead of a simple picture the CxO can internalize. That’s because we have to talk about tools (so many!), work (collection, processing, reporting, analysis), processes, org structure, governance models, last-mile gaps, metrics ladders of awesomeness, and… so… much… more.
Soon, your digital analytics strategic framework that you hoped would provide a true north to the analytics strategy question looks like this…
The frameworks above cover just one dimension of the assessment (!). There is another critical framework to figure out how you can take your analytics sophistication from wherever it is at the moment to nirvanaland.
A quick search query will illustrate that that looks something like this…
It is important to stress that none of these frameworks/answers exist in a vacuum.
Both pictures above are frighteningly complex because the analytics world we occupy is complex. Remember, tools, work, processes, org structure, governance models, last-mile gaps, metrics ladders of awesomeness, and… so… much… more.
The Implications of Complexity.
There are two deeply painful outcomes of the approaches you see in the pictures above (in which you’ll also see my work represented as well).
1. Obvious:
2. Non-obvious:
When one grasps these two outcomes – blind business leaders, blind analytics leaders – it is simply heartbreaking.
Simplifying Complexity.
The dilemma of how to simplify this complexity, to create sighted business and analytics leaders, has lingered with me for quite some time. I’ve intended to create a simple visual that absorbs the scale, complexity and many moving parts.
On this blog, you’ve seen numerous attempts by me to remedy the dilemma. To name a few: Digital Marketing & Measurement Model | Analytics Ecosystem | Web Analytics 2.0. Each aimed to solve a particular dimension, yet none solved the heartache completely. Especially for the non-obvious problem #2 above.
The hunger remained.
I wanted to create a visual that would function as a diagnostic tool to determine if you are lost, trapped in a silo or wandering aimlessly. It would help you realize the extent to which analytics impacted the business bottom line today, and what your future analytics plans should accomplish.
Then one day, a magic moment.
During a discussion around planning for measurement, a peer was struggling with a unique collection of challenges. He asked me a couple of questions, and that sparked an idea.
I walked up to the whiteboard, and excitedly sketched something simple that abstracted away the complexity – and yet preserved the power of smarter thinking at the same time.
Here’s the sketch I drew in response:
Yes, it was an ugly birth. But, to me, the proud parent, it was beautiful.
It took a sixteen hour direct flight to Singapore for the squiggly sketch to come to life – where else, in PowerPoint!
The end result was just five slides. As the saying goes: It’s not the ink, it’s the think.
I want to share the fully fleshed out, put into practice and refined, version of those four slides with you today. Together, they’ll help you fundamentally rethink your analytics practice by, 1. understanding data’s actual impact on your company today and, 2. picking very precise and specific things that should be in your near and long-term analytics plans.
The Impact Matrix.
To paint a simple picture of the big, complicated world of analytics, the whiteboard above shows a 2×2 matrix.
Each cell contains a metric (online, offline, nonline).
The business impact is on the y-axis, illustrated from Super Tactical to Super Strategic.
The time-to-useful is on the x-axis, illustrated from Real-Time to 6-Monthly.
Before we go on… Yes, breaking the x-axis into multiple time segments creates a 2×5 matrix, and not a 2×2. Consider that to be the price I’ve paid in order to make this more actionable for you. 🙂
Diving a bit deeper into the y-axis… Super Tactical is the smallest possible impact on the business (fractions of pennies). Super Strategic represents the largest possible impact on the business (tens of millions of dollars).
The scale on the y-axis is exponential. You’ll notice the numbers in light font between Super Tactical and Super Strategic go from 4 to 10 to 24 to 68 and onward. This demonstrates that impact is not a step-change – every step up delivers a massively higher impact.
Diving a bit deeper into the x-axis… While most data can be collected in real-time now, not all metrics are useful in real-time.
As an example, Impressions can be collected in real-time and they can also become useful in real-time (if actioned, they can have a super tactical impact – fractions of pennies). Customer Lifetime Value on the other hand takes a long time to become useful, over months and months (if actioned, it can have a super strategic impact on the business – tens of millions of dollars).
Here is a representation of these ideas on the Impact Matrix:
[You can download an Excel version of the Impact Matrix at the end of this post.]
Impressions can be used in real-time for decision-making by your display, video and search platforms (e.g., via automation). You can report Gross Profit in real-time, of course, but doing so is almost entirely useless. It should be deeply analyzed monthly to yield valuable, higher impact actionable insights. Finally, Lifetime Value will require perhaps the toughest strategic analysis, from data accumulated over months, and the action takes time to yield results – but they are magnificent.
Pause. Reflect on the above picture.
If you understand why each metric is where it is, the rest of this post will fill you with euphoric joy rarely experienced without physical contact.
The Impact Matrix: A Joyous Deep Dive.
In all, the Impact Matrix contains 46 of the most commonly used business metrics – with an emphasis on sales and marketing. The metrics span digital, television, retail stores, billboards, and any other presence of a brand you can think of. You see more digital metrics because digital is more measurable.
Some metrics apply across all channels, like Awareness, Consideration and Purchase Intent. You’ll note the most critical bottom line metrics, which might come from your ERP and CRM systems, are also included.
Every metric occupies a place based on business impact and time of course, but also in context of other metrics around it.
Here’s a magnified view that includes the bottom left portion of the matrix:
Let’s continue to internalize impact and time-to-useful by looking at a specific example: Bounce Rate. It’s in the row indicating an impact of four and in the time-to-useful column weekly. While Bounce Rate is available in real-time, it is only useful after you’ve collected a critical amount of data (say, over a week).
On the surface, it might seem odd that a simple metric like Bounce Rate has an impact of four and TV GRPs and % New Visits are lower. My reason for that is the broader influence of Bounce Rates.
Effectively analyzing and acting on Bounce Rates requires the following:
* The ability to identify any empty promises made to the users who are bouncing.
* Knowing the content, including its emotional and functional value.
* The ability to optimize landing pages.
Imagine the impact of those insights; it is well beyond Bounce Rates. That is why Bounce Rate garners more weight than Impressions, Awareness and other common metrics.
When designating a metric as a KPI, this is your foremost consideration: depth of influence.
With a better understanding of the Impact Matrix, here’s the full version:
[You can download an Excel version of the Impact Matrix at the end of this post.]
As you reflect on the filled out matrix, you’ll note that I’ve layered in subtle incentives.
For example, if you were to compute anything Per Human, you would need to completely revamp your identity platforms (a strategy I’ve always favored: Implications Of Identity Systems On Incentives). Why should you make this extra effort? Notice how high those metrics sits on the business impact scale!
Other hidden features.
The value of voice of customer metrics is evident by their high placement in context of the y-axis. Take a look at where Task Completion Rate by Primary Purpose and Likelihood to Recommend are, as an example. They are high in the hierarchy due to their positive impact on both the business and the company culture – thus delivering a soft and hard advantage.
You’ll also note that most pure digital metrics – Adobe, Google Analytics – sit in the tactical bottom line impact. If all you do day and night is just those metrics, this is a wake-up call to you in context of your actual impact on the company and the impact of that on your career.
At the top-right, you’ll discover my obsession with Profit and Incrementality, which form the basis of competitive advantage in 2018 (and beyond). Analyzing these metrics not only fundamentally changes marketing strategy (think tens of millions of dollars for large companies); their insights can change your company’s product portfolio, your customer engagement strategies and much more.
The matrix also includes what is likely the world’s first widely available machine learning-powered metric: Session Quality, which you’ll find roughly in the middle. For every session on your desktop or mobile site, Session Quality provides a score between 1 and 100 as an indication of how close the visitor is to converting. The number is computed based on a ML algorithm that has learned from deep analysis of your user behavior and conversion data.
Pause. Download the full resolution version of the picture. Reflect.
It is my hope that the placement of each of the 46 metrics will help you add metrics that might be unique to your work. (Share them in comments below, add to our collective knowledge.)
With a better understanding of the matrix, you are ready to overcome the two problems that broke our hearts at the start of the post – and do something super-cool that you did not think we might.
Action #1: Analytics Program Maturity Diagnostic.
Enough theory, time to some real, sexy, work.
The core driver behind creation of the Impact Matrix was the non-obvious problem #2: How much does your analytics practice matter from a bottom line perspective?
YOU matter if you have a business impact. You’ll have a business impact if your analytics practice is sophisticated enough to produce metrics that matter. See the nice circular reference?
🙂
In our case we measure maturity not by evaluating people, process, and layers upon layers of tools, rather we measure maturity by evaluating the output of that entire song and dance.
Answer this simple question: What metrics are most commonly used to make decisions that drive actual actions every week/month/more?
Ignore the metrics produced as an experimental exercise nine months ago. Ignore the metrics whose only purpose is to float along the river of data pukes. Ignore the metrics you wish you were analyzing, but don’t currently.
Reality. Assess, reality. No point in fooling yourself.
Take the subset of metrics that actively drive action, and change the font color for them to green in the Impact Matrix.
For a large European client with a multi-channel existence, here’s what the Impact Matrix looked like after this honest self-reflection:
More of the digital metrics are green, because there are more digital metrics period. You can see the company’s marketing strategy spans television and other offline advertising, including retail.
You’ll likely recognize many of these metrics as the one that your analytics practice outputs every day. They represent the result of a lot of hard work by the company employees, and external analytics partners.
We are trying to answer the how much does the analytics practice matter question. You can see that more sharply now.
For this company most green metrics cluster in the bottom-left quadrant, with most having an impact of ten or under (on a y-axis scale of 1 to a ). There is one clear outlier (Nonline Direct Revenue – a very difficult metric to compute, so hurray!)
As every good consultant know, if you have a 2×2 you can create four thematic quadrants. In our case the four quadrants are called Solid Foundation, Intermediate, and Advanced:
For our company, the maturity of the analytics practice fit mostly in the Solid Foundation quadrant.
Is this a good thing?
It depends on how long the analytics practice has been around, how many Analysts the company has, how much money it has invested in analytics tools, the size of their agency analytics team, so on and so forth.
If they have a team of 50 people spending $18 mil on analytics investment each year, over the last decade, with 12 tools and 25 research studies each year… You can now infer that this is not a good thing.
Regardless, the Impact Matrix now illuminates clearly that highly influential metrics are underutilized. These are the metrics that facilitate deeper thought, patience and analysis to deliver big bottom line impact.
Recommendation Uno:
Recommendation Dos:
Recommendation Très:
The good news is that it is now, finally, clear where you are going AND why you are going there. Congratulations!
Recommendation Cuatro:
Most companies wait for some Jesus-Krishna hybrid to descend from heaven and deliver a glorious massive revolution project (overnight!). These never happen. Sorry, Jesus-Krishna. Instead, what it takes is each employee moving a little bit up and a little bit to the right while the Analytics team facilitates those shifts. Small changes accumulate big bottom line impact over time.
So. What’s your quadrant? And, what’s your next right or next up move?
Action #2: Aligning Metrics & Leadership Altitude.
When offered data, everyone wants everything.
People commonly believe that more data means better results. Or, that if an Agency is providing a 40 tab, font size 8, spreadsheet full of numbers that they must have done a lot of work – hence better value for money. Or, a VP wants two more histograms that represent seven dimensions squeezed into her one page dashboard.
If more data equaled smarter decisions, they would be peace on earth.
A core part of our job, as owners of the analytics practice, is to ensure that the right data (metric) reaches the right person at the right time.
To do so, we must consider altitude (aka the y-axis).
Altitude dictates the scope and significance of decisions. It also dictates the frequency at which data is received, along with the depth of insights that need to accompany the data (IABI FTW!). Finally, altitude determines the amount of time allotted to discuss findings.
Managers have a lower altitude, they are required to make tactical decisions – impacting say tens of thousands of dollars. VPs have a higher altitude, they are paid a ton more in salary, bonus and stock, because they carry the responsibility for making super strategic decisions – impacting tens of millions of dollars.
This problem has a beautifully elegant solution if you use the Impact Matrix.
Slice the matrix horizontally to ensure that the metrics delivered to each leader are aligned with their altitude.
[You can download an Excel version of the Impact Matrix at the end of this post.]
VPs sit at decision making that is squarely in the Super Strategic realm – on our scale ~40 and higher. This collection of metrics power heavy decisions requiring abundant business context, deep thinking and will influence broad change. Analysts will need that time to conduct proper analysis and obtain the IABI.
You can also see that nearly all metrics delivered to the VPs arrive monthly or even less frequently. Another reflection of the fact that their altitude requires solving problems that will connect across orgs, across incentives, across user touch points, etc.
So. Are the metrics on your VP Dashboards/Slides the ones in Super Strategic cluster?
Or. Is your analytics practice such that your VPs spend their time making tactical decisions?
Below the VP layer, you’ll see metric clusters for slightly less strategic impact on the company bottom line for Directors. The time-to-useful also changes on the x-axis for them. Following them is the layer for managers, who make even more frequent, tactical decisions.
The last layer is my favorite way to improve decision making: Removing humans from the process. 🙂
Recent technical advancements allow us to use algorithms – machine learning – to automate decisions made by metrics that have a Super Tactical impact. For example, there is no need for any human to review Viewability because advanced display platforms optimize campaigns automatically against this metric. In fact an expensive human looking at reports with that metric will only slow things down – eliminating the fractions of penny impact that that metric delivers.
Recommendation Cinco:
For example: Does your last CMO report include Profit per Human, Incremental Profit per Non-line Channel, % Contribution of Non-line Channels to Sales? If yes, hurray! Instead, if they are reporting Awareness, Consideration, Intent, Conversions, Bounce Rate… Sad time. Why would your CMO use his or her valuable time making tactical choices? Is it a culture problem? Is it a reflection of the lack of analytical savvy? Why?
Put simply, the big and complicated is not so big and not so complicated. This simple analysis will help identify core issues that are stymieing the contribution data can make to smarter, faster, business success.
Recommendation Seis:
For the other layers, action will depend on what the problem is. It could require new leadership in the analytics team, it could require a shift in company culture, or it could require a different engagement model across layers (managers, directors, VPs). One thing adjusting the altitude will certainly require: Change in how employees are compensated.
As you notice above, the strength of the matrix is in it’s ability to simplify complexity. That does not mean that you don’t have to deal with complexity – you can be more focused about it now!
Action #3: Strategy for Analytical Effort.
One more slicing exercise for our matrix, this time for the analytics team itself.
Analytics teams face a daunting challenge when figuring out what type of effort to put into tackling the fantastic collection of possibilities represented in the Impact Matrix.
That challenge is compounded by the fact that there is always too much to do and too few people to do it with. Oh, and don’t get me started on time! Why are there only 24 hours in a day??
So, how do we ensure that each has an optimal analytical approach?
Slice the matrix vertically along the time-to-useful dimension…
[You can download an Excel version of the Impact Matrix at the end of this post.]
For any metric that is useful in real-time, we have to unpack the forces of automation. Campaigns can be optimized based on real-time impressions, clicks, visits, page views, cost per acquisition etc. We need to stop reporting these, and start feeding them into our campaign platforms like AdWords, DoubleClick etc. With simple rules – ranges mostly – automation platforms can do a better job of taking action than humans.
If you are investing in machine learning talent inside your team, even narrowly intelligent algorithms they build will learn faster and surpass humans quickly for these simple decisions.
With the day-to-day sucking of life spirit reduced, tactical impact decisions automated, the analytics practice has time to focus on metrics that have a longer time-to-useful and need deeper human analysis to extract the IABI.
For metrics available weekly or within a few weeks, reporting to various stakeholders (mostly Managers and Directors) should adequately inform decisions. Use custom alerts, trigger threshold targets and more to send this data to the right person at the right time. For weekly time-to-useful metrics, your stakeholders have enough tactical context that you don’t need to spend time on deep analysis since the metrics inform the tactical decisions.
More role clarity, a thoughtful shift of the burden to the stakeholders, and more free time to focus on what really matters.
For where time-to-useful is in the month range, you are now truly heading into strategic territory. Reflect on the metrics there – challenging, strategic, Director and VP altitude. It is no longer enough to just report what happened, you need to identify why it happened and what the causal impact is for the why factors. This will yield insights that will have millions of dollars of potential impact on the company. That means, you’ll need to invest in ensuring your stories have more than just insights but also include specific recommended actions and predicted business impact. Amazingly, you’ll have just as much text as data in your output (that’s how you know you are doing it right!).
Finally, we have the pinnacle of analytics achievement. Our last vertical slice includes metrics that measure performance across customer segments, divisions and channels, among other elements. This is where meta-analysis comes into play, requiring even more time, with even more complex analytical techniques that pull data into BigQuery or similar environments where you can do your own joins, unleash R, use statistically modeling techniques (hello random forests!) to find the most important factors affecting your company’s performance.
The distribution of your analytical team’s effort across these four categories is another method of assessing maturity as well as ensuring optimal impact by the precious few analytical resources. For example: If most of your time is occupied by providing data to decision-makers for metrics in the Automate and Reporting vertical slices, you are likely in the beginner stage (and not having much impact on the business bottom line).
Recommendation Siete:
The answers to these questions will cause a fundamental re-imagination of your analytics practices. The implications will be deep and wide (people, process, tools). That is how you get on the road to true nirvanaland.
#sisepuede
At the core of the Impact Matrix is the only thing that matters: the business bottom line. Using two simple dimensions, impact and time-to-useful, you can explain simply three unique elements of any successful analytics practice. The reflections are sometimes painful, but bringing them to light allows us to take steps toward systematic improvement of our analytical practice.
That’s the power of a 2×2 (or a 2×5)!
Here’s an Excel version of the Impact Matrix for your personal use.
As always, it is your turn now.
When your CMO asks, “How effective is our analytics strategy?”, what’s your answer? How simply can you frame it? What are the primary inputs to your near and long-term analytics evolution plans? If your VPs are getting the metrics in the Advanced quadrant, what strategies have been effective in getting you there? If you’ve successfully implemented pattern matching and advanced classification meta-analysis techniques, care to share your lessons with us?
Please share your feedback about the Impact Matrix, and answers to the above questions, via comments below. I look forward to the conversation.
Thank you.
The post The Impact Matrix | A Digital Analytics Strategic Framework appeared first on Occam’s Razor by Avinash Kaushik.
Powered by WPeMatico