Making Sense of KPI Changes

As analysts, we are usually monitoring metrics. Quite often, metrics change. And when they do, it’s our job to figure out what’s going on: why did the conversion rate suddenly drop, or what is driving consistent revenue growth?

I started my journey in data analytics as a Kpi analyst. For almost three years, I’d been doing root cause analysis and KPI deep dives nearly full-time. Even after moving to product analytics, I’m still regularly investigating the KPI shifts. You could say I’ve become quite the experienced analytics detective.

The cornerstone of Root Cause Analysis is usually slicing and dicing the data. Most often, figuring out what segments are driving the change will give you a clue to the root causes. So, in this article, I would like to share a framework for estimating how different segments contribute to changes in your key metric. We will put together a set of functions to slice and dice our data and identify the main drivers behind the metric’s changes.

However, in real life, before jumping into data crunching, it’s important to understand the context: 

• Is the data complete, and can we compare recent periods to previous ones?
• Are there any long-term trends and known seasonal effects we’ve seen in the past?
• Have we launched anything recently, or are we aware of any external events affecting our metrics, such as a competitor’s marketing campaign or currency fluctuations?

I’ve discussed such nuances in more detail in my previous article, “Root Cause Analysis 101”.

KPI change framework

We encounter different metrics, and analysing their changes requires different approaches. Let’s start by defining the two types of metrics we will be working with:

• Simple metrics represent a single measure, for example, total revenue or the number of active users. Despite their simplicity, they are often used in product analytics. One of the common examples is the North Star metrics. Good North Star metric estimates the total value received by customers. For example, AirBnB might use nights booked, and WhatsApp might track messages sent. Both are simple metrics.

You can learn more about North Star Metrics from the Amplitude Playbook.

• However, we can’t avoid using compound or ratio metrics, like conversion rate or average revenue per user (ARPU). Such metrics help us track our product performance more precisely and isolate the impact of specific changes. For example, imagine your team is working on improving the registration page. They can potentially track the number of registered customers as their primary KPI, but it might be highly affected by external factors (i.e., a marketing campaign driving more traffic). A better metric for this case would be a conversion rate from landing on a registration page to completing it.

We will use a fictional example to learn how to approach root cause analysis for different types of metrics. Imagine we are working on an e-commerce product, and our team is focused on two main KPIs: 

• total revenue (a simple metric),
• conversion to purchase — the ratio of users who made a purchase to the total number of users (a ratio metric).

We will use synthetic datasets to look at possible scenarios of metrics’ changes. Now it’s time to move on and see what’s going on with the revenue.

Analysis: simple metrics

Let’s start simple and dig into the revenue changes. As usual, the first step is to load a dataset. Our data has two dimensions: country and maturity (whether a customer is new or existing). Additionally, we have three different scenarios to test our framework under various conditions.

import pandas as pd
df = pd.read_csv('absolute_metrics_example.csv', sep = '\t')
df.head()

The main goal of our analysis is to determine how each segment contributes to the change in our top-line metric. Let’s break it down. We will write a bunch of formulas. But don’t worry, it won’t require any knowledge beyond basic arithmetic.

First of all, it’s helpful to see how the metric changed in each segment, both in absolute and relative numbers. 

\[\textbf{difference}^{\textsf{i}} = \textbf{metric}_{\textsf{before}}^\textsf{i} – \textbf{metric}_{\textsf{after}}^\textsf{i}\\
\textbf{difference_rate}^{\textsf{i}} = \frac{\textbf{difference}^{\textsf{i}}}{\textbf{metric}_{\textsf{before}}^\textsf{i}}\]

The next step is to look at it holistically and see how each segment contributed to the overall change in the metric. We will calculate the impact as the share of the total difference.

\[\textbf{impact}^{\textsf{i}} = \frac{\textbf{difference}^{\textsf{i}}}{\sum_{\textsf{i}}{\textbf{difference}^{\textsf{i}}}}\]

That already gives us some valuable insights. However, to understand whether any segment is behaving unusually and requires special attention, it’s useful to compare the segment’s contribution to the metric change with its initial share of the metric. 

Here’s the reasoning. If the segment makes up 90% of our metric, then it’s expected for it to contribute 85–95% of the change. But if a segment that accounts for only 10% ends up contributing 90% of the change, that’s definitely an anomaly.

To calculate it, we will simply normalise each segment’s contribution to the metric by the initial segment size. 

\[\textbf{segment_share}_{\textsf{before}}^\textsf{i} = \frac{\textbf{metric}_{\textsf{before}}^\textsf{i}}{\sum_{\textsf{i}}{\textbf{metric}_{\textsf{before}}^\textsf{i}}}\\
\textbf{impact_normalised}^\textsf{i} = \frac{\textbf{impact}^{\textsf{i}}}{\textbf{segment_share}_{\textsf{before}}^\textsf{i}}\]

That’s it for the formulas. Now, let’s write the code and see this approach in practice. It will be easier to understand how it works through practical examples. 

def calculate_simple_growth_metrics(stats_df):
  # Calculating overall stats
  before = stats_df.before.sum()
  after = stats_df.after.sum()
  print('Metric change: %.2f -> %.2f (%.2f%%)' % (before, after, 100*(after - before)/before))

  # Estimating impact of each segment
  stats_df['difference'] = stats_df.after - stats_df.before
  stats_df['difference_rate'] = (100*stats_df.difference/stats_df.before)\
    .map(lambda x: round(x, 2))
  stats_df['impact'] = (100*stats_df.difference / stats_df.difference.sum())\
    .map(lambda x: round(x, 2))
  stats_df['segment_share_before'] = (100* stats_df.before / stats_df.before.sum())\
    .map(lambda x: round(x, 2))
  stats_df['impact_norm'] = (stats_df.impact/stats_df.segment_share_before)\
    .map(lambda x: round(x, 2))

  # Creating visualisations
  create_parallel_coordinates_chart(stats_df.reset_index(), stats_df.index.name)
  create_share_vs_impact_chart(stats_df.reset_index(), stats_df.index.name, 'segment_share_before', 'impact')
  
  return stats_df.sort_values('impact_norm', ascending = False)

I believe that visualisations are a crucial part of any data storytelling as visualisations help viewers grasp insights more quickly and intuitively. That’s why I’ve included a couple of charts in our function: 

• A parallel coordinates chart to show how the metric changed in each slice — this visualisation will help us see the most significant drivers in absolute terms.
• A scatter plot to compare each segment’s impact on the KPI with the segment’s initial size. This chart helps spot anomalies — segments whose impact on the KPI is disproportionately large or small.

You can find the complete code for the visualisations on GitHub.

Now that we have all the tools in place to analyse revenue data, let’s see how our framework performs in different scenarios.

Scenario 1: Revenue dropped equally across all segments

Let’s start with the first scenario. The analysis is very straightforward — we just need to call the function defined above.

calculate_simple_growth_metrics(
  df.groupby('country')[['revenue_before', 'revenue_after_scenario_1']].sum()\
    .sort_values('revenue_before', ascending = False).rename(
        columns = {'revenue_after_scenario_1': 'after', 
          'revenue_before': 'before'}
    )
)

In the output, we will get a table with detailed stats.

However, in my opinion, visualisations are more informative. It’s obvious that revenue dropped by 30–40% in all countries, and there are no anomalies. 

Scenario 2: One or more segments drove the change

Let’s check out another scenario by calling the same function.

calculate_simple_growth_metrics(
  df.groupby('country')[['revenue_before', 'revenue_after_scenario_2']].sum()\
    .sort_values('revenue_before', ascending = False).rename(
        columns = {'revenue_after_scenario_2': 'after', 
          'revenue_before': 'before'}
    )
)

We can see the biggest drop in both absolute and relative numbers in France. It’s definitely an anomaly since it accounts for 99.9% of the total metric change. We can easily spot this in our visualisations.

Also, it’s worth going back to the first example. We looked at the metric split by country and found no specific segments driving changes. But digging a little bit deeper might help us understand what’s going on. Let’s try adding another layer and look at country and maturity.

df['segment'] = df.country + ' - ' + df.maturity 
calculate_simple_growth_metrics(
    df.groupby(['segment'])[['revenue_before', 'revenue_after_scenario_1']].sum()\
        .sort_values('revenue_before', ascending = False).rename(
            columns = {'revenue_after_scenario_1': 'after', 'revenue_before': 'before'}
        )
)

Now, we can see that the change is mostly driven by new users across the countries. These charts clearly highlight issues with the new customer experience and give you a clear direction for further investigation.

Scenario 3: Volume shifting between segments

Finally, let’s explore the last scenario for revenue.

calculate_simple_growth_metrics(
    df.groupby(['segment'])[['revenue_before', 'revenue_after_scenario_3']].sum()\
        .sort_values('revenue_before', ascending = False).rename(
            columns = {'revenue_after_scenario_3': 'after', 'revenue_before': 'before'}
        )
)

We can clearly see that France is the biggest anomaly — revenue in France has dropped, and this change is correlated with the top-line revenue drop. However, there is another outstanding segment — Spain. In Spain, revenue has increased significantly.

This pattern raises a suspicion that some of the revenue from France might have shifted to Spain. However, we still see a decline in the top-line metric, so it’s worth further investigation. Practically, this situation could be caused by data issues, logging errors or service unavailability in some regions (so customers have to use VPNs and appear with a different country in our logs).

We’ve looked at a bunch of different examples, and our framework helped us find the main drivers of change. I hope it’s now clear how to conduct root cause analysis with simple metrics, and we are ready to move on to ratio metrics.

Analysis: ratio metrics

Product metrics are often ratios like average revenue per customer or conversion. Let’s see how we can break down changes in this type of metrics. In our case, we will look at conversion. 

There are two types of effects to consider when analysing ratio metrics: 

• Change within a segment, for example, if customer conversion in France drops, the overall conversion will also drop.
• Change in the mix, for example, if the share of new customers increases, and new users typically convert at a lower rate, this shift in the mix can also lead to a drop in the overall conversion rate.

To understand what’s going on, we need to be able to distinguish these effects. Once again, we will write a bunch of formulas to break down and quantify each type of impact. 

Let’s start by defining some useful variables.

\[
\textbf{c}_{\textsf{before}}^{\textsf{i}}, \textbf{c}_{\textsf{after}}^{\textsf{i}} – \textsf{converted users}\\
\textbf{C}_{\textsf{before}}^{\textsf{total}} = \sum_{\textsf{i}}{\textbf{c}_{\textsf{before}}^{\textsf{i}}}\\
\textbf{C}_{\textsf{after}}^{\textsf{total}} = \sum_{\textsf{i}}{\textbf{c}_{\textsf{after}}^{\textsf{i}}}\\
\textbf{t}_{\textsf{before}}^{\textsf{i}}, \textbf{t}_{\textsf{after}}^{\textsf{i}} – \textsf{total users}\\
\textbf{T}_{\textsf{before}}^{\textsf{total}} = \sum_{\textsf{i}}{\textbf{t}_{\textsf{before}}^{\textsf{i}}}\\
\textbf{T}_{\textsf{after}}^{\textsf{total}} = \sum_{\textsf{i}}{\textbf{t}_{\textsf{after}}^{\textsf{i}}}
\]

Next, let’s talk about the impact of the change in mix. To isolate this effect, we will estimate how the overall conversion rate would change if conversion rates within all segments remained constant, and the absolute numbers for both converted and total users in all other segments stayed fixed. The only variables we will change are the total and converted number of users in segment i. We will adjust it to reflect its new share in the overall population.

Let’s start by calculating how the total number of users in our segment needs to change to match the target segment share. 

\[
\frac{\textbf{t}_{\textsf{after}}^{\textsf{i}}}{\textbf{T}_{\textsf{after}}^{\textsf{total}}} = \frac{\textbf{t}_{\textsf{before}}^{\textsf{i}} + \delta\textbf{t}^{\textsf{i}}}{\textbf{T}_{\textsf{before}}^{\textsf{total}}+ \delta\textbf{t}^{\textsf{i}}} \\
\delta\textbf{t}^{\textsf{i}} = \frac{\textbf{T}_{\textsf{before}}^{\textsf{total}} * \textbf{t}_{\textsf{after}}^{\textsf{i}} – \textbf{T}_{\textsf{after}}^{\textsf{total}} * \textbf{t}_{\textsf{before}}^{\textsf{i}}}{\textbf{T}_{\textsf{after}}^{\textsf{total}} – \textbf{t}_{\textsf{after}}^{\textsf{i}}}
\]

Now, we can estimate the change in mix impact using the following formula. 

\[
\textbf{change in mix impact} = \frac{\textbf{C}_{\textsf{before}}^{\textsf{total}} + \delta\textbf{t}^{\textsf{i}} * \frac{\textbf{c}_{\textsf{before}}^{\textsf{i}}}{\textbf{t}_{\textsf{before}}^{\textsf{i}}}}{\textbf{T}_{\textsf{before}}^{\textsf{total}} + \delta\textbf{t}^{\textsf{i}}} – \frac{\textbf{C}_{\textsf{before}}^{\textsf{total}}}{\textbf{T}_{\textsf{before}}^{\textsf{total}}}
\]

The next step is to estimate the impact of the conversion rate change within segment i. To isolate this effect, we will keep the total number of customers and converted customers in all other segments fixed. We will only change the number of converted users in segment i to match the target conversion rate at a new point. 

\[
\textbf{change within segment impact} = \frac{\textbf{C}_{\textsf{before}}^{\textsf{total}} + \textbf{t}_{\textsf{before}}^{\textsf{i}} * \frac{\textbf{c}_{\textsf{after}}^{\textsf{i}}}{\textbf{t}_{\textsf{after}}^{\textsf{i}}} – \textbf{c}_{\textsf{before}}^{\textsf{i}}}{\textbf{T}_{\textsf{before}}^{\textsf{total}}} – \frac{\textbf{C}_{\textsf{before}}^{\textsf{total}}}{\textbf{T}_{\textsf{before}}^{\textsf{total}}} \\ = \frac{\textbf{t}_{\textsf{before}}^{\textsf{i}} * \textbf{c}_{\textsf{after}}^{\textsf{i}} – \textbf{t}_{\textsf{after}}^{\textsf{i}} * \textbf{c}_{\textsf{before}}^{\textsf{i}}}{\textbf{T}_{\textsf{before}}^{\textsf{total}} * \textbf{t}_{\textsf{after}}^{\textsf{i}}}
\]

We can’t simply sum the different types of effects because their relationship is not linear. That’s why we also need to estimate the combined impact for the segment. This will combine the two formulas above, assuming that we will match both the new conversion rate within segment i and the new segment share.

\[
\textbf{total segment change} = \frac{\textbf{C}_{\textsf{before}}^{\textsf{total}} – \textbf{c}_{\textsf{before}}^{\textsf{i}} + (\textbf{t}_{\textsf{before}}^{\textsf{i}} + \delta\textbf{t}^{\textsf{i}}) * \frac{\textbf{c}_{\textsf{after}}^{\textsf{i}}}{\textbf{t}_{\textsf{after}}^{\textsf{i}}}}{\textbf{T}_{\textsf{before}}^{\textsf{total}} + \delta\textbf{t}^{\textsf{i}}} – \frac{\textbf{C}_{\textsf{before}}^{\textsf{total}}}{\textbf{T}_{\textsf{before}}^{\textsf{total}}}
\]

It’s worth noting that these effect estimations are not 100% accurate (i.e. we can’t sum them up directly). However, they are precise enough to make decisions and identify the main drivers of the change.

The next step is to put everything into code. We will again leverage visualisations: correlation and parallel coordinates charts that we’ve already used for simple metrics, along with a couple of waterfall charts to break down impact by segments.

def calculate_conversion_effects(df, dimension, numerator_field1, denominator_field1, 
                       numerator_field2, denominator_field2):
  cmp_df = df.groupby(dimension)[[numerator_field1, denominator_field1, numerator_field2, denominator_field2]].sum()
  cmp_df = cmp_df.rename(columns = {
      numerator_field1: 'c1', 
      numerator_field2: 'c2',
      denominator_field1: 't1', 
      denominator_field2: 't2'
  })
    
  cmp_df['conversion_before'] = cmp_df['c1']/cmp_df['t1']
  cmp_df['conversion_after'] = cmp_df['c2']/cmp_df['t2']
  
  C1 = cmp_df['c1'].sum()
  T1 = cmp_df['t1'].sum()
  C2 = cmp_df['c2'].sum()
  T2 = cmp_df['t2'].sum()

  print('conversion before = %.2f' % (100*C1/T1))
  print('conversion after = %.2f' % (100*C2/T2))
  print('total conversion change = %.2f' % (100*(C2/T2 - C1/T1)))
  
  cmp_df['dt'] = (T1*cmp_df.t2 - T2*cmp_df.t1)/(T2 - cmp_df.t2)
  cmp_df['total_effect'] = (C1 - cmp_df.c1 + (cmp_df.t1 + cmp_df.dt)*cmp_df.conversion_after)/(T1 + cmp_df.dt) - C1/T1
  cmp_df['mix_change_effect'] = (C1 + cmp_df.dt*cmp_df.conversion_before)/(T1 + cmp_df.dt) - C1/T1
  cmp_df['conversion_change_effect'] = (cmp_df.t1*cmp_df.c2 - cmp_df.t2*cmp_df.c1)/(T1 * cmp_df.t2)
  
  for col in ['total_effect', 'mix_change_effect', 'conversion_change_effect', 'conversion_before', 'conversion_after']:
      cmp_df[col] = 100*cmp_df[col]
        
  cmp_df['conversion_diff'] = cmp_df.conversion_after - cmp_df.conversion_before
  cmp_df['before_segment_share'] = 100*cmp_df.t1/T1
  cmp_df['after_segment_share'] = 100*cmp_df.t2/T2
  for p in ['before_segment_share', 'after_segment_share', 'conversion_before', 'conversion_after', 'conversion_diff',
                   'total_effect', 'mix_change_effect', 'conversion_change_effect']:
      cmp_df[p] = cmp_df[p].map(lambda x: round(x, 2))
  cmp_df['total_effect_share'] = 100*cmp_df.total_effect/(100*(C2/T2 - C1/T1))
  cmp_df['impact_norm'] = cmp_df.total_effect_share/cmp_df.before_segment_share

  # creating visualisations
  create_share_vs_impact_chart(cmp_df.reset_index(), dimension, 'before_segment_share', 'total_effect_share')
  cmp_df = cmp_df[['t1', 't2', 'before_segment_share', 'after_segment_share', 'conversion_before', 'conversion_after', 'conversion_diff',
                   'total_effect', 'mix_change_effect', 'conversion_change_effect', 'total_effect_share']]

  plot_conversion_waterfall(
      100*C1/T1, 100*C2/T2, cmp_df[['total_effect']].rename(columns = {'total_effect': 'effect'})
  )

  # putting together effects split by change of mix and conversion change
  tmp = []
  for rec in cmp_df.reset_index().to_dict('records'): 
    tmp.append(
      {
          'segment': rec[dimension] + ' - change of mix',
          'effect': rec['mix_change_effect']
      }
    )
    tmp.append(
      {
        'segment': rec[dimension] + ' - conversion change',
        'effect': rec['conversion_change_effect']
      }
    )
  effects_det_df = pd.DataFrame(tmp)
  effects_det_df['effect_abs'] = effects_det_df.effect.map(lambda x: abs(x))
  effects_det_df = effects_det_df.sort_values('effect_abs', ascending = False) 
  top_effects_det_df = effects_det_df.head(5).drop('effect_abs', axis = 1)
  plot_conversion_waterfall(
    100*C1/T1, 100*C2/T2, top_effects_det_df.set_index('segment'),
    add_other = True
  )

  create_parallel_coordinates_chart(cmp_df.reset_index(), dimension, before_field='before_segment_share', 
    after_field='after_segment_share', impact_norm_field = 'impact_norm', 
    metric_name = 'share of segment', show_mean = False)
  create_parallel_coordinates_chart(cmp_df.reset_index(), dimension, before_field='conversion_before', 
    after_field='conversion_after', impact_norm_field = 'impact_norm', 
    metric_name = 'conversion', show_mean = False)

  return cmp_df.rename(columns = {'t1': 'total_before', 't2': 'total_after'})

With that, we’re done with the theory and ready to apply this framework in practice. We’ll load another dataset that includes a couple of scenarios.

conv_df = pd.read_csv('conversion_metrics_example.csv', sep = '\t')
conv_df.head()

Scenario 1: Uniform conversion uplift

We will again just call the function above and analyse the results.

calculate_conversion_effects(
    conv_df, 'country', 'converted_users_before', 'users_before', 
    'converted_users_after_scenario_1', 'users_after_scenario_1',
)

The first scenario is pretty straightforward: conversion has increased in all countries by 4–7% points, resulting in the top-line conversion increase as well. 

We can see that there are no anomalies in segments: the impact is correlated with the segment share, and conversion has increased uniformly across all countries. 

We can look at the waterfall charts to see the change split by countries and types of effects. Even though effect estimations are not additive, we can still use them to compare the impacts of different slices.

The suggested framework has been quite helpful. We were able to quickly figure out what’s going on with the metrics.

Scenario 2: Simpson’s paradox

Let’s take a look at a slightly trickier case.

calculate_conversion_effects(
    conv_df, 'country', 'converted_users_before', 'users_before', 
    'converted_users_after_scenario_2', 'users_after_scenario_2',
)

The story is more complicated here: 

• The share of UK users has increased while conversion in this segment has dropped significantly, from 74.9% to 34.8%.
• In all other countries, conversion has increased by 8–11% points.

Unsurprisingly, the conversion change in the UK is the biggest driver of the top-line metric decline.

Here we can see an example of non-linearity: 10% of effects are not explained by the current split. Let’s dig one level deeper and add a maturity dimension. This reveals the true story: 

• Conversion has actually increased uniformly by around 10% points in all segments, yet the top-line metric has still dropped. 
• The main reason is the increase in the share of new users in the UK, as these customers have a significantly lower conversion rate than average.

Here is the split of effects by segments.

This counterintuitive effect is called Simpson’s paradox. A classic example of Simpson’s paradox comes from a 1973 study on graduate school admissions at Berkeley. At first, it seemed like men had a higher chance of getting in than women. However, when they looked at the departments people were applying to, it turned out women were applying to more competitive departments with lower admission rates, while men tended to apply to less competitive ones. When they added department as a confounder, the data actually showed a small but significant bias in favour of women.

As always, visualisation can give you a bit of intuition on how this paradox works.