You won’t find true signals of changes in performance by looking at month-to-month comparisons, or trendlines, or moving averages. The signals you really need to know about, the only signals that you ought to respond to, are revealed through one particular graph only.
The typical analysis methods we use for our performance measures are based on assumptions that don’t make much sense.
Month-to-month comparisons assume that there is no routine variation over time and wrongly interpret any difference as a signal.
Trendlines assume that all change is linear and gradual, and that if Excel can calculate a trend line then there must be a trend.
Moving averages assume that seasonal patterns exist, and also that change is smooth and gradual.
The only analysis technique I have ever come across that clearly filters the noise and highlights the signals in our performance measures is the XmR chart.
The XmR chart filters the noisy routine variation in our measures by showing us how much of this routine variation there is, by way of the Natural Process Limits. And coupled with the Central Line, these Natural Process Limits give us a meaningful baseline to quickly assess when performance has changed; when there is something else going on that’s not part of the routine variation.
There are three very specific signals to look for.
SIGNAL 1: Outlier or special cause
When a measure value falls outside the Natural Process Limits, it means that more than just the routine variation is at play. It’s a signal that something else has happened.
If Employee Attendance plummeted below the bottom Natural Process Limit, a likely cause could be a flu epidemic or local natural disaster that kept many more people away from work in that period.
Even though this is a signal that something out of the norm has happened, because it’s just a one-time event, we don’t react to it. We find out what caused it, but we don’t run around madly trying to fix it. That would be a massive waste of time and money because we’d essentially by trying to fix something that won’t happen again, or that is completely outside our control.
SIGNAL 2: Long run
To be convinced that a change in the level of performance has happened, we need to see seven (yes, seven) points in a row on the same side of the Central Line. The probability that a pattern like that is part of routine variation is close to zero (0.78%, to be precise). Seven points, not three or five or one.
If a measure of Invoice Accuracy showed a long run above the Central Line, it might be evidence that an initiative to simplify the pricing strategy had successfully reduced the errors in invoices.
When we see a long run signal in our measure, we certainly need to find the cause for it. Sometimes it will be a signal of improvement, and we want to confirm what caused the improvement. Other times it will be a signal that performance has deteriorated and the cause of that is very important to identify!
SIGNAL 3: Short run
You’re no doubt thinking to yourself ‘I can’t wait for seven months before I can know if I should take action!’ You can either measure more frequently to pick up signals sooner (as long as it makes sense to), or plan for bigger signals.
A bigger signal appears as a short run, of three out of four consecutive measure values closer to a Natural Process Limit than they are to the Central Line. The probability of this pattern happening also has a very close to zero probability.
A short run above the Central Line for On-time Deliveries for a trucking company would likely be due to an initiative that had a substantially large impact. It could be something like doubling the fleet size. But it also could be a new competitor in the market that poached a large percentage of their customers.
Again, with a signal like the short run, it’s really important to find the cause before responding.
XmR charts take only a little effort to create, but their usefulness is so powerful they are absolutely worth trying.
Stacey Barr is a specialist in organisational performance measurement and creator of PuMP, the refreshingly practical, step-by-step performance measurement methodology designed to overcome people’s biggest struggles with KPIs and measures. Learn about the bad habits that cause these struggles, and how to stop them, by taking Stacey’s free online course “The 10 Secrets to KPI Success” at www.staceybarr.com/the10secretstokpisuccess.