X

The KPI.org Blog

How to Cheat at Wordle Using Indirect Measurement

By: David Wilsey

Mar 9, 2022 268 Views 0 Comments FacebookTwitterLinkedInGoogle Plus

You have likely heard of the word-guessing game Wordle. If you are on social media, you might even be tired of seeing the related green and yellow grids on your feed each day. You might not know, however, that a few folks have discovered a sneaky way to cheat.

Apparently, the Wordle word-of-the-day has been showing up on the list of top searches on certain online dictionary sites around 9:00 am US east coast time lately. So, if you are so inclined, you could scan that list for 5-letter words before playing and have a good idea what to guess.

This example demonstrates how an organization might use indirect measurement (or a “proxy” measure) as a KPI. An indirect measure can be used in cases where it is not practical to directly measure an intended result or goal and is based on a hypothesis around correlation or contribution. Metaphorically speaking, let’s say the word-of-the-day is something an organization wants to accomplish but is finding it difficult to measure directly. You could create a KPI around the number of searches for any particular 5-letter word on the dictionary sites and then use that data to inform your word-guessing decision. The hypothesis is based on an expected correlation between the dictionary searches and the game.

It also demonstrates the imprecise nature of indirect measures. The dictionary search measure informs my word-guessing decision, but I can’t assume that there isn’t some other reason why a 5-letter word might show up on the list. Direct measurement is almost always preferred. If we want more sales, the direct sales dollars tell us what we want to know precisely. Once I shift to something that is indirect, like web traffic on my shopping cart website, my measurement becomes less reliable and requires more work to connect to the intended result.

But in the management world, many times a direct measurement is not obvious. A non-profit that works in advocacy might not be able to identify tangible intended results of their work. Teams that succeed through collaboration and innovation sometimes end up with softer, more indirect, measures of success. Or in the example above, maybe I want to measure the web traffic to better understand what is contributing to sales results. The point is that sometimes indirect measurement is useful, as long as you understand where the weaknesses are.

If you would like to learn more about indirect measurement or our general approach to KPI development, please consider our KPI Professional certification program.

Source: https://www.washingtonpost.com/video-games/2022/02/19/wordle-cheat/

Image source: Wikipedia

David Wilsey David Wilsey

David Wilsey is the Chief Operating Officer with the Balanced Scorecard Institute and co-author of The Institute Way: Simplify Strategic Planning and Management with the Balanced Scorecard.

KPI Development: When to Use Ratios

By: David Wilsey

Nov 30, 2020 595 Views 0 Comments FacebookTwitterLinkedInGoogle Plus

Let’s say you are managing a large organization and your HR director comes to you with bad news. The total number of employees leaving the company has skyrocketed. They show you the chart to the right to indicate that the raw number of employees leaving last year was below 20 and this year it is 50. Should you panic?

The answer is that it depends. What if I told you that this year the company acquired a competitor and that the total number of employees had expanded from just over 500 to over 2200 (see the chart below), and so the turnover percentage (# leaving / total # of employees) had actually decreased slightly (see Turnover ratio). Would that make you panic? Probably not.

 

The point is that raw counts can be misleading if you are not careful about the comparisons you are making relative to the underlying population. You might have seen debates in the news about how political leaders or writers have highlighted raw COVID 19 counts to support an argument when a per capita ratio suggests a different conclusion. It’s not that raw counts aren’t sometimes useful, but it is important to be aware of what you are comparing and how the overall population might impact interpretation. We recommend that any measures that could be misinterpreted be expressed as a ratio. Common examples of ratios include:

  • Percent completion
  • Coverage, fraction of the total possible
  • Error or defect rate
  • Per capita
  • Efficiency = Output / Input
  • Productivity = Output / Cost (or Output / work hour)

Beginners in the KPI space often make the mistake of developing raw count measures where ratios would be more meaningful.

One type of ratio you do NOT see on this list is percent increase. Many of our clients mistakenly think that if they create a ratio comparing the current period with the previous period that they have solved this problem. While a percent increase is technically a ratio, creating a ratio using the 2017 raw count and the 2016 raw count is still just as misleading, as more than 150% increase in turnover from year to year is still unnecessarily alarming. Of course, you could choose to track a percent increase of the ratio, but that would be addressing a different question, one that I will address in an upcoming blog.

To learn more about ratios and other KPI development issues, please see our KPI Professional Certification program.

KPI Development: When to Use Ratios

By: David Wilsey

Nov 30, 2020 1002 Views 0 Comments FacebookTwitterLinkedInGoogle Plus

Let’s say you are managing a large organization and your HR director comes to you with bad news. The total number of employees leaving the company has skyrocketed. They show you the chart to the right to indicate that the raw number of employees leaving last year was below 20 and this year it is 50. Should you panic?

The answer is that it depends. What if I told you that this year the company acquired a competitor and that the total number of employees had expanded from just over 500 to over 2200 (see the chart below), and so the turnover percentage (# leaving / total # of employees) had actually decreased slightly (see Turnover ratio). Would that make you panic? Probably not.

 

The point is that raw counts can be misleading if you are not careful about the comparisons you are making relative to the underlying population. You might have seen debates in the news about how political leaders or writers have highlighted raw COVID 19 counts to support an argument when a per capita ratio suggests a different conclusion. It’s not that raw counts aren’t sometimes useful, but it is important to be aware of what you are comparing and how the overall population might impact interpretation. We recommend that any measures that could be misinterpreted be expressed as a ratio. Common examples of ratios include:

  • Percent completion
  • Coverage, fraction of the total possible
  • Error or defect rate
  • Per capita
  • Efficiency = Output / Input
  • Productivity = Output / Cost (or Output / work hour)

Beginners in the KPI space often make the mistake of developing raw count measures where ratios would be more meaningful.

One type of ratio you do NOT see on this list is percent increase. Many of our clients mistakenly think that if they create a ratio comparing the current period with the previous period that they have solved this problem. While a percent increase is technically a ratio, creating a ratio using the 2017 raw count and the 2016 raw count is still just as misleading, as more than 150% increase in turnover from year to year is still unnecessarily alarming. Of course, you could choose to track a percent increase of the ratio, but that would be addressing a different question, one that I will address in an upcoming blog.

To learn more about ratios and other KPI development issues, please see our KPI Professional Certification program.

David Wilsey David Wilsey

David Wilsey is the Chief Operating Officer with the Balanced Scorecard Institute and co-author of The Institute Way: Simplify Strategic Planning and Management with the Balanced Scorecard.

4 Ways Your KPIs Remind Me of Our New Puppy

By: David Wilsey

Aug 31, 2020 832 Views 0 Comments FacebookTwitterLinkedInGoogle Plus
We welcomed a new puppy named Romeo to our home recently, having finally given in to the kids after about five years of pleading. While we are generally happy with the decision, late last night I stood outside in the rain watching the dog refuse to relieve himself and realized that there are several similarities between a new puppy and the challenges that my KPI clients face.

 

They Lack Discipline

I keep telling myself that it’s not that puppies want to create a big mess, it’s that they just don’t know better. Many organizations develop KPIs that are designed by folks that are new to the KPI world also lack discipline. Without a disciplined development process, organizations tend to brainstorm KPIs that monitor activities or operational elements that aren’t very meaningful. Even when decent measures are selected, implementation requires discipline. I can’t tell you the number of training participants who have told me that they’ve selected measures but haven’t gotten around to tracking anything yet.

They Bite When They Play

My kids love playing with the puppy but complain about his playful nipping. Similarly, measures can play rough at first too as you begin to learn things you didn’t know about your performance. I’ll never forget the KPI champion I worked with that stopped supporting the program soon after he realized that their new performance measure was making him look bad. Just like it takes puppies a few months to learn not to bite, every organization must learn how to manage the fear that is associated with measuring performance. Bad news should be welcomed as it provides focus for the organization.

They Are More Work at the Beginning

With puppies, things can get easier over time provided you invest the time and effort to establish good habits. Performance measures are the same way. Some clients get discouraged by the mountain of work required to develop, define, and implement good measures. It is hard to convince some it does get easier once the measures are in place and you establish an effective reporting routine.

They’re Often Excited About the Wrong Things

Romeo hasn’t met a roll of toilet paper yet that he doesn’t want to devour. KPIs can also cause clients to fixate on inappropriate things if no one has clarified what is important. One recent training delegate told me that they still track and report on network uptime on a regular basis simply because the data was handy. Network uptime was of no great strategic importance for their organization and so the entire effort was a waste of time. KPIs are supposed to be “KEY”, meaning they indicate improvement of an important performance result. Strategic measures tell you if the organization is moving forward or changing to meet the needs of a new world. Operational measures tell you if you if you are delivering quality products and services effectively and efficiently. Most of the clients that tell us they need help with measurement actually need help articulating what they are trying to accomplish.

To learn more about how to overcome these and other measurement challenges, please see our KPI Professional Certification program.

David Wilsey David Wilsey

David Wilsey is the Chief Operating Officer with the Balanced Scorecard Institute and co-author of The Institute Way: Simplify Strategic Planning and Management with the Balanced Scorecard.

The “Formula” for KPI Success

By: David Wilsey

Jan 31, 2020 901 Views 0 Comments FacebookTwitterLinkedInGoogle Plus

I was facilitating a client performance measure development session recently, and the team was wrestling with measure definition. When I reviewed the customer perspective team’s work I found:

Measure Name: % of Customers Satisfied
Description:
% of Customers Satisfied
Formula:
% of Customers Satisfied

The team was mystified by what the formula should be. The following definitions and examples helped them get back on track. Measure name is a word or short phrase that names the specific measure and is understood by users. The description details what the measurement is about, including its intent, why it matters and what it includes and excludes. Imagine describing the measure to someone on an elevator. A formula is the mathematical equation or rule used to calculate the measurement, and expressed in symbols, essentially a set of instructions for creating the desired metric. To demonstrate, below are a few common measures from various walks of life, with an example of a description and formula.

Name: Customer Satisfaction Survey Score
Description:
Periodic survey of a sampling of customers, who are asked to quantify an “I am satisfied”-type response on a five-point Likert scale.
Formula:
# indicating Agree or Strongly Agree / total # of responses

Name: Fuel Efficiency
Description:
the distance traveled per unit of fuel used. Used in the US, UK, and much of Asia. The higher the value, the more efficient the car is at using fuel.
Formula:
# of miles driven / # of gallons of fuel used (or # of km driven / # of liters used)

Name: Fuel Consumption
Description:
amount of fuel used per unit distance driven, often expressed as liters used per 100 kilometers driven. Used in much of Europe. The lower the value, the more economical a vehicle because it takes less fuel to drive a certain distance.
Formula:
(# liters of fuel used / kilometers driven) *100

Name: Batting Average (baseball)
Description:
Batting average is used to compare the consistency of batters in baseball. It is calculated as the number of hits divided by the official number of at-bats and is expressed as a decimal to three places of accuracy.
Formula:
Total # of hits / total # of at bats, not including walks, sacrifices, or hit by pitches

Name: Steps Description: Tracking the number of steps walked each day is an effective way to ensure that you are getting enough physical activity each day. Countless research has indicated that measurable benchmarks in fitness can help improve health outcomes.
Formula:
Total # of steps walked per day

Name: Graduation Rate
Description: Graduation rates is the percentage of students who graduate from a school within a prescribed time period (which varies between school levels but is typically 150% of the published time for to complete a program). Graduation rates are considered important because it shows how committed schools are to helping students learn all that is required to meet the graduation standard within a reasonable amount of time.
Formula: # of students graduating within a prescribed time period / total # of students in the graduating class

Name: Employee Turnover Rate
Description:
Employee turnover is a measurement of how many employees are leaving a company. High turnover can be costly, as replacing employees is expensive. High voluntary turnover of top performers can also be an indicator of poor management or morale issues.
Formula: # of employees (voluntarily) exiting the job / average actual # of employees (during the same period)

For strategy-minded people more interested in discussing lofty strategy options than grinding through data collection and validity details, data definition can be a challenge. But I’ve found that if you can master these three basic data definition steps, you’ll be a long way towards consensus about exactly what you are counting and why.

For more information about KPI / performance measure development, please visit KPI.org/KPIP.


9 Things We Wish We Had Known About KPIs

Recorded October 30, 2018 - Replay Available

Oct 30, 2018 3433 Views 0 Comments FacebookTwitterLinkedInGoogle Plus

Recorded October 30, 2018 - Replay Available Below

Isaac Newton famously said: "If I have seen further it is by standing on the shoulders of Giants” which means it is easier to discover truth when you can build upon previous discoveries.

BSI's COO David Wilsey and Corporater's former Chief Strategy Officer & VP, Gail S. Perry will share the 9 things they wish they had understood about KPIs from the very beginning, instead of learning them thru mistakes, hard experience, and untangling KPI messes at numerous clients. They have a combined 50 years of performance management experience and they have the battle scars to show for it. 

While neither Gail nor David claim to be giant or genius, the 9 lessons they share will jump start your understanding of KPIs and help you leapfrog over similar missteps. 





Sign up for our newsletter to get the latest information about our next webinar! 

 

Gail Stout Perry Gail Stout Perry

Gail is co-author of The Institute Way. With a career spanning over 30 years of strategic planning and performance management consulting with corporate, nonprofit, and government organizations, she enjoys speaking, training, and writing, sharing her experience with others. She currently is the Chief Strategy Officer and VP Americas for Corporater.

The Four Things I Wish I’d Known - Part 3

By: Gail S. Perry

Oct 15, 2018 5004 Views 0 Comments FacebookTwitterLinkedInGoogle Plus

KPIs Are Essential (But Know Your Audience)

I wanted to sink into the conference room floor. I was so embarrassed and was convinced that I must have just asked the stupidest question in the world. To this day, I cringe at the memory of standing there, in front of the entire leadership team at a prestigious, world-renowned, non-profit organization, while the entire team stared blankly at me. I was well into the second decade of my consulting career and was accustomed to taking on major projects. This time, I’d been asked to design a dashboard of metrics for this organization.  I’d gathered the heads of all functions and departments to explain the purpose of the project, their roles as stakeholders, and then, poised to write responses on the whiteboard, I asked the question: “Can each of you tell me what three to six key metrics you use (or would like to us) to manage your part of the organization?” My thinking was that this would give us a quick and rough outline of what metrics mattered most to the people who ran the organization – these same people who were now staring at me. Finally, one gentleman spoke up and said, “I believe this is what we hired you to do – don’t you know what metrics we should use?”  

Fast forward another decade and I have a lot more KPI experience under my belt and I’ve worked with dozens of major organizations on performance management as well as implemented KPIs for my own use in managing an organization. And in hindsight, I now realize that the managers who were staring at me should have known their key processes and value drivers and been able to articulate what they were trying to accomplish and how to measure it.

I have since learned that there are two kinds of managers/leaders. Those who operate at a tactical level and those who see the full picture. The tactical managers keep very busy managing what Covey calls the whirlwind of daily operations. Some focus only on the day-to-day actions that are required of them.  Some are great at people skills. Some are more entrepreneurial and implement innovations, initiatives, and projects they feel are needed as they sense and respond to risks and signals at the tactical level. But after all these years, I see how these sorts of managers consistently fail to achieve meaningful long-term results. They perform well on daily operations, but few can achieve sustainable improvements in those operations.  And that is exactly what happened to every one of the managers in that conference room. They did their daily jobs well, but they couldn’t produce long-term results for the organization.  Within five years, all were replaced.

The other type of managers/leaders see the full picture of key processes and value drivers and they leverage KPIs to monitor and manage performance. They know KPIs (metrics) will enable them to better manage overall performance as well as to assess the impact of any innovations, initiatives, and projects. 
  
I’ve since learned that I didn’t ask a stupid question. I simply was asking it of the wrong sort of manager/leaders. I’ve asked that same question of the other type of managers and they rattle off metrics faster than I can write them down.  

I have learned to assess the audience first and be sensitive to the fact that not everyone knows about KPIs or how they enable managers with insights and power for improving performance. Some individuals may need some basic education about the topic, they may have a long change management journey to buy into the value and use of KPIs, and they most likely will need coaching help to figure out their key processes and value drivers, as well as how to determine appropriate KPIs to use.  

It’s not rocket science. To some of us, it is simply common sense. But not everyone is wired this way.  We are all born with different natural tendencies so I’ve tried to learn to keep that in mind. And I no longer sink into the floor when someone stares blankly at me. I simply start asking more questions until we find common ground and then work forward from there.  

Read Part 2 of The Four Things I Wish I’d Known here. Read Part 4 here
Terry Sterling Terry Sterling

Terry is a Certified Balanced Scorecard Master Professional and the Training Manager and Senior Associate with over 30 years of experience working in both the private and public sectors.

A World Without Measures

By: Terry Sterling

Aug 29, 2018 4606 Views 0 Comments FacebookTwitterLinkedInGoogle Plus
Welcome to a miserably hot, sweltering day at Yankee Stadium with the temperature sitting at 95° F in the shade and 90% humidity.  It’s the bottom of the ninth and the Yankees are down by three runs. A collective sense of expectation can be felt throughout the stadium as Didi Gregorius, who has 10 home runs already this season, steps back into the batter’s box with bases loaded and a full count.  Boston’s ace leftie, Chris Sale, with a 4-1 record and a 2.39 ERA going into the game, goes into his windup.  The pitch is a 99-mph split-finger fastball at the knees on the inside corner of the plate.  Gregorius swings, connects and the ball travels 420 feet clearing the center field wall by 12 feet; a grand slam and the Yankees win their 28th game of the season!

Rewind 
Welcome to another day at the empty lot where an unspecified number of people have shown up to play baseball. It feels hot, but since no one measures temperature no one is sure if it is any hotter than usual. The game is loose – the bases and fence are randomly placed, and the game continues until the players decide they are done, as no one tracks innings or measures time.  No one knows the score; no one counts strikes or balls; no one tracks how many runs or hits are made on a team or individual basis; much less  the type or speed of the pitch being delivered.  No talent is required to play because no one keeps track of how one performs; baseball statistics don’t exist. Alas, there is no excitement and no tension; no pressure to improve. No one loses or wins…pure UTOPIA!!
Measurements play a relevant part of our daily world, both in our personal and professional lives.  H. James Harrington summed it up like this, “Measurement is the first step that leads to control and eventually to improvement.  If you can’t measure something, you can’t understand it.  If you can’t understand it, you can’t control it.  If you can’t control it, you can’t improve it.”

Everyone uses measurements every day of their life and usually don’t even pause to give them consideration.  We all check the weather to see how warm or cold it is.  We use measurements to determine our budgets, how much money we need for vacations and how well our children are performing in school.  We calculate how many minutes we need to add to our workout routine to allow us to eat that extra piece of cake; how much we can afford to pay for things, and so much more.  Face it, life in today’s world doesn’t exist without measures!

What confounds us and causes confusion, missed opportunities and misguided, unproductive efforts is our inability and/or lack of experience in determining what measurements are key.  How do we determine how effective we are in achieving our desired goals and objectives, whether in our personal life or at work? How do we measure success and how do we know when we have achieved it?

The answer centers around the concept of developing Key Performance Indicators (KPI’s). The challenge is in developing and knowing what measurements are “Key” in determining the success of our performance and that of our organizations.

If you want to learn more about KPI’s and how to develop useful and meaningful measures for your organization, visit: http://kpi.org

Sources:
H. James Harrington Quotes.  Retrieved from: https://www.goodreads.com/author/quotes/42617.H_James_Harrington

David Wilsey David Wilsey

David Wilsey is the Chief Operating Officer with the Balanced Scorecard Institute and co-author of The Institute Way: Simplify Strategic Planning and Management with the Balanced Scorecard.

Why "World Class" Performance Isn’t Measurable

By: David Wilsey

Nov 14, 2017 9064 Views 0 Comments FacebookTwitterLinkedInGoogle Plus
Let’s say our organization needs to buy a fleet of vehicles and we have two procurement teams. We tell team 1 that we want quiet, blue, four-door, fuel-efficient cars. We tell team 2 that we want world-class, high-quality, great-value, high-performing cars. Then we give both teams a few weeks to find their vehicles. Guess which team will be able to produce measurable results?

Team 1 will have the easier time, as it is clearer what is meant by the criteria provided. Team 2 will struggle because their criteria are too ambiguous. Without further clarifications, “world-class” could be interpreted to mean a hot rod sports car, a luxury sedan, or even a nice SUV. And if the team cannot agree on the specifically desired result, how can it measure success? 

This example demonstrates an important principle of good measure design. Before you can design a measure, you first must agree on what result you are trying to achieve. And not all results are created equal. Results written in abstract language are less measurable and harder to implement than those written in concrete language.

Abstract language refers to concepts or vague ideals. Examples of abstract words or phrases include sustainable, innovative, reliable, leadership, quality, effective, leverage, efficient, resilient, optimized, or responsive. Strategic plans are often littered with this type of language, as we aim to deliver best practices, thought leadership or world-class performance. These “weasel words”, as they are often called, are notoriously hard to measure without first translating into concrete terms. 
Concrete language is sensory-specific, meaning it describes things you can see, hear, smell, taste, or feel. Because they are observable, concrete results are measurable. Team 1 will have no problem determining the percentage of cars procured that meet their specifications. Concrete results are also more memorable and easier to implement. 

So if you are struggling to design measures for your organization, your first step should be to clarify what result you are trying to achieve, in concrete terms.

To learn more about developing concrete results or related measures, please look into one of our KPI training or certification programs or visit kpi.org.
123


Get our tips straight to your inbox, and 
become more strategy focused