Following the 2001 Major League Baseball season, Oakland A’s general manager Billy Beane found himself in a pickle.
Three of the A’s key players — Jason Giambi, Johnny Damon, and Jason Isringhausen — announced their departure from the ball club because their free-agency salary demands could not be met.
Losing three key players and with only a $44 million budget (the second lowest in the league), Beane was forced to make the most of the little money he had.
If you’re familiar with Michael Lewis’s bestselling book Moneyball: The Art of Winning an Unfair Game, you probably know what happens next.
Fed up with his inability to outbid other teams for good players, Beane reached out to Paul DePodesta, a Harvard alumni with a background in economics and a knack for baseball statistics.
Beane and DePodesta used data analytics to take a second look at how the A’s were scouting talent.
What they found was that baseball scouts were overlooking statistics that could accurately predict a player’s success.
Stolen bases, runs batted in, and batting average, typically used to evaluate players, were outdated and less important metrics compared to on-base percentage and slugging percentage.
Using this new approach, something they christened “Sabermetrics,” the A’s finished first in the American League West in 2002 with a 103-59 record, besting several big spending teams.
Before the A’s analytical approach to scouting players became ubiquitous among the major league, teams relied heavily on a scout’s gut feelings and hunches about players.
Moneyball received lots of scrutiny over devaluing the importance of baseball tradition. Pundits claimed that statistics could not show all of a player’s intangibles.
Nevertheless, Beane and the Oakland A’s strategy, which Michael Lewis popularized, quickly became adopted by several teams in the MLB.
The debate between analytics and experienced hunches will never cease to exist. However, there are some key lessons and takeaways we can learn from the value of understanding ‘the numbers’ in any business.
Julie Zhou, Facebook’s VP of design, recently wrote this article highlighting what you need to know about measuring goals and metrics in your business.
We’ve taken Julie’s 7 rules of thumb and explained each with relevant examples.
Rule #1 — To assess for product-market fit, look at retention
What Julie means is that if you want to see if your product or service is truly successful, the best metric to look at is how long people continue to use it or pay money for your product or service.
For example, in the first two days after Pokemon Go was launched, the game was downloaded on more than 5% of Android and iOS devices in the U.S., Australia, and New Zealand.
The game’s daily active users quickly matched Twitter’s numbers, but did so in only two days of existence (compared to almost a decade for Twitter)!
But the number of people who downloaded Pokemon Go is not the best indicator of the game’s success. It’s the number of people that use it regularly, and more importantly, make frequent in-app purchases, that prove whether it’s a hit.
Rule #2 — To optimize for growth, understand your funnel
Zhou says, “In order for people to become regular users of your product, they have to pass through a bunch of hurdles. First, they have to be aware of your product. Second, they have to be interested enough to check it out. Third, they have to convert (download an app, fill out a form, confirm e-mail, etc.) Fourth, they have to do enough within your product to understand why it might be valuable in their lives. Fifth, they have to remember to come back. At each of these steps, you will lose people. If you can track and measure what that rate of loss is, you can then start to figure out where to focus your efforts to make your funnel less leaky.”
Rule #3 — Figure out which metrics are truly important, and focus on those
At ETR, we once made the mistake of caring too much about the number of Facebook Likes we received. Sure, when we reached over 250,000 likes on the Turbulence Training fan page, it was a nice ego boost. But that was a vanity metric more than anything.
Rather than focus our attention on getting more people to like our fan pages, we doubled down on getting people to sign-up for our email lists and to become customers, which we know is more valuable in the long run, and as a side-benefit we grew our Facebook fan base as well.
“Recognize that most things don’t matter, and that only a small handful actually do. Don’t waste time talking about the unimportant stuff, and don’t sweat letting some of the less important metrics go up or down,” says Zhou.
Rule #4 — To figure out the best metric to track, use the magic-wandtechnique
Zhou suggests asking this one important question:
“If I could wave a magic wand and know anything about my users in the world, what would I most want to know to tell me whether my product will be successful?”
It doesn’t have to be something you can measure. For example, it could simply be, “Do the people who watch my videos actually take action on my suggestions?”
Your answers are “a helpful starting point to work from,” Zhou says, and from there you can figure out other questions to ask in order to figure out your best metric.
For example, perhaps your customers have to watch your entire video before taking action, so you could start measuring video completion rate. That might lead you to creating shorter and more entertaining videos to get more views. With more views, customers will take more action, have more success, and refer more people to your products.
This is a shift in mindset, and going through the magic-wand technique removes some of your team’s blind spots.
Rule #5 — Don’t just accept a metrics goal without understanding it
“I can’t emphasize this enough: the goals you and your team agree to will be hugely impactful to your work, so make sure you buy into them,” Zhou said.
But she also encourages you to challenge one another.
“Do not accept metric goals at face value. Ask why,” she says.
“Ponder whether or not they make sense, and what behaviors they will incentivize. Are there situations where something will feel like a good decision but the metric doesn’t move? Conversely, are there situations where you could imagine the metric going up a lot but not be convinced that the product is actually better? If so, would another metric (or set of metrics) do a better job of tracking what actually matters?”
The best example of this is click-through rate.
If I changed the title of this article to “Read This Article and You Are Guaranteed to Become a Millionaire,” more people will click to read it, but they will be very disappointed with me – and tar-and-feather me as a click-baiter – after the fact.
Which is why you need to follow rule #6…
Rule #6 — View data skeptically by suggesting counter metrics
This is one rule most people either skip or don’t consider in the first place.
“If the data is showing you what looks like good results, ask yourself: “What else can I look at to convince me that these results aren’t as good as they seem?”
Zhou calls these counter metrics, and believes every success metric should have some.
“For example,” Zhou writes, “don’t look at click-through rate without looking at the number of fast bounces back, don’t look at the sales numbers of a product without looking at how many returns or cancellations there are, etc.”
That’s why the click-bait approach mentioned above almost always backfires on the author.
“It’s much better to be paranoid about interpreting data so you can quickly catch your mistakes and adjust your strategy,” Zhou says. “Don’t fall into the trap of confirmation bias where you’re just looking for signals that prove your intuitions are right.”
Let me (Nick) give you a personal story to illustrate:
One of my recent fitness goals was to gain weight. Over the past two months, I’ve added 15 pounds. That’s a lot of weight in just eight weeks. The chances of having gained 15 pounds of pure muscle is slim to none. Some of that extra weight is fat. But since my goal right now is to increase my body weight, I’m fine with gaining some fat so long as I’m building strength and muscle mass as well.
The counter metric in my case is how much body fat I am gaining. If I was gaining more body fat than lean muscle mass without any increases in strength or stamina, I wouldn’t be happy, and I would need to adjust my calorie intake or change my training routine to compensate. I only know this if I am analyzing a counter metric.
Rule #7 — Use qualitative research to get at the why
“Quantitative data that tells you what people did is best paired with qualitative research that gives you insights into how people felt. Conduct usability testing, utilize focus groups, and run surveys to get at the why behind the behavior you’re seeing,” says Zhou.
In baseball, like in any business, you have teams that get along and teams that don’t. You might be able to field an entire ball club with the best batting and defensive statistics — increasing your probability of a winning season. But if some of your players aren’t the right fit, you could run into ego-clashing and a host of other problems that can ruin a season. This is why it’s important to weigh your quantitative data with qualitative data.
Take the temperature of your customers and team members with surveys, interviews, and as many interactions as possible to understand how they really feel. Combined with the right metrics, this approach will pay off in above average results.
It’s been fifteen years since Billy Beane and the Oakland A’s started the Moneyball trend. Today it’s more popular than ever, and has spread to many other sports. It’s almost overwhelming how many statistics have been invented to judge players – and make our fantasy sports leagues more entertaining.
But knowing your numbers in your business doesn’t need to be intimidating. It should be simple. Use Zhou’s rules to track the right metrics for your business, and you’ll experience significant growth in the numbers that matter, allowing you to do more of the right things and make more money.
About the Authors: Nick Papple is the Managing editor of Success Formula Daily and one of Early to Rise’s youngest up and comers. Nick works closely with editor Craig Ballantyne, curating content, writing and editing essays for the daily newsletter. Nick has a Bachelor of Science degree in Human Kinetics from the University of Guelph in Canada. Get Nick’s free newsletter Success Formula Daily sent straight to your inbox every weekday. Craig Ballantyne is the editor of Early to Rise (Join him on Facebook here) and author of The Perfect Day Formula. His straightforward, sometimes “politically-incorrect” advice has helped millions of people transform their lives both physically and financially. Craig’s secret weapons for success include his personal commandments, his 5 pillars, and his Perfect Life vision. Click here to learn more about how you can implement his 3 secrets to success and live your Perfect Days.