World Statistics Day, Back to the Future, and the “Signal and the Noise” of TSL

Last week was a geek-fest.

First, World Statistics Day on the 20^th and then “Back to the Future Day” on the 21^st.  For me, last week also included an in-depth, somewhat geeky discussion with a station about Time Spent Listening.

That was a trifecta that begged for a blog.

While World Statistics Day didn’t go completely unnoticed, we were tripping over pieces on BTTF2 predictions that did or didn’t come to pass.

The focus on predictions, statistics and the TSL discussion reminded me of an excellent read from a few years back: Nate Silver’s The Signal and the Noise: Why Most Predictions Fail – But Some Don’t. It’s long on everyday examples from weather to baseball to poker and more. It’s deep enough to be challenging (especially if you’re like me – a non-statistician who like stats) yet it’s not so deep that you’ll drown.

Statistics play a role in most businesses; they certainly do in radio with ratings at the forefront for most programmers. 

Part of a consultant’s job is to provide the greatest understanding of the factors that impact/impacted ratings – not just to explain what happened, but to predict and recommend what changes (if any) could lead to better results.

The examination of ratings data is often the process of separating the signal from the noise: what factors were the primary drivers and what factors were ancillary or irrelevant.

To inspire you on your next analysis (ratings or otherwise), here are few quotes on signal and noise from Nate.

If you really want to understand something, you’re likely going to have to spend time under the hood. 

That’s not a new thought of course, but a good reminder that – especially where ratings are concerned – the deeper you dive, the more you’re likely to discover.

Silver suggests using past as well as collective experiences to form probability theories before diving into data.

“The Bayesian approach toward thinking…encourages us to hold a large number of hypotheses in our head at once, to think about them probabilistically, and to update them frequently when we come across new information that might be more or less consistent with them.”

Armed with a list of as many possible factors/scenarios that could have contributed to the outcome allows you to “stop and smell the data” which, Silver says, leads to better decision making – the reason you’re doing the deep dive in the first place.

A case in point was the Time Spent Listening discussion. The rise or fall of TSL may be related to your most obvious on air components like music, commercial load, or talent.  But it’s also very possible that any of the other 13 variables associated with TSL could be major factors, including 100+ QHR diaries, format partisans in the sample, weighting, how early your first cuming occasions occur, etc.

Before jumping to a conclusion about what drove an increase or decrease, examine each variable your scenarios suggest and determine 1) whether or not that variable was a factor and, 2) if so, to what degree.

As you work your way through the data, new information may challenge or strengthen your original assumptions.

As it turns out, the TSL drivers in the station discussion last week did ultimately prove to be something different than the original hypothesis. 

Trending data is a quick way to add context. Compare not only your most recent performance to past performances but also format averages, audience composition, sample, and any other relevant data.

And about those times when, to the best of your knowledge you did everything right, yet the outcome was disappointing?

Focusing on the process isn’t a “pass.” Instead, it’s an opportunity for self improvement and a review of the procedures that have been associated with success over the long term.

Bottom line: the next time you’re working through a report, seek to eliminate the noise by:

1. Committing the time it takes to do a deep dive
2. Creating multiple theories about what might have driven the results and a corresponding checklist of data to examine
3. Evaluating and providing context for all the data relevant to your theories
4. Review the process with an eye toward self-improvement 

As Nate points out, “Good innovators typically think very big and very small. New ideas are sometimes found in the most granular details of a problem where few others bother to look…sometimes we let information that might give us a competitive advantage slip through the cracks.”