As radio moves inexorably towards posting, it might be a good time to review some fundamental realities about Arbitron ratings. The most fundamental fact that we need to keep in mind is that Arbitron ratings are estimates, estimates that are subject to error and imprecision.
The second fundamental reality that we need to keep in mind is that the in-tabs we have now are not based on any arcane statistical reliability calculation or formula. In-tabs were essentially negotiated, Arbitron trying the keep them low and broadcasters trying to raise them as high as possible. The current negotiated in-tabs are too low creating considerable imprecision and inconsistency.
Neither Arbitron nor broadcasters are willing to spend enough to make ratings significantly more reliable. So it is the financial decision maximizing Arbitron's profit margin while minimizing what radio stations pay that creates the wobbles and spikes that every station experiences. It is this "slop" in the numbers that causes us to look like geniuses one book and idiots the next.
There is no reasonable statistical explanation for why New York ratings are based on 12,000 diaries and Atlanta ratings are based on fewer than 4,000. For the Winter book the in-tabs for New York, Los Angeles, and Chicago were 12134, 7210, and 6212 respectively. From a statistical perspective, this makes no sense. The relationships between population, in-tabs, and margins of error are not linear. Just because Atlanta is half the size of Chicago, we can’t cut the in-tab in half for Atlanta and expect the same level of accuracy. Because the negotiated in-tabs drop more or less linearly with population, the precision of Arbitron ratings plummet as we get beyond the top few markets.
Think of Arbitron ratings as a game of darts where the thrower is blind-folded. Every throw ends up in some different section of the target. Once in a while the dart may hit the bull's eye, but it is just as likely to hit the outer ring. Even with too few diaries, the ratings can still be exactly correct once in a while. A book might hit every share, rank, and cume exactly as it should be. But there’s an equal chance that the next book every share, rank, and cume will be wrong and possibly very wrong. We need to keep that in mind when we start guaranteeing rating points.
PPM may seem like the answer to posting concerns. For reasons we'll explore in a future post, PPM may actually cause more posting problems than the diary method. PPM was pitched to broadcasters as more accurate than the diary method. While that seems a reasonable assertion, the whole notion of more accurate ratings means nothing unless we agree on a definition of accuracy and then offer proof that PPM is more accurate. And Arbitron simply cannot prove that PPM is more accurate. Arbitron also promised more precise ratings with PPM, and thus far, even that hasn’t turned out to be the case. Apparently, Arbitron’s dart thrower is still blind-folded.