You be the judge.
In 2004 the Radio Advertising Bureau investigating the technical aspects of PPM posed 188 questions (PDF) to Arbitron regarding PPM. Here’s one of them:
Question: What percentage of the transmitted codes is captured? What percentage with low volume?
Answer: In November of 2004 the PPM participated in a test where 33 radio stations were played in various environments and levels of “noise.” The PPM correctly identified 59% of the total radio sessions, in an environment where the goal was 50-70%.
The RAB seemed satisfied with the response. However, there’s no evidence that the RAB actually looked at the study Arbitron referenced.
And strictly speaking, Arbitron didn’t really answer the question. More on that after we look at the study.
The study was not conducted by Arbitron. The company played no role in it other than supplying the equipment.
Makes one wonder why the company didn’t cite its own research.
The study was funded by the U.K. radio community. We first reported on the study in 2010 because it was (and remains) the only publicly released study on PPM.
We personally spoke to the lead researcher and concluded the study was thorough, objective, and we believe well designed.
The investigators created over thirteen hundred different scenarios, testing various combinations of formats, radio listening levels, and interfering ambient noise levels.
They then divided the scenarios into categories based on how easily the meters should be able to identify the stations.
In the 28 cases where conditions were nearly perfect with the radio playing loudly in a quiet room where the meter was exposed and close to the radio, PPM got it right every time.
However, as soon as conditions dropped from perfect, success rates fell rapidly.
For example, the meter was able to identify less than three-quarters (73.4%) of the stations in the next two easiest scenarios.
At level 4, a level at which the researchers thought the meter should be able to identify the station, PPM only got 151 stations right out of 266 tries, a little better than half (56.8%) of the trials.
PPM identified only one quarter (24.0%) of the stations played in the most difficult three scenarios. These were where the stations were played at a low level in a noisy environment where the meter was tucked away in a bag or jacket.
So the response to RAB that PPM correctly identified 59% of the radio stations is somewhat misleading. In the most natural scenarios where real panelists are listening to the radio, the success rate could be considerably lower.
(It should be noted that in a comparison PPM alternatives proved no better, so PPM is not alone in failing to ID stations.)
But there’s a bigger issue here.
The only criterion used in the study was identifying the station. The sessions were ten minutes long, so the meter had ample time is identify the stations.
The Question “What percentage of transmitted codes is captured?” goes well beyond identifying a station.
The RAB was not asking what percentage of stations are identified. The question is about how often the decoding process works. And the longer it works, the more quarter-hours.
Arbitron cleverly dodged the question.
So based on Arbitron’s admission to RAB that PPM captures 59% of radio station IDs over a ten minute period, that means 41% of stations are unlikely to be identified in a ten minute period.
Let’s just round that to a quarter-hour.
Isn’t it curious that Arbitron/Nielsen report that average listening spans for all formats are ten minutes?
Is it because nearly half of stations remain unidentified after ten minutes? And is this why Nielsen cross-tuning reports show that most listeners don’t switch stations, they just disappear?
The bottom line is that we still don’t know how much listening PPM is missing. Looking more closely at the study Arbitron quoted to the RAB, it could be a lot.
We don’t think the Voltair controversy is going away any time soon.