I missed this from Mid-December, but I can't help but comment...
Investor's Business Daily put out an editorial declaring their IBD/TIPP poll the best in the nation - again. They called the popular vote margin to the decimal in 2008 and were the closest (of by 0.3%) in 2004. I extend all due congratulations - particularly in an astute allocation of the undecideds, but I just love this link...
Well they certainly were unique in calling the 18-24 year old vote for McCain 74%-22% in late October. Campus was CRAZY that week...
Saturday, January 31, 2009
Saturday, January 24, 2009
Cell Phones and Surveys
Behind the Numbers ran a post by Jennifer Agiesta on cell phone users and surveys that caught my interest. She aggregated their tracking poll data and looked at the pool to study the effect of cell phones on sampling. She makes two fascinating statements that I've seen before, and I take issue with both:
1. The six versus nine point margin difference between the landline only sample and the complete one with cells is not statistically significant.
2. Weighting the landline sample to the age composition of the exit poll takes care of the discrepancy.
Both of these points are flawed. On the first point, I doubt that the difference is insignificant. The daily margin of error might be three points but these numbers are from aggregated tracking. The margin of error has to be pretty small. But even so the argument is flawed. If I didn't know the results but you asked me which candidate's supporters would be under-represented in a landline only sample, I would be a fool not to pick Obama. The margin of error is a function of the random and unbiased variation we can expect from the nature of proper sampling. Since we can predict the direction of effect of not sampling cell phones, it is specious to compare the size of the effect to the unbiased error we can expect to see. The margin works both ways. One day the sample might advantage the Democrat, and the next it could be the Republican. In a given election cycle, the cell vote does not change direction randomly as a bloc (per the currently accepted theories of the cell vote). Adding in the cell sample changed the margin of victory by three percentage points. That's a decent amount.
In sum, error is random, and bias is not. It is senseless to compare one to the other to downplay its importance.
The second point is a dangerous one. It leads to practices that cause inaccuracy. It trends toward the idea that undersampling and weighting to counter is always an acceptable way of correcting for a cell bias. First, putting a particular weight on cell user turnout is a tricky game. Age corrected the discrepancy for McCain-Obama, but what about other candidate pairs, issues and other poll questions? Second, it justifies undersampling that leads to absurd results. Check that link out. McCain beating Obama 74-22 in the 18-24 vote? There's no way.
1. The six versus nine point margin difference between the landline only sample and the complete one with cells is not statistically significant.
2. Weighting the landline sample to the age composition of the exit poll takes care of the discrepancy.
Both of these points are flawed. On the first point, I doubt that the difference is insignificant. The daily margin of error might be three points but these numbers are from aggregated tracking. The margin of error has to be pretty small. But even so the argument is flawed. If I didn't know the results but you asked me which candidate's supporters would be under-represented in a landline only sample, I would be a fool not to pick Obama. The margin of error is a function of the random and unbiased variation we can expect from the nature of proper sampling. Since we can predict the direction of effect of not sampling cell phones, it is specious to compare the size of the effect to the unbiased error we can expect to see. The margin works both ways. One day the sample might advantage the Democrat, and the next it could be the Republican. In a given election cycle, the cell vote does not change direction randomly as a bloc (per the currently accepted theories of the cell vote). Adding in the cell sample changed the margin of victory by three percentage points. That's a decent amount.
In sum, error is random, and bias is not. It is senseless to compare one to the other to downplay its importance.
The second point is a dangerous one. It leads to practices that cause inaccuracy. It trends toward the idea that undersampling and weighting to counter is always an acceptable way of correcting for a cell bias. First, putting a particular weight on cell user turnout is a tricky game. Age corrected the discrepancy for McCain-Obama, but what about other candidate pairs, issues and other poll questions? Second, it justifies undersampling that leads to absurd results. Check that link out. McCain beating Obama 74-22 in the 18-24 vote? There's no way.
Subscribe to:
Posts (Atom)