Fun with numbers

In a comment, a reader raised concern about a CBS poll showing a Bush lead of 54-40 among veterans. I'm active in veteran's organizations and spend a substantial amount of my social time in the company of veterans, and that doesn't map to my anecdotal experience at all, so I raised some questions about the polling sample, writing

A poll like that raises some interesting questions about the sample. Did they examine the DD-214 of each respondent? Or are the 'veterans' they polled self-selecting? Or did they toss in a little regional bias to insure they got enough veterans in their net? Or what?

It turns out that the answer is 'or what?'

I took a look at the poll, headlined CBS Poll: Vets Favor Bush, and indeed, they made much of Bush's lead, not only in the overall preference, but on a wide range of issues. While there wasn't anything said about how they ID'd the veterans in the survey, there was this end note.

This poll was conducted among a nationwide random sample of 1,113 adults, interviewed by telephone May 20-23, 2004, including 170 veterans. The error due to sampling could be plus or minus three percentage points for results based on the entire sample.

What that note doesn't reveal is what the MOE might be for the small subset of veterans in the larger sample.

Thankfully, Atrios did the math, coming up with a +/- figure of 7.5%, meaning that any margin would have to be 15% to be statistically significant. In other words, every margin on every issue that CBS polled is statistically irrelevant, since the small sample size provided results within the MOE on every question.

Atrios, being a generous soul, notes that

...I don't think this is the sign of evil bias, just statistical stupidity. This is a standard error made by news organizations reporting on the poll results of subsamples.

I'm less inclined toward such a charitable interpretion, since CBS chose to spin the poll's primary import on the veteran's positions, burying the one number they report that is statistically significant.

Kerry is leading nationwide by a margin of 49-41, well above the overall poll's +/- 3% MOE.

Figures lie, liars figure, and CBS isn't exactly telling the truth, either.

MORE: Over at Salon, the folks in the War Room did a little figuring of their own.

Assuming that the sample was chosen randomly, there will always be an uncertainty that is n^(-1/2) [in other words, one over the square root of n], where "n" is the number of samples taken. ... Since there were 170 veteran respondents in this case, the square root is around 13. The reciprocal of that is about 0.077, which is 7.7%. So the margin of error for the veteran subsample is ± 7.7%.

In other words, the pollsters at CBS are either bigger fools or bigger liars (or some combination of the two) than Atrios' numbers imply.