Tuesday, October 17, 2006

Updating Brading/Minnis Poll--It's Legit

Carla posted the press release from the House Democrats yesterday, noting with delight that for all her spending profligacy, House Speaker Karen Minnis can't get a lead outside the margin of error against Rob Brading. Knowing I'm the stat wonk of the pair, she deferred to me to discuss the weight and integrity of the poll, which I'm happy to do.

When I saw the numbers showing Minnis "in the lead" by 45-42, and then saw that the poll was conducted for the House Dems by Amy Simon of Goodwin Simon Victoria Research, I took a step back. "Internal poll--ugh," I said. After all the time I've spent railing about partisan flimflammers like Bob Moore, how would it look if I uncritically accepted an internal poll from the Democrats? Biased, is how it would look. So rather than take the numbers on faith, I sought to check them out and see just how serious GSVR was about conducting their polls.

There are several potentially major pitfalls when it comes to internal polls, that prevent them from being reliable, scientific indicators of popular opinion:
  • A non-random sample, perhaps culled from a mailing list or even a focus on certain precincts;
  • A poor method for determining which respondents are likely to actually vote;
  • A sample that skews towards one party affiliation over another, compared to known party registration figures;
  • Placing the "poll test" question after other questions that may influence the poll test response--particularly when those questions are intentionally leading.
I picked up the phone and immediately found myself in contact with Simon herself. She was very open and happy to explain her methodology and peripheral information--as any respected member of the polling community is. That's an excellent rule of thumb: if they don't include sufficent information about their methods, and won't tell you what they are, be skeptical of the results unless replicated elsewhere.

I began to ask questions based on the pitfalls above, and got these answers:
  • The sample universe was all registered voters in HD49, so that kills pitfall #1--all eligible respondents were available to be chosen for the survey.
  • According to Simon, respondents were asked to characterize their likelihood of voting in the next election. Those answering "definitely will" or "probably will" on a scale of 1 to 5 (where 5 is "definitely will") are considered likely; the rest are tossed.
  • The poll test question was relatively "up front;" it followed only the likely voter screen, a favorable/unfavorable question for both candidates, and a re-elect question for Minnis.
The final potential pitfall, party affiliation, takes a little more explanation than a bullet point. The registration pattern in District 49 (as of July; hopefully we'll get final reg numbers soon now that the window for registering is closing) shows a Democratic edge of a couple thousand voters. Democrats make up 40.6%, Republicans 32%, unaffiliateds 23.4%, and the rest for minor parties.

Simon's full sample of registered voters was fairly close to the actual breakdown: 42/34/21. Sometimes the discrepancy on decent polling can miss the mark by as much as four points, so a two-point variance is fine. Once the likely voter screen was performed, as expected the unaffiliated voter proportion dropped off to only 16%--which makes sense, since non-affiliated voters are less likely to vote or be interested in voting. Democrats made up 47% of likely voters, with 35% claiming a Republican affiliation.

What does this mean? Clearly, Democratic voters are more fired up about turning in their ballots--and as long as they do, Brading has a good chance of victory, even without the independents. Of course, how those voters will break down may be key in a close election. But the bottom line is that Simon's poll passes each of the tests I threw in front of it: it's a random sample, the likely voter screen is sensible and doesn't discard previous non-voters, the poll test question comes early in the survey without leading questions beforehand, and the base sample is highly reflective of the actual registration breakdown.

Conclusion? There is a 95% probability that Karen Minnis is capturing less than 50% of the vote (since she's at 45% and the margin of error is less than 5%), and that Brading is close enough to win it. Not bad for an internal poll the Democrats weren't even planning on releasing!