Virginia's 5th District

TO: LAURENCE VERGA FOR CONGRESS (VA-5)
FROM: WILSON RESEARCH STRATEGIES
SUBJECT: BOYD’S VA-5 POLL
DATE: JANUARY 11, 2010

This memo summarizes methodological shortcomings and the questionable analysis provided by the Boyd campaign of a recent poll conducted in Virginia’s 5th Congressional District for GOP candidate and Albemarle County Board of Elections Chair Ken Boyd.

Summary

The methodology of this poll is at best suspect and at worst invalid as an indicator of primary voting preferences:

The question used to test ballot preference is not a valid or standard predictor of primary voting intent.

The lack of disclosure of the demographics of the sample makes it impossible to judge the potential bias of the sample and makes it, at best, suspect.

The failure to disclose something as simple as the name of the polling organization makes the results even more suspect as this is a very basic element of polling professionalism.

The Interactive Voice Response (IVR) methodology used to collect the data has been demonstrated to be unreliable, particularly in the hands of less experienced or less careful polling organizations.

Even if the poll is accurate, the data do not support the conclusions offered. In fact, they support the conclusion that the Republican primary in the 5th district is wide open and no candidate has a particularly sustainable advantage.

Detailed Findings:

Based on the information provided in Boyd’s campaign email we find the following methodological inconsistencies with this poll:

The survey was conducted using a methodology–Interactive Voice Response (IVR) methodology—that has been proven to yield unreliable results.

o While some highly professional polling organizations have successfully employed this methodology, studies of all IVR polls released in previous election cycles have demonstrated that on average it is prone to significant errors and variability not explained by the typical Margin of Error associated with a poll.

o One particular concern with IVR polling in a primary election is respondent confirmation in households where not all registered voters are Republican primary voters. In a live interview poll there is both the obvious confirmation of respondent gender and the “social cost” of lying to an interviewer about one’s identity. Neither of these is present in an IVR poll.

o In addition, the information released by the Boyd campaign does not reveal any details about the methodology used to confirm that the respondents were indeed the targeted Republican primary voters.

It is likely that the pollster who collected this data is less than reputable as their name is not disclosed in the release in clear contravention of standards of the profession.

o The name of the polling organization is a required item of disclosure for both of the professional organizations governing polling (AAPOR, the American Association for Public Opinion Research, see http://www.aapor.org/Disclosure_Standards.htm, and the National Council on Public Polls, see http://ncpp.org/?q=node/19).

Because they risk serious censure for violating these requirements, reputable polling firms would not allow data to be released without this disclosure.

No demographic information is provided to demonstrate that the sample is representative.

o The information released about this poll does not provide any information about respondent demography. We do not know the number of men and women interviewed, their ages, or where in the district they live.

o Demographic characteristics of the sample and the degree to which they reflect the likely primary electorate are the simplest test for bias in a poll and without these data no one should accept that the findings reported for this poll as accurate.

The “ballot test” reported by Boyd is not a standard ballot test format for polling and is does not satisfy even a simple test of validity as a measure of primary voting plans.

o In polling it is standard to ask the question about intended ballot choice using a phrase such as “if the Republican primary election for U.S. Congress were held today, for whom would you vote if the candidates were…”

o The question asked by the Boyd campaign does not mention the Republican primary, does not ask how the respondent intends to vote, and introduces a possible bias by first mentioning the name of the Democratic Congressman.

o The first, and simplest to meet, test of validity for a measurement is “construct validity” which simply means that the question is linked to the behavior it purports to measure. This survey question fails even that relatively simple test.
Even if we accept for argument’s sake that the poll is valid and unbiased, the data do not support the conclusion that Boyd attempts to draw.

o Nearly two-thirds of the sample remains undecided in the race, more than enough for any candidate to win.

o There is no measure of intensity of commitment, which coupled with Boyd’s and Hurt’s likely name ID advantages, would suggest that their support is “soft” and even more than the 62% of voters reported in the poll as “undecided” are “in play.”

About Wilson Research Strategies

Since 1998, WRS has been a leading provider of political polling for campaigns from Mayor and City Council to Governor and U.S. Senate in 47 states and several foreign countries. In 2007-2008 alone, WRS conducted polling in 252 races for campaigns, caucuses and independent expenditures efforts.

In addition to our political and policy research, WRS provides donor research to Christian and other not-for-profits and alumni research to colleges and universities. More than 200 Christian and other not-for-profit organizations around the country and dozens of large and small colleges and universities have relied on WRS’s data and analysis.

WRS’s corporate research arm has provided market research to more than 100 of the Fortune 500 and to hundreds of small and medium businesses nationwide.