JOUR 271
Lecture outline: How to know if a poll is valid (follows lecture on writing a poll story)

Opening

• Distribute copies of the sample Middle Tennessee Poll story if students have not already seen it (and they should have). The results reported in the story are real. They are from the Spring 2000 Middle Tennessee Poll. Blake's quotes are real, too, as is the background information about Workman and Coe. The two other sources, Sage and Skinner, are fictitious.)
• Ask students to read the story and be prepared to discuss the following questions:
• How do we know that this is a valid poll – that is, a poll that faithfully represents the opinion of Middle Tennesseans?
• What exact questions were asked of respondents and what was the “response set” (the multiple-choice answers given to respondents)?
• How do we know that a representative sample of Middle Tennesseans was interviewed?
• How can we know whether the responses to each question are faithfully represented?
• Teacher’s note: Not all of these things need to be included in a story, but the reporter should know them.

Lecture & discussion

• Defining reliability and validity
• Reliability, strictly speaking, refers to any measurement that produces the same results under the same conditions each time a measurement is made. Thus, a thermometer that reports the same “normal” temperature each time you use it under “normal” conditions is reliable. But it may not be valid; that is, it may actually be off several degrees. A rifle that misses the target in exactly the same place when aimed at the bull’s-eye is very reliable. It is just not very valid.
• Validity refers to any measurement that produces that faithfully reports the “true” condition of the subject measured. To be valid, your thermometer must produce a reading that is accurate. One way of assuring this is to compare it with other thermometers of known reliability. Note, however, that a measure can be reasonably valid but also only fairly reliable. Here, you might imagine a gun that never hits the bull’s-eye but is always close, thought its shots are scattered around the center of the target.
• Validity and reliability in polls

• Like our gun that doesn’t always hit the same spot each time it is aimed, most poll questions vary a bit each time they are asked. Part of this is due to measurement error. (When you’re asked to tell me whether you approve of something “strongly” or “somewhat” and you feel that you are somewhere in between, you may pick one option one time and the other option the other. Thus, your opinion might appear to bounce around even when nothing has changed.)
• Part of the variation in poll results is also due to sampling error. That’s what all that “margin of error” stuff in the “nerd paragraph” of a polling story is all about. Any time we draw a random sample from a population, we know that it will never match the population exactly, but it will come within, say, +/- 5% of the true value 95% of the time.
• Another major source of error is question wording, the fact that asking a question in a different way may trigger a different response. For example, a question asking about whether Congress should increase spending for “the military,” for the “military-industrial establishment,” for the “Pentagon,” or for our “soldiers fighting for democracy abroad” may trigger different results.

• Sampling validity: How can only 1,000 people give a faithful picture of millions?
• Painting a picture of millions from only a few hundred interviews is possible because of random sampling. In a random sample, each element of a population is given an equal chance of inclusion; hence, the resulting sample is representation of the population as a whole with in the sampling error for the size of the sample (in polls, this is usually 5% or less). One way of producing a random sample is to mix up all elements of a population, then draw them lottery style (either manually or by computer).
• A sample does not have to be excessively large to represent a population faithfully. Imagine a chemist at Betty Crocker Foods trying to determine the secret ingredients in an award-winning cake from the Murfreesboro Women’s Club spring bakeoff. Just how big a piece of that cake (whose ingredients are randomized in the mixing process) does the chemist need. The whole cake? A big slice? A pinch? A few molecules?
• As the cake example suggests, as a sample gets progressively smaller, it doesn’t lose much accuracy until it gets infinitesimal. That’s why a sample of 500 can produce a +/- 4.5% margin of error, while 1,000 produces about 3% and 1,500 produces 2.5%. In fact, you must quadruple the sample size to cut the margin of error in half (and that’s expensive in polling).
• As the cake example also suggest, the accuracy of the sample has nothing to do with the size of the population. That is, it doesn’t matter whether we’ve drawn a sample from a cupcake, a standard cake, a wedding cake, or the world’s largest cake. The accuracy is the same. That’s why 1,500 people can produce a good portrait of a whole nation.
• To assure that a poll is valid, the communications expert (reporter, PR person, etc.) should ascertain whether the sample is randomly selected by standard methods such as random-digit dialing (the MTPoll uses this). Was this reported in the sample Middle Tennessee Poll story? On the MTPoll website?
• Another way to check the validity of the sample is to see how closely the sample approximates the demographics of the area, based on U.S. Census estimates. Was this reported in the sample Middle Tennessee Poll story? On the MTPoll website?
• Standard sampling methods include: 1) random drawings from a list, 2) drawing from an alphabetized list by adopting a randomly selected skip interval (every nth name), 3) computer generated random numbers (telephone numbers, social security numbers), and other procedures designed to make sure each element in a population receives an equal chance to be included.
• Discussion questions and exercises:
• How would you divide this class into two random groups? Here are two easy ways: 1) Put all names into a pot, shake it, and draw the names into two piles; 2) Go down the roll, assigning even names to one group and odd names to the other (there’s no known systematic bias to the alphabet). After this exercise, explain that sampling theory works well only with large numbers, say, over 100.
• How would you draw a random sample of Murfreesboro voters? Same as above, though you might select only every 5^th or 100^th name, depending on the size sample you want.
• How would you draw a random sample of listed Murfreesboro phone numbers? Same as above.
• How could you make sure you get unlisted numbers? Find out what percentage of the phones have what prefixes. Weight by prefix. Then have the computer spit out random 4-digit numbers to place after the prefixes. You’ll get unlisted numbers but also non-working ones as well.
• Questionnaire validity: How do we know the questions don’t bias the results?
• Some evidence is circumstantial. Who sponsored the poll? What organization conducted the poll for the sponsor? Does the sponsor have an ax to grind and is this visible in question wording? Is the polling organization recognized as reliable? Does the polling organization belong to and abide by the standards of the American Association of Public Opinion Research. Was the sponsor reported in the sample Middle Tennessee Poll story? On the MTPoll website?
• What exact wording was used in the questions? Can you detect bias? Ask the person issuing the poll to explain why questions were phrased the way they are.
• Have other pollsters used the same wording? What are the national results from such questions? What were the results from other comparable areas of the country? Ask the person issuing the report.
• What is the “response” set for the question? Is the response set “balanced” or “unbalanced” (excellent/fair/poor v. excellent/very good/good/poor)? Unbalanced response sets can easily be used to bias the results. The Tennessean reported that  “Sundquist's ratings drop below 50%” when the actual results were 46% “excellent,” 29% “fair,” and 14% “poor.” It’s hard to see how this is a rating “below 50%.” Click here to see the story.
• Are categories are collapsed? And are they collapsed fairly? For instance, “approve strongly/approve somewhat/disapprove somewhat/disapprove” strongly can be collapsed fairly into “approve/disapprove” to make understanding results simpler.  But can “excellent/good/fair/poor” be collapsed fairly? In a recent poll story about Metro Council, The Tennessean reported, “Poll indicates 67% unhappy with Metro Council.” The reporter collapsed “fair” and “poor” and considered this “disapproval.” But why shouldn’t fair be collapsed with “excellent” and “good,” making 83% approve of the council. Click here to see the story.
• Discussion questions and exercises:
• Try to come up with alternate questions to measure approval and disapproval of a political figure. See how what measures are used in PollingReport.com or in the Gallup Report.
• Kick around hypothetical ways of collapsing common non-poll categories such as shirt size: XXL, XL, L, M, and S. Ask members to list their shirt sizes. Then see how you can manipulate the results by collapsing categories differently.
• Pick a topic and see how various polling organizations have asked questions about the topic. Search the Lexis/Nexis database under Markets/RPOLL for various topics. Write a simple poll story based on the results.