When is a public opinion poll a valid representation of how people feel? That turns out to be a tricky and ever-evolving question, particularly in San Francisco — thanks to its prevalence of tenants and technology — and even more particularly when it concerns the approval rating of Mayor Ed Lee.
Traditionally, the central requirements for public opinion polls to be considered valid is that respondents need to be representative of the larger population and they need to be selected at random. Polls are often skewed when people need to opt-in, as is the case in most online polls.
So the Guardian took issue with claims that 73 percent of voters approve of the job that Mayor Lee is doing, a figure derived from an opt-in online poll focused on “Affordability and Tech” that was conducted by University of San Francisco Professors Corey Cook and David Latterman and released to the San Francisco Chronicle on Dec. 9. That figure quickly wallpapered the comment section of the Guardian’s website as the answer to any criticism of Mayor Lee, his policies, or the city’s eviction and gentrification crises.
“Any survey that relies on the ability and/or availability of respondents to access the Web and choose whether to participate is not representative and therefore not reliable,” is how The New York Times Style Guide explains that newspaper’s refusal to run such polls, a quote we used in our Jan. 10 Politics blog post on the subject, and we quoted an academic making a similar point.
We also interviewed and quoted Latterman discussing the challenges of doing accurate and economical polling in a city with so many renters (64 percent of city residents) and so few telephone landlines. “San Francisco is a more difficult model,” Latterman told us. “So Internet polling has to get better, because phone polling has gotten really expensive.”
So we ran our story dubbing the poll “bogus” — and the next day got angry messages from Cook and Latterman defending the poll and educating us on efforts within academia to craft opt-in online polls that are as credible as traditional telephone polls.
“The author is so quick to dismiss the findings of the study, which is based upon accepted methodology, and which had nothing to do with mayoral approval scores, that he actually misses the entire thrust of the study — that voters in San Francisco are deeply ambivalent about the current environment, concerned about the affordability crisis, and not trusting of local government to come up with a solution,” Cook wrote in a rebuttal we published Jan. 13 on the Politics blog.
Cook told us the survey’s methods are endorsed by the National Science Foundation and peer-reviewed academic papers, including a Harvard University study called “Does Survey Mode Still Matter?” that concludes “a carefully executed opt-in Internet panel produces estimates that are as accurate as a telephone survey.”
That study went to great lengths to create a sample group that was representative of the larger population, while Cook and Latterman both admit that their survey’s respondents had a disproportionate number of homeowners. But they say the results were then weighted to compensate for that and they stand by the accuracy of their work.
Yet Cook also notes that the mayoral approval rating number wasn’t even part of the package they developed from this survey, it was just a finding that they decided to give the Chronicle. “I don’t think the 73 percent means anything,” Cook told us, noting that snapshot in time doesn’t reflect Lee’s actual popularity going forward, despite how Lee supporters focused on it. “The number they use politically is not a meaningful number.”
What Cook found more significant is the “tepid support” for Lee indicated by the poll, including the 86 percent that expressed concern about affordability in the city, a concern that cuts across all demographic groups. Most respondents had little faith in City Hall to address the problem and many felt the tech industry should be doing more to help, particularly companies that have received tax breaks.