"Poll" showing 73 percent approval for Mayor Lee was flawed

Mayor Ed Lee appears larger than life in the USF "poll."
Tim Daw

There was a poll conducted in late November by the University of San Francisco, the results of which were released in conjunction with the San Francisco Chronicle, claiming that 73 percent of San Franciscans approve of Mayor Ed Lee's performance.

It didn't take long for Lee's supporters to begin touting the figure as fact; soon after the poll appeared on SFGate.com on Dec. 9, the results wallpapered the comment section of the Guardian's website as the answer to any criticism of Mayor Lee, his policies, or the city’s eviction and gentrification crises. 

After all, it was a big number that seems to suggest widespread support. But closer analysis shows this "online poll" wasn’t really a credible poll, and that number is almost certainly way over-inflated. [Editor's update 1/13: The authors of this survey contest the conclusions of this article, and we have changed the word "bogus" in the original headline to "flawed." The issue of the reliability of opt-in online surveys is an evolving one, so while we stand by our conclusions in this article that the 73 percent approval figure is misleading and difficult to support, we urge you to read Professor Corey Cook's response here and our discussion of this issue in this week's Guardian.]

The problems with the USF “poll” are numerous, but the most glaring of those issues has to do with its lack of random selection. According to the New York Times Style Guide, a poll holds value in what's called a "probability sample," or the notion that it represents the beliefs of the larger citizenry.

The USF poll registered responses from 553 San Franciscans. That number itself isn't the issue, or it wouldn't be if those 553 individuals were procured through a random process. But they weren't, and it wasn't even close.

The survey participants were obtained via an "opt-in" list that, according to David Latterman — a USF professor, co-conductor of the poll, and downtown-friendly political consultant — meaning that anyone who participated in this particular poll had previously stated they were willing to participate in a poll. This phenomenon is known as self-selecting.

"We work with a rather large national firm and they have a whole series of opt-in panels," Latterman told the Guardian. "So they've got lists of thousands of people who have basically said, ‘Yes, we'll take a poll.’ And the blasts go out to these groups of people."

That means that even prior to conducting the poll, results had already been tailored toward a certain set of citizens and away from anything that could be classified as "random." And even the Chronicle acknowledged in the small type that “Poll respondents were more likely to be homeowners,” further narrowing the field down to one-third of city residents, and generally its most affuent third.

Even if pollsters could match the demographics of the polled with the "true demographics" as Latterman called them, it still wouldn't address the issue of self-selection. But that's not all: The list of "opt-in" participants, which was acquired through a third party vendor, according to Latterman, only contained English-speaking registered voters. And anyone contacted was contacted via email, another red flag in the world of accurate of polling data.

Interestingly, the USF “poll” also found that 86 percent of respondants said that lack of affordability was a major issue in the city, while 49.6 percent of that same group considered housing developers to be most at fault for the astronomical real estate prices. So, to recap: This poll, touted by many people as gospel in the comment section of this site, found that while the City is totally unaffordable, the man in charge of the City is barely culpable for that situation, and he remains incredibly popular.

According to the NYT Style Guide, "Any survey that relies on the ability and/or availability of respondents to access the Web and choose whether to participate is not representative and therefore not reliable." 

Uh oh. 

Russell D. Renka, professor of Political Science at Southeast Missouri State, conveyed far stronger feelings on the matter in his paper "The Good, the Bad, and the Ugly of Public Opinion Polling," saying that a self-selected sample "trashes the principle of random selection… A proper medical experiment never permits someone to choose whether to receive a medication rather than the placebo."

Strike two.

He then writes, "Any self-selected sample is basically worthless as a source of information about the population beyond itself."

Strike three.

So then why were such frowned-upon methods used in this poll?

Latterman attributes the tactics to many things, but mostly to the rapidly changing technological landscape of San Francisco, coupled with the high costs of alternative methods and a large renters market. 

"San Francisco is a more difficult model," Latterman said. "So Internet polling has to get better, because phone polling has gotten really expensive."

But even if Internet polling needs to improve, it is still important to prominently note that in original source material, lest you give folks the wrong ideas. Or even just misinformed ones. Unless what you’re trying to present is less about polling that trying to sell San Franciscans on the idea that Mayor Lee enjoys widespread support.






In fact I rarely meet anyone who hates him - at worst, it is indifference.

But given that there is nobody even remotely in the frame for mayor until lee is termed out after his re-election, isn't the issue moot?

In the real poll, 50% more SF'ers voted for Lee over Avalos. If that isn't a landslide, I do not know what is.

Posted by Guest on Jan. 10, 2014 @ 5:29 pm

In fact, rarely do I meet anyone who likes him. At best, they express indifference.

The people I talk to also say that winning an election with such a low voter urnout can hardly be called a landslide.

Posted by Guest III on Jan. 10, 2014 @ 8:01 pm

Even with RCV, if he were so popular (which is wasn't and is not) he would have won 51% of the votes in the first round and been declared the winner.

Instant-runoff voting: From Wikipedia:

"Ballots are initially distributed based on each elector's first preference. If a candidate secures more than half of votes cast, that candidate wins."

Lee did not.

Posted by Nonpartisan Guest on Jan. 10, 2014 @ 8:10 pm
Posted by Guest on Jan. 11, 2014 @ 9:44 am

They do if they are extraordinarily, exceedingly and extremely popular, as you like to pretend your god Lee is. Do you ever remove your head from your saviour's upper colon? It has got to be pretty rank in there by now.

Posted by Guest on Jan. 11, 2014 @ 6:13 pm

time around, so the moderate vote was split many ways.

Avalos had no serious competition on the left, so the initial rounds made him look good.

But Lee won in the end because far more people didn't want Avalos than didn't want Lee.

Posted by Guest on Jan. 11, 2014 @ 6:56 pm

They sound like objective people you talked with versus a Lee hack from the Chamber of Commerce or a shill for Ron Conway.

With a low voter turnout, no candidate even one I would support can be declared as having a "landslide." That's crazy and desperate thinking.

Posted by Guest on Jan. 10, 2014 @ 8:22 pm

50% more people voted for Lee over the anti-jobs candidate Avalos.

Posted by Guest on Jan. 11, 2014 @ 10:29 am

Please elaborate.

Posted by Guest III on Jan. 11, 2014 @ 12:57 pm

question about how he would create jobs. Insofar as he covered the topic at all, he talked about only public-sector jobs and extending the local hire rules.

I cannot recall a single thing Avalos said during his entire campaign that would have helped create a single extra private sector job. But he suggested many things that would have hurt business and jobs.

While Lee's top three topics were jobs, jobs and jobs.

Given that voters polled as regarding jobs as easily the number one issue, Avalos was doomed from the start. He would only talk to his own people i.e. unions and activists.

Posted by Guest on Jan. 11, 2014 @ 2:37 pm

What are the things he suggested that would have hurt businesses and jobs?

You m.o. is to write general claims that are unsubstantiated and insert rhetoric.

Posted by Guest III on Jan. 11, 2014 @ 5:44 pm
Posted by Guest on Jan. 11, 2014 @ 6:00 pm

You shouldn't have wasted your time asking. Probably another phony poll that Reed Nelson will expose as bogus in another couple of weeks or so, that that Lee sucker that you responded to fell for again. That Lee sucker that you responded to falls for anything if it's positive for their savior Lee. That person has got to be on someone's payroll or getting something out of it. Who would still be campaigning for Lee every single time there's an article published about him such as this?...other than some corrupt piece from the Chamber of Commerce, RonConway or some other corporatist pleb.

Posted by Nonpartisan Guest on Jan. 11, 2014 @ 6:30 pm

And there is no viable alternative for the next election so you are further away from ever from having a lefty mayor.

Suck on that.

Posted by Guest on Jan. 11, 2014 @ 6:57 pm

What a very mature response. It speaks so highly of Lee and the type of people he has supporting him.

Posted by Guest on Jan. 11, 2014 @ 8:58 pm
Posted by Guest on Jan. 12, 2014 @ 4:31 pm

lee is popular? are you 12 or are you on the payroll? do some research.

Posted by the Hun on Jan. 19, 2014 @ 10:20 am

Many are over 60% and one is over 70%.

So yes, Lee is popular. That is why and how he won in the first place.

Posted by Guest on Jan. 19, 2014 @ 10:45 am

so...what does that mean?? really? some misguided people checked a box so he is a good mayor?? lets just look at the facts of how this city is being compromised more and more. How Lee spreads his legs for big business, silicon valley, and taking away the livelihood of our cities cab drivers with his blatant nepotism. name one positive thing he has done that has actually BENEFITED the CITY!!!!!the entire city! not his daughter, or Google, or the Elite scumbags....but the CITY!!! good luck on your search for that .

Posted by the Hun on Jan. 19, 2014 @ 10:18 am

I'd like a pony too.

But that is meaningless since there is no coherent policy that anyone can offer to magically make SF homes affordable to most people.

It's an expensive town, and not everyone who would like to live here, can afford it.

Posted by Guest on Jan. 10, 2014 @ 5:31 pm

Steven, the citizenship test is conducted in English. If you cannot understand English, you cannot pass the citizenship test. And if you are not a citizen, you cannot vote anyway.

So, yeah, 99% of non-citizens in SF may hate Lee. But that is irrelevant because they do not have a vote.

Posted by Guest on Jan. 10, 2014 @ 5:36 pm

You do reaize that, given the city's huge Asian Pacific population, if the poll had included a Chinese language version Lee's ratings would have been higher, not lower.

Posted by Guest on Jan. 10, 2014 @ 8:38 pm

Asian and Pacific Islanders would vote for Lee just because he is Asian. (And many Asian and Pacific Islanders do not speak Cantonese / Mandarin or read Chinese.)

You are also assuming that a large percentage of Asian and Pacific Islanders are Democrats.


1 out of 10 Asian and Pacific Islanders I talk to like Lee. Most I have spoken to don't like him. At best, they express indifference.

And...many Chinese I know are Republicans.

Posted by Guest III on Jan. 10, 2014 @ 9:00 pm

Actually, Ed Lee is about as popular as Syphilis in the Ping Yuen housing projects at Stockton and Broadway.

Posted by Murphy Womg on Jan. 10, 2014 @ 9:05 pm

No, Ed Lee is more popular than Rose Pak.

Posted by marcos on Jan. 13, 2014 @ 11:44 am


Posted by Nonpartisan Guest on Jan. 10, 2014 @ 7:13 pm

Reed, thank you so much. That lays that to rest.

Posted by Nonpartisan Guest on Jan. 10, 2014 @ 7:14 pm

Test is all I've been able to get on this site today. Unable to comment.

Posted by Nonpartisan Guest on Jan. 10, 2014 @ 7:20 pm

Only trite one-liners seem to get through. Serious analysis gets blocked by the capctha thing.

Posted by Greg on Jan. 12, 2014 @ 9:13 pm
Posted by Guest on Jan. 13, 2014 @ 7:32 am


Posted by Nonpartisan Guest on Jan. 10, 2014 @ 7:54 pm

Another test.

Posted by Nonpartisan Guest on Jan. 10, 2014 @ 7:57 pm


Posted by Nonpartisan Guest on Jan. 10, 2014 @ 7:58 pm

Ed Lee is a puppet to Rose Pak and the entire 3 candidate voting is a sham. Shame on SF!

Posted by Guest Vince on Jan. 10, 2014 @ 8:00 pm

Responding to: Posted by Guest on Jan. 10, 2014 @ 5:36 pm

Steven did not write the article. Reed Nelson wrote the article. This phony poll should have been conducted in multiple languages, just as the San Francisco Registrar of Voters prints all of their voter paperwork in multiple, international languages for all eligible registered voters to use in their preferred language that they are most comfortable in.

Posted by Guest on Jan. 10, 2014 @ 8:37 pm

and therefore for voting.

Posted by Guest on Jan. 11, 2014 @ 9:46 am

The United States has no official language.

Posted by Guest on Jan. 11, 2014 @ 11:59 am

conducted in English and so, in practice, non-English speakers cannot become citizens (with an exemption for those over 70).

Posted by Guest on Jan. 11, 2014 @ 2:34 pm

That's why voting materials are printed in multiple languages.

Posted by Greg on Jan. 12, 2014 @ 5:39 pm

What do you expect from the "Lie Ed, Lie" team?

Posted by Guest on Jan. 11, 2014 @ 7:48 am

to my knowledge.

Posted by Guest on Jan. 11, 2014 @ 10:27 am

The Guardian would be better off if it spent more time electing people who support its policies to office and less trying to delegitimize those with which it disagrees. There hasn't been a progressive mayor for decades, no thanks to your lack of focus.

Posted by Guest on Jan. 11, 2014 @ 4:50 pm

That's why a moderate mayor always wins.

The rest is just hope and despair.

Posted by Guest on Jan. 11, 2014 @ 5:49 pm

In fact, I don't believe this City will ever elect a progressive mayor again.

San Francisco elected Gavin Newsom in 2003. They elected a raging douchebag socialite from Pacific Whites who was heavily tied to SF's billionaire elite and who gained political notoriety by promoting an anti-homeless agenda. And that was 11 years ago. Since that time, SF has become only more expensive, more gentrified and less progressive. The SFBG reports on this almost daily.

So considering the demographics have shifted even further towards the "moderate" base, I have no idea how in the hell you think the established moderate incumbant who already has the full Chinese vote on lock will ever lose. Delusional.

Posted by Guest on Jan. 11, 2014 @ 5:46 pm

And there is nothing that can be done about that

Posted by Guest on Jan. 11, 2014 @ 6:04 pm

I'm glad you put the word "moderate" in quotes. Even though they hide behind the word "moderate" to get elected here, it's time to stop this newspeak and bull shit of using the word "moderate" (just because they use it) to describe CONSERVATIVE politicians who come with a broad smile who hide behind the word "moderate" in order to get elected in San Francisco.

It's a tactic/strategy that has been used for some time here. It's time for everyone to stop falling for it and repeating it. Conservative is what it is and what they are. They are Corporatists. Just look at the rabid, conservative, smug, nasty supporters this man has on this site. There's nothing so-called "moderate" about them, or their rabid in-your-face-fuck-you-suck-on-this agenda and tone.

Posted by Guest on Jan. 11, 2014 @ 9:31 pm
Posted by Guest on Jan. 12, 2014 @ 4:32 pm

...for some town in Texas. San Francisco can do better.

Posted by Greg on Jan. 12, 2014 @ 9:54 pm

What has Texas done to deserve that piece of work? I wouldn't even force him on Topeka.

The man is not that bright to begin with. He just does what Rose, Willie and Ron tell him to do. He's their puppet, and an embarrassment to the city.

Posted by Nonpartisan Guest on Jan. 12, 2014 @ 10:22 pm

He is doing what he said he would do if we elected him.

Posted by Guest on Jan. 13, 2014 @ 7:31 am

Post new comment

The content of this field is kept private and will not be shown publicly.