Jordan: It’s an uncomfortable feeling to realize that a machine can probably do your job. Last week a company launched a new podcast service the claims it can synthesize the hosts voice and have them read whatever is tight. So I’m familiar with this feeling right now, but for half a century or so as our habits have changed in our communication methods have evolved and technology has taken over. We have not yet found a better way to gauge people’s political opinions. And the old-fashioned poll calling people up asking them what they think and how they’ll vote and doing it until you have a representative sample internet: Has not fared much better automated robocalls have not topped it either and honestly polls are still mostly accurate despite what you hear but what if as accurate as poll can be they are still flawed but if part of that flaw is that you have to actually ask people to take a poll. So they know what their opinions are for. And what if you can probably guess where this is going you could design a computer program that can find people’s real opinions because you’re not doing it by asking them. You’re just compiling all the opinions that they share for free every second of every day. What if you called that program Polly and what if she was better than the. I’m Jordan Heath Rawlings, and this is the big story Max Fawcett is a writer at many Publications across Canada. He wrote a piece about Polly for the walrus. Hi Max.
Max: Hey, Jordan.
Jordan: Why don’t you start by telling me where the narrative that polls are crap came from where did that kind of start?
Max: It’s a funny thing because poles to some degree of always been crap. I mean if you think back to the you know, the famous 1948 American election where they had Dewey defeats Truman on the front page, right? This is not a new phenomenon, but I think. In the last decade there have been a number of elections in this country and then particularly in America in 2016 where people were suddenly awoken to the fact that polls were not a predictive and perfectly accurate tool. I mean, I remember I had a bunch of friends who we were watching the election together in 2016. And you had the New York Times little prediction arrow that traumatized everyone and oh God. Yeah and it had you know, it’s pointing to Clinton and we all had okay, the polls are right and then all of a sudden it’s swung to Trump and there was a sort of collective moment of. Like disbelief of what how could this have been? We know we’re all told that it was just a matter of her winning by a large number or a small number and and I think that kind of really lit the fire in the public Consciousness, you know, they’ve been big misses. In Canada before I’m thinking of the Alberta provincial election where Daniel Smith was predicted to be the first non conser non-pc government in 40 years, and she did not come close. It was a, you know, almost a landslide re-election for Alison Redford in the PCS. There was a big Miss in BC. There’s a Miss in Ontario. Even the Quebec election recently. There was there was a miss so all these misses start to add up and I think people quite rightly start to ask questions about whether this is something that can be relied on are there more mrs.
Jordan: Now or do we just hear about the more because we hear about polls globally and we pay attention to politics globally.
Max: That’s an interesting question. So Nate silver who sort of the The Godfather of the polling industry. He has this blog in America. Called 538 is a King Of The Nerds and I worship Him appropriately because he’s a baseball guy. But you know, he came up with this very fiery piece after 2016 saying basically the polls are fine. They’re as right or as wrong as they’ve ever been problem is in the way the media is talking about them and that the way the media is representing the level of accuracy and I have some sympathy for that because. You know, I’m a former journalist, but I’m also a former math teacher and most journalists don’t have a real strong sense of numeracy, right?
Jordan: Yeah, that’s where Silver’s entire career comes from the fact that we don’t understand what he does
Max: exactly and he does and then he sort of has built up a brand around bringing journalist on who can explain it understand these things. So I think there’s definitely some truth to the fact that the media has a role to play in. Exaggerating the degree of wrongness in the polls. I think the other part of it is just what was at stake in 2016. And interesting thing is that the polls in 2016 were actually more accurate than they were in 2012 in the presidential election. The in terms of the percentages that were predicted in the popular vote. And then the the result the problem is that in 2016. They got it wrong. So bit smaller are but they got the winner wrong in 2012. They did say Obama was going to win that they didn’t think he was going to win by nearly as much as he did we’re fine with that. I think is the public. I think what we really want is for them to get the people right that we don’t really care if you stick the landing on on the percentages. We just want to know that when you say candidate X is going to win candidate X actually does win.
Jordan: You began by saying the polls have always been crap. So why are the poles crap what’s wrong with opinion polling in is it changing?
Max: So it is changing II and I’ll get to that in a second. I do want to say I don’t think that the poles are necessarily crap, but the you know, the polls are imperfect and all the pollsters I talk to are really clear but emphasizing that that look we’re not magicians. This isn’t a physical science where we can you know, go into a laboratory and measure chemicals or way. For things like that. We’re trying to predict how human beings will behave and we can barely predict how we behave as individuals when we get up in the morning. So to predict how all of us will behave down to a level of precision they pull on his is phenomenal. I think you know in some respects. It’s amazing. They can even get in the neighbourhood part of the problem is that they have with the media’s help to some degree created a set of expectations that aren’t realistic. Right? So polling is a competitive industry. It’s a business. Do you know they’re not doing it for fun or. Free well, they’re doing it for fun. They’re not doing it for free. And so when they get something right, it’s naturally their instinct to brag it up and to go look at us. We nailed the Ontario election to within one percentage point and then when the other guys get it, right they brag it up. And so there’s this kind of nuclear arms race of bragging where I think it takes people to a place where they expect all the poles to be right all the time. Of course, they never brag up the misses. Right, they sweep those under the rug and forget. They never existed because it’s not good for business and understandable. So there, you know the dialogue around polling. I think creates a level of expectation that can’t be lived up to and I think that’s what’s so interesting about the company advance symboliks and their their AI technology is they’re saying we think we can turn this into a scientific operation or scientific exercise that we can actually replicate the kinds of results that you get in a scientific trial. Big boast but it’s you know, some respects one that the polling industry has set itself up for.
Jordan: So tell me what Polly is
Max: So Polly is an algorithm that traces its roots back to Los Alamos and the the jet propulsion laboratory there and you know, the Manhattan Project there’s these really sort of smartest man in the world people and. There was a riff on that algorithm that basically allows the people at Advanced symboliks to create what’s known as a sample. So all polls are built around samples, right you you talked to a thousand people and it is representative of the whole country just kind of amazing but that is at the heart of polling is the sample and the problem with traditional polling is number one. We don’t answer our phones anymore right back in the 80s back in the 90s, you know, we people if. Got a phone call you answer it you didn’t have caller ID. You didn’t have the internet. You probably were a little less busy. So yeah, I’ll answer
Jordan: now they’re calling to clean my air ducts and I won’t even pick it up unless I know the name
Max: we barely will barely answer phone calls from our own parents. And so good luck. If you’re a random polling company trying to get someone’s opinion. So, you know response rates that were in the you know, the the 15 20 25 30 % back in the day have gone down to 6% which sent which makes it very hard to get. The random sample that you need to have an accurate poll because you want a sample that looks like the country and it’s really hard when no one answer the phones right? You get a lot of older people you get a lot of a certain kind of person but you don’t get young people. You don’t tend to get new Canadians. It’s not good and so pollsters the way they’ve worked around that is they do what’s called waiting where they basically you need to talk to 20 young people and you only get four will you just. Amplify their voice right and make each one of them count for five. Yeah, you make them count for a little more than they should and that’s where things can get really dicey because if you talk to only for young people and let’s say those four young people are not like most young people and you wait up their response. Your poll is garbage and what Advanced Symbolics does is it says rather than trying to find people call them get in touch with them. Why don’t we go where they are? Right and they’re on the Internet. It’s where all of us are so they go to Twitter. They go to Instagram. They go to the social media networks, and they just look at what people are saying and that’s where they basically construct their public opinion. So they take their sample. And this is where the algorithm comes in from people who are on social media is it generates a very accurate and large sample? And that’s the magic of it. Is that polling traditionally you only really talk to about a thousand people because beyond that the way the margin of error the math behind it works if you double the number of people you talk to your not doubling the accuracy, it’s diminishing. So pollsters being business people at the end of the day. They don’t want to keep calling people. They stop it around a thousand cuz that’s where you get good good accuracy and it’s not too expensive. Polly can can find 10,000 people her sample is so much bigger and she can look at what those people have said in the past about certain things because obviously all of our Twitter histories are. Public to varying degrees so she has a really good sense of the way you talk about an issue and how it’s perhaps changed over time. And that’s what she uses to figure out how people are feeling doesn’t ask questions. That’s the really interesting and kind of revolutionary part. There are no questions asked their no survey sent out. It’s simply looking at what people have said and how they’re talking about it now and drawing certain conclusions around what that means for a question.
Jordan: Give me some real world examples of things that Polly has been used for the. She got right where maybe traditional poles would have been off.
Max: So the big one is Brexit back in 2016 is a really good example of where she kind of broke from the crowd conventional wisdom among most pollsters was that remain was going to win it was going to be close but the leave campaign didn’t have didn’t have the horses and it didn’t turn out that way. Obviously, you know, you was a very much a traumatic year for polling because you had the Brexit mess which horrified. Millions of people in Britain then had the Trump miss that horrified millions of people around the world but also in America and she didn’t miss she saw that leave was actually going to win and it’s interesting it wasn’t that she saw it the whole way she was with most pollsters for most of the campaign. She thought remain would win it would be close, but they would win. And then there was a horrible incident where a labor Member of Parliament was was assassinated by a right-wing radical and the Prime Minister David Cameron called a halt to the campaign for a few days. So everyone needs to cool off lets, you know, no more politics. Let’s take care of what’s important and everything went back to campaigning and people went back to polling Polly noticed that something has shifted. It’s interesting. It wasn’t that people were talking about things differently. It wasn’t that she all of a sudden noticed people who are remain were now leave. She noticed a pattern in the way people talked in the past. So whenever a poll would come out that had, you know, remain leading it would be followed by a surge in leave voters. It was sort of like this weird back and forth and the timing of it was such that. The way the election date lined up that surged lined up perfectly with the voting day. So she she figured that out. I don’t know how and said nope now leaves going to win because of the way this is time to up and I talked to the guy who you know created her and is the the genius behind it. And he said look if the vote had been held a week later remain probably would have won because this surge in leave feeling would have died out the way it had the last two or three times.
Jordan: Wow, so tell me about where Polly came from who made her
Max: So Polly is is the creation of a guy named Kenton white who is a mathematician scientist genius very soft-spoken. He’s a very gentle but very confident that he has figured something out that the rest of the world needs to pay attention to so he you know, he had started different businesses in this field before and and you know, the late 2000s he sold his last one and this was right around when social media was kind of taking. Oft you know Twitter and Facebook and he got really interested in it. He got really interested in this idea of having a window into how millions of people were talking. About how they felt about you know any number of issues and he thought it represented an interesting opportunity to use that to gather information not in like a Cambridge analytical creepy way where you know people’s information was being harvested but just you know as a someone who is a data researcher who thinks a lot about these sorts of mathematical things just this new data set was so intriguing and people tend to be a little more honest on social media than they are when. a Pollster calls them and says how do you feel about same-sex marriage or how do you feel about immigration where you know people may know that their views are outside the mainstream and so they won’t want to talk about it little different on social media.
Jordan: I’m fascinated by how Hollywood pull real responses out of the social media Aether like how do you not end up with? Parts of this sample including Russian Bots as the common parlance would take it.
Max: Well, so part of it is that the way the algorithm that is the engine of Polly’s public opinion research it throws, you know accounts and people out of the sample if they don’t meet certain criteria, you know, if your if your user name is galactic Defender six, two four nine seven. You’re not getting into our sample because I don’t really know where you are where you’re from. You could be a bot all that kind of stuff. So they’re very careful. They apply all these different again mathematical tests that for the math wonks out there. This is all available in publicly. Published papers. So if you know if you want to go a mile deep on this it’s there for people but you know, the the Coles notes version is they apply a bunch of tests to their sample to make sure that the people that are in it are real they live in the place. There’s they say they live in and they’re not they’re not there to manipulator or misrepresent their opinions. And because there’s so many people on social media and you know again, this is a number that grows every day their sample gets better and better. That’s one of the interesting sort of disconnects between traditional polling. And what poly does is their sample gets harder to tap into and smaller. Every year hers gets bigger. They’re sealing is her floor to use a sports metaphor. Right? She has a lot of upside. They have a lot of downside
Jordan: isn’t there a particular type or types? I guess of people who discuss Politics on social media. We always hear about how divisive the conversation is, and how polarized it is on there. So if you’re sampling. Discuss Politics on social media. Are you not just going to get the loudest voices in the room?
Max: You will and I think one of the interesting things that Canton said what I talked to him was that’s not necessarily a problem. You want people who are engaged and passionate because they are the most likely to vote number one, right and they are you know, the most likely to be more representative of their particular, you know, you know, whatever they represent in the sample then someone who is indifferent. In different, you know in difference is hard to model. Whereas people who are a little more engaged you have more certainty about. Well, you know when they say they’re going to do X they probably are going to do X, you know, there’s certainly not at a point yet where. Their their work is totally predictive. They’ve made mistakes. They whiffed a little bit on, you know, the recent Ontario election where they said the Liberals were going to get shut out and they didn’t and so they’ve you know, they calibrate Polly to learn from that mistake, but their work I think is it’s an interesting challenge to the industry, you know, when I talk to people and about their work, the response is ranged from Total skepticism to more sort of careful watching of yeah, this is something we need to be. Mindful of me. I think that’s the interesting thing about the people who work in the polling industry as they’re very curious like a lot of scientists. They’re not they’re not ideologues. If there’s a better way to do things. They will do it that better way. I think they just need to be shown that it’s better and some of them remain unconvinced at this.
Jordan: That was going to be my next question is how how do you marry the tenets of traditional polling with new technology that can help it? And is there a way to do that in would that be better or should we just throw out the old way and try to perfect the new its interests?
Max: Yeah, I mean, it’s interesting. I don’t think we need to throw out the old way because I think the old way. Still has value.
Jordan: I think it’s good. It creates a lot of content for us
Max: it well. Well, I mean don’t don’t get pollsters started about the media taking any poll they can get because it creates content because it does drive a lot of them crazy in the media were so starved for let’s call it low-cost content that when someone throws you a poll that says something, you know unexpected. You’re probably going to run it even if you may be know, it’s not. You know a great Pole or even if you don’t know it’s not okay
Jordan: outliers are good for business and many places.
Max: That’s true. As you know, as long as people are talking about it and you know, there’s been much internal discussion in the polling industry about what to do about that right how to how to get quote unquote Bad actors sort of in line, but it’s It’s tricky because there’s sort of a sort of a prisoner’s dilemma where the Bad actors aren’t really that inclined to cooperate with everyone else because their Works getting picked up and maybe they’re getting paid. So, you know, I think the interesting thing that pollsters do is not just ask the questions they do the analysis. And when I talked to a lot of the you know, the veteran pollsters in the industry, they’re really clear with me about that. It’s not just sending out surveys and collecting results and putting out tables with percentage points in it. It’s explaining what it means and drawing those those dotted lines between 64 percent of Canadians care about climate change to this is what this means for the Federal election right. It’s providing that analysis function and that’s not really something that an algorithm can do.
Jordan: Well what kinds of questions can Polly asked because I can imagine. I mean like you said she’s not asking questions, but what kind of questions can she answer traditional polls are great for the horse race questions, but they can also get at a whole bunch of other things pertaining to issues around the Race pertaining to how you would rank. Issues on your ballot right? Are you more likely to vote on climate change versus a tax cut etcetera Etc. And can she get into that kind of stuff or is she analyzing kind of either or Pro against
Max: I’m very reticent to say there are things she can’t do because I’m fairly sure that people who work there will take that as an opportunity to prove me wrong. But at this point the what she can do the why is harder? Yeah because that’s more discursive. It’s more of a conversation. You know, she like it’s interesting. They don’t really they’re not. Passionate about public opinion polling around politics over at events symbolics. They’re not they don’t love politics they do it because it’s the best way to prove their concept right? It’s a thing that is very visible and it’s a thing we all understand and it’s also a thing you can measure I say it’s going to be twenty six percent liberal and it turns out to be 26% liberal. I’ve shown you my formula works, but where they do much more of their research is. I mean there’s a lot of commercial stuff, you know corporations want to find out what are people saying and how much do they like our product? Social media having that sample of people on social media is a very good way to do that, you know Health Canada after you know a school shooting or something are there. Is there a spike in in talk of of self-harm or suicide they worked with health Canada to kind of get ahead of that and and having that information they could then dispatch resources to the school districts and questions. So, you know, there’s some really interesting applications there where it’s not just about the horse race. It’s about giving policymakers. Better information in a more timely fashion so they can make better decisions. I think that’s great. I think that helps everybody but you know in terms of the the why do we vote the way we do stuff, you know is interesting. I was talking to David Hurley who is a you know, he’s very famous liberal pollster advisor smart guy all around and he was sort of saying that in his work kind of wants to take a step back from the horse race. Not that he doesn’t like it because. You know, if you’re if you’re in an election race and you are in a riding where strategic voting is a thing you need to know but his he says he wants to know why pollsters are there’s a disconnect between what they’re seeing and the populace ferment that’s happening underneath. Why are people so angry when the you know, the Top Line GDP numbers are pretty good, you know like that all the data we have says things aren’t that bad and all the election results keep saying people are angry. Angrier than should be kind of the case based on the data. And so he really kind of wants to go back to the ground. He was telling me and do some more deep listening kind of more holistic qualitative work rather than kind of continuing to just only focus on the horse race stuff. So I think that’s an interesting opportunity for the polling industry and the conventional public opinion research industry is Go deeper on the why yeah, because that’s a place where I don’t think AI can necessarily be.
Jordan: Where does Advanced symbolic see Polly in five years from now?
Max: I assume they see her being more accurate being involved in more markets covering more subjects growing her sample. I know they’re they’re constantly feeding her new information. So it’s I mean this is going to be a reference that won’t work for some of your listeners. But did you ever watch that movie short circuit
Jordan: not even going to work for me
Max: now. Okay. It was like a Steve Guttenberg movie in the in the late 80s and it was about this this military robot that. Left the robot Factory and then became independent, right and it was called his like number five was the name of the robot and they would feed the robot information. He would just go through the book like the turn the pages so fast, and he would know all the information. And that’s what she does they will feed her informational studies new special show her newspapers. They teach her how to recognize humor in tweets. So are people tweeting about self-harm sarcastically or is it authentic she’s always growing and becoming smarter. And so I kind of don’t really want to put a prediction out on what she’ll be doing in five years because. The thing with a i and it part of the part that’s terrifying for a lot of us is its kind of Limitless in its potential, you know, it can do things that we didn’t really think we’re possible. It puts a lot of people out of work that didn’t think they could be put out of work. And and so yeah, I don’t know the sky’s the limit I guess.
Jordan: That is an appropriately ominous way to end the conversation Thanks Max.
Max: Thanks, Jordan,
Jordan: Max Fawcett is a writer at many places including the walrus. That was the big story. And if you want more big stories, all of them are at thebigstorypodcast.ca and all of our tweets are on Twitter at @thebigstoryfpn. If you want to make everybody around here feel real good head over to Apple or your favourite podcast app hit subscribe hit rate. Leave a five star review. If you want to find us we’re on that podcast app any other you got Apple, Google, Stitcher, Spotify. Thanks for listening. I’m Jordan Heath Rawlings. We’ll talk tomorrow.
Back to top of page
