David Lazer: Facebook has this large pile of things it could show you. It chooses some and not others. If we’re trying to characterize the global emergent tendencies of those social algorithms in promoting some content versus others, what are those kinds of tendencies? On this new observatory, the objective of this is to create a large panel of subjects where we’ll both monitor their online behaviors as well as the behaviors of the platforms with which they are engaged.

Noshir Contractor: Welcome to this episode of Untangling the Web, a podcast of the Web Science Trust. I am Noshir Contractor, and I will be your host today. On this podcast we bring in thought leaders to explore how the web is shaping society, and how society in turn is shaping the web. Today, my guest is David Lazer, who you just heard talk about an effort he’s leading, funded by the National Science Foundation, to build an observatory to study online human behavior as well as the algorithmic strategies of social media platforms. 

David is a University Distinguished Professor of Political Science and Computer and Information Science at Northeastern University. He’s among the leading scholars in the world on misinformation, with some of the most highly cited papers. His research, published in journals such as Science and Proceedings of the National Academy of Sciences, has received extensive coverage in the media. In 2019, he was elected a Fellow of the National Academy of Public Administration. Welcome, David.

David Lazer:  Thank you for having me, Noshir. It’s delightful to be here. 

Noshir Contractor: I want to start by trying to trace the ways in which you started out as a political scientist and then got interested in issues related to the web. 

David Lazer: Well, I was really interested in the role that the web and related technologies played in our political system and in how the government works. In the early years, I was interested in the notion that there were transformational effects of network technologies on both the organization of government – that is how it ran at odds with the hierarchical structures of government – but also could change the relationship between government and citizens. With colleagues, we did a number of online experiments, for example, of having citizen town halls where citizens met with their members of Congress online. This was back in 2006. These highlighted the potential for raising up the discourse about politics between citizens and their representatives. That was where we started or where I started, was thinking about how government could be rewired because of the web and related technologies.

Noshir Contractor: People in general, at the time, were still excited about the web, and all the potential good it could do.

David Lazer: I still actually have some fair degree of optimism. There was a 2018 session that Harvard was holding for all the newly elected members of Congress in 2018. And we gave literally every sitting member of Congress a copy of our book on the potential of the internet to change their relationships with their constituents in positive ways. 

Noshir Contractor: I imagine you’re talking about the book Politics with the People: Building Directly Representative Democracy that you published along with your co authors in 2018. 

David Lazer: It was explicitly a book about the potential transformation of our democracy. We did these experiments with constituents meeting with members of Congress, and they were full-blown, randomized, control treatment experiments, and so we could really make robust scientific inferences about the impact that these kinds of discussions had on individuals.

Noshir Contractor: For those who are uninitiated, what does an online experiment in this context look like? 

David Lazer: In these experiments with members of Congress, we used a survey firm to recruit a sample of people who said that they were willing to participate in an online discussion. And what the actual session was was a member of Congress having a discussion with them around a hot button issue, immigration. And we then randomly assigned people to participate or not participate. Comparing the people who participated in the sessions versus the people who didn’t, people who participated in the sessions did have a sort of shift in the direction of their members. And they were also more likely subsequently to vote. So we were able to look at voter data, and to see who voted and who didn’t vote. These are sort of administrative data that are generally available. The critical flavor here is we can actually take people from around the world and put them in the same virtual room, and then make that room structured in a way that enables certain kinds of communications, disables other kinds of communications. So in a sense, the web is the ultimate laboratory for studying human interaction, because it’s so malleable, and because so much of the world is readily accessible wherever they are. 

Noshir Contractor: And that web is a treasure trove both in terms of having digital traces, but as you point out now, also in providing a platform to collect data from individuals. 

David Lazer: One of my favorite quotes from one of the citizens coming out of it was, “Huh, policy is a lot more complicated than I thought it was.” Which is certainly the truth. I think that these sessions made me feel quite proud of our democracy and what our democracy could be. And so part of our commitment was to figure out how to translate those findings into actual democratic practice. And we actually wrote up a report, a guide on how politicians should and could use the web to support deliberative democracy. We’ve talked to politicians about this, to all the big tech companies. We did a speaking tour in Silicon Valley to talk about the role that these platforms could play in enabling democracy and not just disabling democracy.

Noshir Contractor: In addition to your own work in online platforms, you’ve actually helped develop platforms for other scientists to be able to conduct research. Talk a little bit about Volunteer Science and what hopes you have for that moving forward. 

David Lazer: So Volunteer Science is a platform we started building around 10 years ago. There are a lot of startup costs to doing an experiment. So you have to build all that infrastructure and so on, then you run your experiment, and then you stop running your experiment, and then it just sort of decays, it’s not replicable, or it’s expensive to replicate. And so the objective with volunteer science was to make it easier for experimenters to get experiments up and running quickly. The platform carries a lot of the weight. It allows you to rapidly instantiate versions of your experiment. So it basically lowers the startup costs of running an experiment quite a lot, and as well as the management costs of running an experiment as well as the management costs of running an experiment. And of course, you can also then recruit samples from around the world. We’ve done data collection where we’re trying to do the same kind of data collection for people who are participating from India and South America and North America. And so we’re able to recruit much, much more interesting and diverse samples, as well. These screens are almost literally like windows into people’s lives. And we can literally reach through those windows and say, “Please, participate in, in this experiment with us, help out science, volunteer for science.” And we get on the order of 100,000 people a year who come and volunteer to participate as subjects. 

Noshir Contractor: And you have spent a lot of time and been amongst the leading scholars looking at the use of big data to understand human behavior. Tell us about the article in 2014 in Science where you critique the Google Flu Trends. So tell us what the Google Flu Trends was and then how you critiqued it. 

David Lazer: So Google Flu Trends was a landmark paper published in Nature that examined the relationship between searches for the flu and prevalence of the flu. And in particular, it was aggregating flu related searches, or things that I should note are correlated with the prevalence of the flu. Being used at the time, was nowcasting, the idea that standard methods for evaluating prevalence of the flu involve a multiple week lag of collecting all the data, aggregating it, and then looking at how many cases there were two weeks ago. When a contagious disease is spreading, you’d like to know when and where it’s spiking, because that can direct both preventive measures, as well as measures that will mitigate the harm, like, you know, surge of capacity in hospitals and the like. And so the Nature paper was, I think, it asserted the fact that you could do this all much faster. It offered a method to do that by saying we have tons of searches on Google, and we can see how they correlate with the prevalence of the flu in the U.S. 

Noshir Contractor: So the assumption here is that when a flu is breaking out, people will go on the web and search for terms that are relevant to the flu like “fever.” And that list of search queries is then raw data that they put into a model to make a prediction about when and where the flu is spiking.

David Lazer: Exactly. Although, they looked at all search queries, including ones that were clearly not related to the flu, and then fit them to flu prevalence. Then what happened was it repeatedly stopped working well. You know, high school basketball turned out to be predictive of flu prevalence. Well, it’s because high school basketball has its peak of its season apparently at what is typically roughly the peak of the flu season. They said they looked at some and weeded them out by hand, which is not best methods. There was an offseason flu, I think was it H1N1 if I’m remembering correctly, that it then did very badly at predicting, and partly it’s because they had built something that was partially flu predictor and partially winter predictor. And so what we were doing was, in our critique that appeared in 2014, was not to sort of discredit the whole notion of what was being done in that Google Flu Trends paper, but to say that there are ways that if you’re not careful, that big data will lead you in the wrong places. And then in a follow up paper we proposed a new method, which involved what we called human computation, which involved humans looking at some of the search terms. Because we’re, humans are much better at interpreting what the intent was when someone searched, like, why did they search for this? Oh, I bet it’s because they were sick, right. And so we devised this sort of thing for human coders to come up with an evaluation like that. We were then able to fit it to a sample of people who had agreed to have their search terms evaluated. And we asked them whether they had the flu, and we were able to predict whether individuals had the flu, and then aggregate that upwards, nationally, and that showed a very different kind of methodology that in part leveraged the human interpretive abilities, in addition to the big data component. 

Noshir Contractor: So you got some ground truth data by actually collecting the search queries from the same individuals who then reported, whether they had the flu or not.

David Lazer: That’s right. Also the other thing we saw was that there are – and this is not shocking, but – there are real differences in how people search. And so we found, for example, very sizable gender differences in search tendencies. Women in our sample actually had higher levels of searching about the flu before flu season. And then when there was flu in the household, their search level didn’t go up that much. Men, on the other hand, had a baseline level of searching for flu information that was pretty darn close to zero, and then they’d freak out online when they had symptoms. And so what we’re really finding out when we look at these kinds of search queries, is how often men have the flu. Now, that actually may be fairly predictive of how often the population has the flu. There’s some perverted element to that in terms of where we’re getting signal. That’s obviously a more general concern in the social sciences is that who’s on the web, how much, from where is, is not representative of the general population. And even if it were, their behaviors may not be because there may be these very differentiated ways that those behaviors manifest.

Noshir Contractor: And of course, today, during the pandemic, we have really hastened and become much more focused on being able to get these data in as close to real time as possible so that the policies can be responsive to those. I want to switch our attention from pandemics to infodemics. Talk to us a little bit about what got you excited or intrigued or interested, or concerned about fake news? 

David Lazer: I was particularly concerned, as many people were after the election of 2016 in the United States, because it felt like there was a real breakdown in our information ecosystem. And so with a number of collaborators, we put on a major conference in February 2017 on the science of fake news. There were psychologists looking at this, there were political scientists looking at this, there were computer scientists looking at this. But rarely do they connect outside of their disciplinary silos. And we pulled together all the speakers from that conference, and out of that came a 2018 paper that appeared in Science, titled “The science of fake news,” and it was putting together a multidisciplinary perspective on misinformation and fake news. Fake news is a very narrow and specific thing, but misinformation is the more general thing. 

Noshir Contractor: Can you point to some insights that might not have been gleaned in “The science of fake news” were it not for having an interdisciplinary take on it.

David Lazer: I’ll point to another paper of mine from 2019 that also appeared in Science that examined the prevalence of fake news on Twitter. That paper was a collaboration among three computer scientists, a cognitive psychologist, and myself, a political scientist by training. From the social science perspective, it involved thinking about how do we build a high quality sample? So we said, “Well, we really care about humans.” And so how do we develop a large sample of humans? And the way we did that was we built these computational methods to disambiguate and match the Twitter data to voter data. And then we were able to then computationally extract from Twitter the prevalence of fake news as well as make certain inferences about what people were exposed to. There’s no way that just a team of social scientists could have done this. But also, there’s no way that team of computer scientists would have done it either. It required, I think, really the best elements of computer science and, and the social sciences.

Noshir Contractor: That’s an excellent example of why web science calls itself an interdiscipline, because it transcends specific disciplines that contribute to it. I want to touch on some of the recent efforts that you’ve just launched with a major grant from the National Science Foundation that picks up some of your earlier work back from 2013 on issues such as algorithmic auditing and online personalization. So, to begin with, what do you mean by algorithmic auditing?

David Lazer: So much of what we see on the web is mediated by platforms. You know, Facebook has this large pile of things it could show you. It chooses some and not others. And so what we really have are these computational curators. So Facebook is looking at your Facebook friends and saying, you know, that looks awfully boring, we’re not going to show it to you. You know, typically, we’ve described that curation process as being algorithmic. And really, it’s something that I’ve called and others have called the social algorithm. It’s this interplay of humans and computers that results in emergent behaviors in terms of what you see and what you don’t see. These algorithms are predictive models. We’re trying to optimize on some metrics. And that metric might be something like, what gets you to click on something? Because basically, what platforms are able to do is translate your time on platform into profits in various ways. And so the question when we talk about algorithmic bias – and this is a phrase that manifests not just about the web, but like everything from criminal justice to housing to credit decisions, and so on – if we’re trying to characterize the global emergent tendencies of those social algorithms in promoting some content versus others, what are those kinds of tendencies? And what are the kinds of patterns? So for example, to what extent do we see personalization? Like, if you’re searching for something on Google, do you see the same thing as I do? And the answer is, actually, generally you do. One of the things that Google does do is that they will geo locate some of your searches. If I search for pizza, it will show pizzas around Boston. It doesn’t do it with politics. So like, if you’re looking at my for my member of Congress, Google doesn’t geolocate that. So it’s making decisions as to how to treat different kinds of information, different kinds of queries. And there’s also a question of, like, what kinds of sources Google promotes. Do they promote misinformation sources? But then we could think of, does social media, we’re looking at Facebook, tend to promote more emotional content? Or does it tend to demote civic content. So we can imagine, like, auditing a platform like Facebook in terms of understanding what content it systematically promotes or demotes. So in any case, on this new observatory, the objective of this is to create a large panel of subjects, a very large panel of people of 10s of 1000s, where we’ll both monitor, all with consent, their, online behaviors, as well as the behaviors of the platforms with which they are engaged. And in that way, we’ll be able to get more of a handle on these algorithmic structures that are so, so very important in modern day society.

Noshir Contractor: So tell us what you expect you will be able to do once you have this web observatory up and running. What are the kinds of questions that you will be able to answer more definitively than we are able to do today?

David Lazer: I should note that this is really a communal resource. It’s not just a resource for me and my collaborators. The objective will be to set up an infrastructure that provides access to these data – analytic access – while guarding the security and privacy of people who are participating in it. And so the objective then is to understand everything from what kinds of content do different platforms promote? You know, do we see higher quality or lower quality information being promoted? Do we see biases in terms of what’s promoted? Do we see certain demographics being targeted? So really trying to understand how people get information, the role that the platforms play, and how the platforms play together. Our objective is to have a picture of the emergent and evolving web and people’s behaviors on the web.

Noshir Contractor: Wow, what a tour de force we have gone through today, going back to you being amongst the early people to see how web could change politics and journeyed all the way to now setting up this communal good as an observatory for studying online human and platform behavior, and as you pointed out, not just a single platform, but the ecosystem of platforms as they all interact with one another. David, such a pleasure to have you as my guest on this 30th episode of Untangling the Web. I look forward to continuing to see the research that comes out of this observatory and all of your other very exciting initiatives. And so thanks again very much for joining us today.

David Lazer: Noshir, it’s been a pleasure and an honor speaking with you.

Noshir Contractor: Untangling the Web is a production of the Web Science Trust. This episode was edited by Susanna Kemp. I am Noshir Contractor. You can find out more about our conversation today, provided they are not demoted by the algorithms, in the show notes. Thanks for listening.