Esther Duflo, PhD - The Impact of Physician (and Economists!) Communication During the COVID Crisis: Evidence from Randomized Trials in Disadvantaged Communities

This morning's lecture is part of our department's quarterly equity diversity and inclusion grand rounds lecture series, which is a collaboration with Dr. Egg Buddha, who is our departmental director of equity diversity and inclusion. This lecture series is aimed at incorporating more didactics into our departmental curriculum that address health disparities, racism, and social injustice. And it is my great pleasure and honor to introduce our speaker this morning, Dr. Esther DuFlo, who is the Abdul-Atif Jamil Professor of Poverty, Illegiation, and Development in the Department of Economics at MIT. Dr. DuFlo is originally from France. She completed her undergraduate and master studies in Paris and then received her PhD in economics from MIT. She subsequently joined the faculty at MIT in 1999, and shortly thereafter in 2004 became among the youngest tenured professors at MIT. In 2004, she co-founded and is currently co-director of the Abdul-Atif Jamil Poverty Action Lab at MIT. In her research, she seeks to understand the economic lives of the poor with the aim to help design and evaluate social policies. She has worked on health, education, financial inclusion, environment, and governance. She has received a number of awards and honors for her work, including a MacArthur Genius Grant in 2009. She was awarded the Nobel Prize for Economics in 2019, and she is the youngest person, and only the second woman to be awarded the Nobel Prize for Economics. We are extremely fortunate to have her with us today for this lecture, and incredibly grateful for her willingness to speak to us on the impact of physician and economist communication during the COVID crisis. So thank you again, Dr. DuFlo, and I will turn things over to you. Thank you very much, Dr. Huy. Thank you for the invitation. It's my pleasure to be here, albeit, in this virtual format, and speaking from France, actually. So, in some sense, it would not have been possible without this pandemic. If you don't mind, I'm going to share my screen to discuss with you results from a few studies that we've engaged with mostly during the COVID crisis and some just before. So this is a group of paper that involves many people, as you can see, from the long list of authors on the page. You cannot even fully see it because a part of the authors are hidden under this morning of COVID-19 working group, which is a group of doctors mostly from MTH, and I'll describe their contribution in a moment. So before a vaccine was available, and even now with the vaccine, communication is a critical way to protect people against the pandemic. But before the only thing we could do was behavior change. That was the only thing available. Even with the vaccine communication remains critical, first of all, because this behavior has to continue, second of all, because we have to persuade people to actually get immunized. And this, of course, has not escaped the keen attention of anybody, and as soon as March 2020, people were bombarded with information and disinformation on COVID-19, tour radio, tour TV, tour social media, and it has continued to these days. Very army on what I would describe as credible people, like Dr. Fauci, but also, you know, the poll stars, and then regular doctors, or usual doctors, tried to convey the most relevant information and advice to be purly in a 16-way, to make sure that they could act on what was known at least at the time. And it made a lot of sense for physicians to take a part of that role in particular on social networks because physicians and nurses are aware before the pandemic and remain during the pandemic, some of the most trusted experts on their field of expertise. I should say, by the way, that economists are some of the least trusted experts on their field of expertise. In a poll in 2008, that we did, we found that on the 25% of people respect economists when they talk about economics, but 87% of people respect nurses when they speak about nursing. So this kind of proliferation of messages by a credible people raised a few questions. First is whether it can actually work, whether it makes any difference, or whether it gets drowned either in the noise, because there were so many people who were saying so many things, or in the politics. And there's certainly some feeling that has been expressed in the run-up to the presidential election in the US that everything became politicized. And maybe in this context, people speaking from a position of expertise doesn't become is not really credible anymore. So this series of projects I'm going to show you today are an attempt to answer those questions, also an attempt to answer the questions of whether these messages needs to be particularly well designed, how they can be designed to be more effective, and also along the way you'll see that we are trying to work at scale to along the way make a difference, and not just doing research, but also trying to impact things as we go. So the first study is a study we did in West Bengal. We started two study together almost at the same time with this group of people, one in India, one in the US. The India studies started in, was launched in May, and in fact the intervention was done over just two days in the May 4th and 5th of 2020. They had been strict nationwide lockdown, very early on, and early May so loosening of some of the restrictions. The population was generally absolutely bombarded with messages about COVID-19. The people we interview claim that they had received 20 messages on average on social distancing in the last two days from various sources, including on their telephone, on their radio, on TV. The question is whether it made sense to have yet another messages or whether the impact of that message would be ineffective. People had reasonably good high level, the knowledge level from the beginning, in other words they kind of knew what they ought to do, but they weren't doing it necessarily. According to self-report, on the two-serve, West-Tent was soap on coming home from the trick outside. So, the seven had traveled to the outside village in the last two days, even though it was under lockdown. And on average, they had at least 11 interaction within two arms length, which is the thing that was asked to be avoidable in the Indian balance over the last two days. So, our experiment asks a simple question is whether an extra notch from a reliable source improve compliance. So, the interesting aspect of it in this case was that we sort of had in the research team our own little reliable source, because Abidit Banagjig, who is the author, happened to be an extremely well-known figure in West Bengal, first because you won a Nobel Prize in economics alongside with me and Michael Kramer in 2019. And he is originally from West Bengal. So, he has become, you know, his face is on billboards and he has become a bit of a celebrity. People are much more serious about their Nobel prizes in West Bengal than they are in the US. He also happens to be the chair of the COVID-19 scientific council, so the non-doctor equivalent of Dr. Fauci, although he is less often on television than Dr. Fauci, and therefore, he had some credibility to be the one relaying some methods. So, what we did is that we had in record short videos on basic symptoms, encouraging people to report symptoms to the local health worker, because at the beginning of the epidemics in the absence of testing, that was the only mood of surveillance that was available. And then we weren't sure what was important in the communication, so we varied depending on the, in different villages, we sent different type of information. So, in some cases, he emphasized some behavior, other than some others. In some cases, he emphasized whether you should do that to protect yourselves, because you could have, you could hire, you could have a long run symptoms, long run scarring, or whether in some messages, he emphasized instead the risk to the most fragile people in the communities. So, very much the same type of questions that we have here. Is it more effective to appeal on people's desire to protect their gun mothers or to appeal on people's sense that they might actually be hot? And then, he also discussed ostracism, which is a big issue in India, where people with symptoms get kind of shunned away. The advantage of working, of being in this situation, of being working as a chair of the scientific council is that this was immediately part of policy in a way. So, we were able to send the campaign to 25 million subscribers of the GeoFund network, which is one big Fund network in India. And this was randomized in 1214 pin code. I was also, almost all of West Bengal was in the experiments, and a small little part was left as a control group, and the rest was all blanketed by the messages. So, the endomization allows measurement of seeing whether there are effects of any message compared to the control group, as well as peel over, because we only targeted the subscriber of one network, we can note that subscriber of other networks, and also as looking whether the type of message actually matters. But what data can we have, so we were not even though we could do this on a very large scale, there was at the time absolutely no data available, even the data on cell phone mobility that since then become available was not there. So, we had to do a small survey. The first thing we did is a survey of the frontline health workers, because if you remember, all of the intervention asked people to report symptoms. So, we surveyed very quickly within five days of treatment to make sure that we were not responsible for an increase. We make sure that we wouldn't... How's it going? You all right? We would not confuse reporting with actual making people sicker, so we surveyed very quickly. And the key outcome there is whether anybody reports symptoms to them. Then we found large database of anybody who had been a council member, so a public elected official, is there in the past or in the present, and we surveyed about 2,000 people from that group. What did we find? So, one of the very large impacts we found very early on is that ASHA reported, much more reporting of symptoms to them, in treated villages. So, people really heard the information that they needed to report, and they reported twice as much. And that's in a sense our most objective measure of behavior, on which we see a very large effect, which you can see here on the bottom left, in the top left corner. The other behavior also changed. There was an increase in handwashing, a decrease in going outside the village, and a small increase in mask querying, even though it starts from a at least self-reported very, very, very high level. In May, I should say, if you go to India this day, you will not see very many people with a mask anymore. Interestingly, the content of the message didn't matter at all. But seem to matter was that people heard anything. They're just any type of, so these are all the various type of treatments. They all work just the same. So, people seem to generally have picked up on the information that, oh, we better do something, and then started paying more attention to all the information. There was also a very large spillover of the non-geo subscribers, which did not directly receive the video. So, with success that either they got them, they were shared with them, which is the possibility, or the people told them that, oh, you know what, a bit of energy told us that we should respect these behaviors. So, there spread, you know, even all of it, to this world of mouth. So, this was encouraging, in the sense that this was really already at scale, 25 million recipients, indicative of significant benefit from continuing messaging campaign by credible source, even once you have the feeling that society is informed. Of course, we don't know, maybe it would have worked if it was anybody, but all we know is that at least if it's a credible source, it is still effective to continue to convey messages. So, because this was effective, this is also very, very cheap to do, in the course of the pandemic, I bet it by the way, sent a few more of those messages, not as part of a study anymore, just as part of a policy. And one of the most important set of messages is sent, which is going to, we're going to see some echo in another study we completed in the US, is in on suspicion of Dorgapudra, which is a massive holiday in West Bengal, where people travel from everywhere to everywhere, and then assemble in massive possessions to visit informal, makeshift temples that people put together and then to have religious possessions. There was a decision to keep these festivals this year, so the messages were about, don't go if you don't have to go, and if you have to participate, participate to just one, stay outside, to pure distance, etc. So, more mitigating behavior idea. We don't have a direct evaluation of this intervention, but what we do know is that there was no uptake of COVID-19 cases in West Bengal, post the Dorgapudra holiday, which was in October, compared to other states. Whereas if you look at holidays in other states at different times, shortly after, in the week or 10 days after the holidays, you saw a big uptake in cases. So, it is plausible that actually just sending these messages on kind of responsible behavior during Dorgapudra was effective. So, that's the first set of studies, this is what we did in India. At the same time, we of course, we were all in Boston, in lockdown in Boston, and very keen to do something in the community. And in particular, it appeared, a very soon, that the pandemic was a syndemic in the sense that it was combined with systemic racism to lead to a very unequal burden for the black and Latinx communities, particularly in Boston, the disproportionate impact of the Latinx communities was very visible in Boston hospitals, you know, that were even more better than it. So, we have the same set of people who had worked off the West Bengal study, got in touch with the doctors at the MDH, who actually, my pediatrician, who kind of asked, it was a new set of people who would be interested in working together to develop some messages that would be particularly relevant to the black and Latinx community, and that we could convey, first of all, that we could test the effectiveness of, and then if it's effective, spread in a large way. So, from the beginning, again, there was the same idea of action research projects if you want of doing a research that is mainly designed to give us the best possible answer to what should be done, and then trying to do it on a large scale. So, what I'm going to do now is to walk you to that arc. So, the first project turned into a study, Dr. Fatima Kodistanford at the MDH and Harvard and Dr. Marcela Alsen, who is actually both an economist and an infectious medicine doctor who are the lead author, and I was kind of the producer if you want to do the last author. So as you all know, 2020 was a very unusual year for both from the recognition of the systemic racism and its role in health care or in June 2020, the American Medical Association actually issued a statement to recognize it. As of today, there are 1.4 times greater cases in COVID-19 among the black U.S. population, and almost three times greater death rate, but it adjusted that corresponds to four times a greater mortality. Many physicians have tried to reach out to people directly through social networks, not just black and Latinx people, but everyone, to encourage social distancing, to encourage not traveling around the holidays, etc. And it makes sense because, as I was saying before, doctors are among the experts most trusted in public opinion, along with nurses, so I think we should use this trust as much as possible in order to convey those messages. So again, the question was there, is it effective? And in particular, does it effectively reach out the black American and Latinx who bear the biggest burden? And can of should anything be done to make such communication more effective for black Americans? So many people are. I've pointed out that a lot of the Dr. Fauci, Dr. Berks, and all white faces, there was really only Jerry Adams, that was a black face and not necessarily had a, it didn't necessarily was the most apparent in front of the people. So does it matter and should we, that should diverse work for the position, make a difference by reaching out to people via their direct communication channels? So we collected two studies, I'm going to describe only one today, two shorter studies, I'm going to describe only one today because the other one is under review, but I'll just give you the, it's up short. Both studies were conducted in the US online using the Lucid platform, which is an online surveying platform. They had fairly large samples of low-income Americans, so people making mostly less than $80,000 a year with less than college education. So for this first study, we recruited 11,000 black and American and 19X, and the intervention took place between the 13 and the 24th of May 2020, so very much the same time as the Westman Gold Study. So we had a group of doctors who, who, white, black, different age, different gender, from MGH and from a community health center, Lin Community Health Center, who had three information statements against about COVID-19, about what to do. So I'll try to show you an example of how it looks like. So this is our project webpage, if you want to go there, you have everything about it, including the messages. Can you see Dr. Garcia Quitte on the screen? I hope so. Tell me if you can not hear her. We are not able to hear her. You are not here but we are. So I need to do something. I'll do it right away. Can you know her? Yes. Yep. So I can stop here but it gives you an idea. So this is very pedagogical. It was not like advertising. And we had three messages. These first, so people saw three videos about two to three minutes each. This first one was about the description of what COVID-19 is in general. And there was a second set of video about which was directly taken from the CDC about what was known about social distancing at the time. Actually masks were not part of the official recommendation yet, if you remember. And the third video was actually about mask wearing. And I'll give you a little more detail about the nature of these videos in a moment. The basic video said, even if it's uncomfortable, please wear a mask. This is the best way to protect yourself and others. And then there was a variation on it which I'll describe in a second. So this was basically the gist of the intervention, a long-ish message on COVID-19. And we vary different aspects of the message and the messageers which I'll describe in a minute. And then we surveyed online immediately to measure the outcomes. So there are questions that are due to the video have any impact on knowledge, on belief, on intended behavior, whether the risk on the doctor and the respondents have an impact. So there has been previous study that African-American are more likely to act when the doctor looks like them. So something with Hispanic, we were asking this question, does it matter? Another thing that we wanted to know is whether acknowledging prior injustices or in economic insecurity have an impact in terms of it would make people more receptive to what people have to say. Finally, people who are complaining a lot about Dr. Berks at the time for being too white and too proper and too associated with a trampoint house to be an effective communicator. So that's why we took her message and it was read either by her or by the regular doctors. And finally, this was really the beginning of wearing masks and the social norm of wearing masks was kind of quickly establishing itself. And in particular, I don't know if you remember at the time in May, there was a lot of discussion that for African-American wearing a mask might be dangerous because they might not be well regarded by the community if they were wearing a mask. So everybody had an anecdote about a black person coming up to a shop wearing a mask and being asked to remove it because people were afraid. So here's how the study worked. So people were recruited via this platform. We collected a few demographic information on them. Then they were randomized into either no intervention or to different scripts targeted to the black and to the Latinx separately. And they were randomized. If they were randomized to receive the messages, they were randomized to either receive them from a concordant doctor. So black, if you black or Latinx, for Latinx. To either receive a doctor box for the second video or to receive one of the doctors saying exactly the same thing. To either receive this acknowledgement of inequality, so this is something that was a random decision done in the first video where not only she says, before saying even though there is no cure, she says, I understand that it is not always easy for an African-American person in a hospital or something like that, that we haven't always treated you well. So we are now able to acknowledge this. And finally, the social norm intervention. So this is for the concordants. You can see we had the Latinx doctor. In fact, the Latinx could decide to have the script in Spanish or in English if they wanted. But regardless, it was in Spanish. A black doctor, a white doctor, they could... Of course, here I'm showing all men. But we had also women, so people were randomized by... By... By... According to gender also. So this is a social norm intervention. So just before we conducted this large study, we conducted a first little study online to among white people. And we asked them... We showed them images and we asked them what they thought of the person represented by the yellow arrow. So do you believe this person is... And then we ask, are they sick? Are they up to no good? Are they protecting their communities? And eight out of ten people think that a black person wearing a mask is protecting their community. So already by May, the idea that mask were the way you protect your community was kind of already in the spirit. Although of course you could say, well, two out of eight don't think that. But having that information in mind, the social norm intervention was to tell people, you know, in a study we conducted, we found that eight people out of ten thinks that a black person wearing a mask is protecting their community. So the doctor said that in the script. So the idea is by shifting the social norm, you make it more acceptable to be wearing a mask. And similarly for the Latinx, we had the same... We had the same... We had Latinx people wearing a mask and we also asked whether they were sick up to no good or protecting their community. So on average, if you compare the entire intervention group and regardless of the intervention that they got to the control group, we found a significance and non-negligible impact on knowledge. So here I'm reporting the knowledge gap score, which is how many mistakes you're making on a list of symptoms plus the list of ways that you have to protect yourself from the infection. And you can see that actually knowledge was very, very good in me. And the 72% have no gaps. 73% almost of the respondents have no gaps. But it increased to 80% when they receive any intervention. So this is actually a fairly large increase in the number of people who have perfect knowledge which is distributed across all of the other scores. So the intervention was effective on knowledge. So one weakness of this study is we didn't have a great measure of behavior because we could only measure things immediately after and at the time it was impossible to offer people to sell their masks or things like that and we couldn't really follow people either. So what we did is we asked people to click for links if they wanted more information. So this click through rate is an indication of what they hope to do in the future. On average for the population, we don't really find a large effect or a much effect at all on this demand for informational links. So in general, we don't find much of an effect in this study on links. We just affected knowledge except for African American where they were where they were when all the messages were read by a black person. So when they were black when they were concordants. So when they were concordants, you can see that people for example are quite a bit more likely to want for links or more and they are less likely to want zero link or one link or two link or three. They want for link, they are more likely to want for links or more. That looks a little small on the bars but when you, but it's quite significant when you're looking at the, the, the, the I.I. of this, of, of, of this measure. So there is a quite, there is a significant increase in demand for links only if the doctor was black. So this was encouraging that there was an effect on knowledge. We, not quite enough, maybe, we, to, to go to turn that thing, well, you know, we know it, it works. So we should scale it up. Interestingly, besides the social, no, besides the race of the doctor who was speaking, we didn't have much of a difference between the, the other version of the messages. In particular, when we ask for video rating, people rated the Dr. Burke's video very low, but it's not that people knew less after she had spoken. So even though people didn't like her or didn't like her video, they still listened to what she had to say. The, the social norm intervention that I showed you had a very large effect on perception of social norms. So people tended to overestimate particular black respondents, overestimate the fraction of white respondents who would say that a black wearing a mask is up to no good. And they were swayed by what we told them. So this was useful for, it changed their view on the social norm, but it didn't change their view on, on, on what to, on whether to wear a mask, for example. So it didn't increase the click true on a demand for a mask. Which is, you know, everybody was somewhat affected for the man for a mask and it didn't make much difference. And the acknowledgement of inequality to the recognition by the doctor that the health system has not, not always treated African American in an, in an equal way or for the Latin X, the intervention was the fear of migration, immigration issues if you go to a hospital that made no no difference. So the doctors video appeared to be affecting a changing knowledge, but there was some limitation in study. The first one is that there was no had much of behavior, even self-reported. We didn't have the whites, just the black and the Latin X. The world also had changed quickly after May. First of all, May, you know, shortly after we finished this intervention, which is actually why we couldn't do follow up is because on, on, on memorial dead, as Floyd was killed, and that was followed by racial justice protests, which brought much more attention to the racial issues we have in the US. And after that, there was polarization and getting closer to the election. The one was also getting fatigued. There was a report that people were less likely to adhere to the, to the, to the individual a measure of protection against the epidemic, so all of these reasons. Before scaling up, we, we decided to contact a second study. So the second study is the one that I'm not going to describe in, in detail, but it took place in September. And just to summarize the results, we replicated that the video improved COVID-19 knowledge, even though the knowledge level were much lower, self-reported knowledge level were lower, the video improved them even more. And perhaps more importantly, they also improved behavior. Both the demand for links, the willingness to pay for, we asked people to, if they were interested in buying a mask and we reported their willingness to pay, so that was higher with the intervention. And we were able to follow a group, a, a sample, a few day after intervention and their self-reported mask wearing, etc., also improved. And finally, these effects were present in all groups, even in the Republican, and this time, none of the racial-specific framing matter. So the white doctor was just as effective to talk to black American and the black doctors. So in a sense, it was very good news because it suggested that, okay, it can work to send those messages. And in fact, it's pretty simple. You don't need to be, you don't need to be worried that some group is going to not react well to what you have to say. You can just say it and a lot of people will just be willing to listen. It was in particular quite striking that there is no, not much effect of the tailoring of the message and it works on everyone. Low educated, not well educated, high, very, very low income and low income. Black white. So having done that, we decided that it was time to do, after all, what was our objective all along, which was to do this in scale, a scale. And after we had publicized the West Angle study, we got contacted by Facebook, we who have an armed data for good, with the following offer, which is that they would give us many, many, many, many ad credits to send messages to people. So Facebook, of course, has reached that is unrivaled, almost 3 billion as active subscriber in the US, 230 million US accounts. And in November 2020, you'll probably remember how there was a lot of fear for Thanksgiving travel as engine for spreading COVID-19. And Facebook had contacted us not long before Thanksgiving. So we put something very quickly with our set of doctors to report, to record a very simple message. I'll show you, they end to send that message as an ad, as a push to Facebook user. So it was delivered as sponsored content advertisements to the users. So let me show you the page. If it works, because I'm not on Facebook myself, so sometimes it says that I have to go on Facebook to shoot. So this was the Thanksgiving, this was the Thanksgiving campaign. So as you can see, oops, now I have to stop her. Sorry, I will have to. It goes on repeat, if you. I wanted to show you the Christmas video, because after Thanksgiving we did the Christmas one. So I said that this was my foray into Facebook advertisement. Because Facebook gave us the money, but nothing else. So we kind of designed the script and we hired someone to put the video together, but it had this very natural, not very processed field to it. So in Thanksgiving, the intervention was launched on November 14. We had just this one message, but delivered by various doctors. The hope was to the maximum extent possible to saturate the zip codes that where we would where we would randomize and to make sure that each person sees it more than once, because apparently that's important to react to an ad. So in total, 11 million study messages at least once and in 30 million video posts got sent. At Christmas, the intervention was launched on the 17th of December. 23 people saw a study message at least once and in total 80 million video posts were sent. Our sample frame is 820 countries, counties in 13 US states which have high COVID-19. And we randomized counties between high and low intensity and within counties zip codes into treatment and control. The reason why we did that is that the COVID-19 data once we get it is often available at the zip code level. And it gives us more power to look at the zip code and is affected zip code level. But the mobility data use county is much better at the county level because it's aggregated from a lot of people and to maintain anonymity. It's really available at the county level. So that's why we decided that level of randomization. So right now I'm going to show you mobility data. We haven't fully combined compiled the COVID-19 result yet. So there are two measures. One is whether people are traveling less than before. So are people moving less and in particular traveling less long distance. And the second measure is whether people are exactly home, exactly in their home. So spent exactly within 600 square meter of their home. So here are the results for both for both campaign. The first is whether people are exactly in their home. And then there we don't find a difference. You can see that it's percent of people are exactly in their home on Thanksgiving. So 80 percent of people have left their home on Thanksgiving. And this is very different and it's for Christmas. It's also about 80 percent of people and it's very similar in treatment and control people. This is the difference between a high intensity country and a low intensity country and the P value. And they are all just the same. However, we do find that people travel less far. So and given that what they've received is one 30 second messages. We found it was quite striking that there is this this effect where on the days leading to leading to. Thanksgiving and the days leading to Christmas. So these are the days before Christmas and the days before Thanksgiving. People are quite a bit less likely minus seven on the on the basis of minus five. So it's a 15 percent decline in distance traveling or distance traveled away from their house. If they were in a high intensity country. So people seem to have responded to the messages post at Christmas and at Thanksgiving. And we were very pleasantly surprised by that. Especially since we got a lot of very aggressive post reaction to our posts. So we thought people really don't like it. They don't like the structures from the east coast to give them advice, etc. But the reality is that despite that, you know, some people still don't like it. But a lot of people are quite responsive. So the takeaway is that we detect response to public health messaging by medical professionals. Ask it. We are talking about about 20 million people touched. Facebook users respond to the sponsored content. They travel less. And there's no impact on staying home. And therefore we don't know if they are net clinical impact. So I don't want you to stay home and send that this made a difference on COVID. We don't know. Probably it won't because any effect that there is is quite small quantitatively. We are now trying to put together and look at the COVID related outcomes. But the very fact that you know everything has a small effect and then it thinks when you're progressively at up. So to conclude on COVID-19 is despite the all of this information, the the the politicization. Like traveling on Thanksgiving, wearing a mask had become very political issues. People still are affected by simple message sent by a physician, regardless of race and politics. Therefore, they are very responsive to actionable information. So before leaving you, I want to discuss quick, go to go back quickly to in a sense what is probably the next issue at hand in the U.S. Which is the question of immunization. And I'll do that. I'll do a small detour via India again. So on vaccines, we have discussed a lot in the U.S. to the question of vaccine reluctance and mistrust in particular in the black population. Which to start with more mistrustful of the health system, particularly in new drugs, etc. because of the test kit trial and the rest of the history. Work by Marcel Alsen, one of our one of the doctors in our team, shows that the acceptability of preventive error is in fact larger when the doctor is black. And in fact, she has a paper on the flu showing it for flu vaccines. Our study shows that doctors of all races are a trusted source of information, COVID-19. And to combine, suggest that there would be a key role of physicians and in particular a diverse workforce of physicians on communicating on the topic of vaccine. And I know that you're not, you know, you're honest and still a decent surgeon. But still, I don't think it matters at all in terms of your credibility in talking to people about vaccines. And your role in your community, is there in person or in an online community in communicating about vaccines, particularly in more disadvantaged community and particularly among African Americans? It turns out that I've been working a lot on immunization in my life. So one of my bigger life projects is to encourage immunization vaccinations against childhood diseases in India. So in India, there is no mistrust for immunization. In principle, everybody wants to get their chisimmonized, but in practice, they just don't. For example, in North India, less than 40% of people get the measles shot. 99% of people get the BCG because it's the first one, they get it at birth, but then they can slowly lose interest and they don't get to measles. Which is of course a huge weight because that's really one of the cheapest with civil life. So we worked on developing and testing the effectiveness of nudges for immunization. In particular, we launched a large randomness control trial in Paris, and now a couple of years ago, trying various combination of nudges in about almost 1000 villages. One more small incentives and the idea of incentive for immunization has been discussed in the US. One is SMS reminders and one is to have ambassadors, local influencers, community members who can remind people to do it. And what we found is the effect of each of these policies. What we found is that the most effective policies on average are either providing small incentives. This is the blue, the high blue bar. In fact, the highest of the small incentive. That's the high blue bar. Or to use ambassadors who and to ask to unroll them to unroll influential community member in relaying and reminding the musician. And we also tried all of the combinations of this and the most effective things which increase immunization by about 50% for immunization per month more on the basis of about 10 is to combine everything to give people incentives to send them regular SMS reminders and to have an ambassador who is an influential person in the community. But it's very expensive because if you want to provide incentives, you have to provide them even to people who would have done it anyways. So the most cost effective things to do. And in fact, the only thing that's more cost effective to do something and to just immunize is to provide people to send people SMS and to employ this human immunization ambassadors. So to use people, note in the social networks that are that like to talk and can be useful in relaying the need for being immunized. So the reason why I brought it up is that I think that can give a lot of it. I can even further increase the role of of of physicians in the spread of the vaccinations once you know logistical issues and the ability of doses are not the constraint anymore. Once we get to the issue that people have to be convinced. Simple actionable message by trusted people matter and physician of just that trusted people. And so it's good to take to Twitter and Facebook to speak to your community. It's good to be the ones who are kind of keep repeating it until you are blue in the face that people should do it without really worrying about your specialty. And beyond that, you can have this role of ambassadors and also other people can be ambassadors. So both being ambassadors yourself and maybe being the ones who are also recruiting other ambassadors that can help in relaying the message over and over again. So regardless of their actual involvement of vaccinating itself. So I'm going to stop here to leave a little bit of time for questions if people have them. Thank you so much, Dr. Adouple. That was a really terrific talk. As you alluded to, you know, containment of this pandemic has been so marked by misinformation and disinformation and as you also mentioned this send them. So it's truly impressive and fascinating to hear what you've done in studying this targeted messaging messaging. Does anyone have any questions either you can enter them into the chat and I can read them out or you can go ahead and chime in. I asked her, how are you? Good. How are you? But thank you so much for talking and I have a quick question. Are you kind of alluded to it? Are you doing any studies right now about vaccinations in this country and what maybe why people are not vaccinating and getting a message out in a similar fashion? The way that you did it through Facebook? Not yet. I hope to an item intent to and I've been in contact with some people to discuss it and I would love to do it. But the reason why we I haven't rushed to it yet is that at the moment it seems that there are more people who want to be vaccinated than either doses or logistical capabilities to do it. So it seems to be that one of the things that people who are somewhat indifferent get really discouraged by is any small barrier. The opposing of an ad you know. So it seems to me that trying to convince people now only to see them being pushed away because there are either not a little bit or they can't get an appointment or they get an appointment but those who is not there would be counterproductive. So I think it's more a second phase issues. But it certainly would would love to do that in particular go back to this issue of conglardens and to see whether we need it is particularly important to have black physicians being on my side in the black community. Mr. Duflo. Hi. I have a quick question and a comment as well. Let me start with the comments. I am absolutely stunned by the speed with which you've produced these investigations have analyzed them and are cogently presenting them to us within a year span of being able to accomplish that as extraordinary given the fact that this phenomenon really started manifesting itself only about a year ago. So this has been absolutely stunning. I think that you have done the real operation warp speed. So I congratulate you and your team. This is absolutely magnificent work. The question that I have is in your efforts at the various interventions, were you able to discriminate between the effects of authority and the effects of power on the messengers that were involved? No, it's a great question. So, first of all, thank you very much for the comment. In part, I think it stems from, as you can see, a large group of people and who had some experience doing experiments on fairly large scale in developing countries particularly. And this group of physicians from MGM's and Linn Community Center, who were just like amazing. So it was a great collaboration. And the authority thing, it is one thing that we actually did not explore that I wish we had in the sense that when people talk about credible messengers, there is always that those two aspects of it, the trust aspect and the authority aspects. And in a sense, a doctor embodies both. So we put the best foot forward by asking them, for example, to be dressed like you in medical garbs. And to kind of to embody being a doctor. But we didn't play with it. And by and large, we didn't find much difference across many, many, many different variants of the way that people presented themselves. So my hunch is that to the extent you put a doctor that is clearly identified as such a variation on this, on this dimension might not be a first order, but I don't know because I haven't tested. The reason that I made the comment is that we've seen over the course of the last year a very natural experiment in many countries in the world, perhaps the most notable or egregious, depending on where you stand, is in the United States where figures of power were able to undermine and over it in covert ways. The substantial authority of people like Tony Fauci, Bob Redfield, Jerome Adams, and a host of others. And at least in conversations with two of those three, both of whom are acquaintances and friends, they found it particularly difficult to navigate that boundary between the influence of power and the influence of authority. Oh, that's that's that's really striking. So the one way, the one place where we tried with Dr. Berks where there was some. We really believe that given what the doctors had told us that that we had focus group with that the doctor Berks message would be not as well received because she was so aligned with with with power. So that's why we had one version where we tried her and then a doctor saying exactly the same thing, but who was not Dr. Berks. And people really didn't like her. They didn't read the video well, but on the other end, they still retained the information. So I think she still had we didn't try with someone else who wasn't a doctor. She still had, you know, she still came to this with a lot of natural authority coming from a very distinguished career. So she still had a bit of both. Thank you again for all this herculean effort. Thank you. Dr. Duflo, my question isn't specifically related to your work, but how you were able to get the work done. So like you, I'm not on Facebook, but you obviously know a lot more about the impacts of Facebook on social science research. If I understood correctly, they came to you and donated the advertising. Is it common for Facebook or other large social media companies to get involved in social science research at their own expense? So I don't know. They had never came to me to ask to to throw money at me before. But they have a small group called Data for Good, who is I think a little bit marginal within Facebook, but is trying to do just that. And they came to us because we when we finished the West Bengal study, we we put out a working paper. And it was kind of out and people discussed it a little bit on Twitter again, not me because I'm not on Twitter either, but the world at large. Because they found it funny that one of the other was the one doing the messages. So it had a little bit of play for just a few days, which was enough to get in touch for to attract Facebook's attention. And then we had discussions with them. And in the end, the timing was good because what we had to offer them was very, very precise and very easy. We didn't need any data that was confidential. We didn't need any help from them except to accept the the ad credits. And so that was a good, you know, that was kind of we could organize this their contribution in this within couple of meetings. And then the interesting part was later where I found myself with really no skill in this particular domain, which might be obvious when you see the videos in, you know, getting our colleague, Dr. colleagues to record their. A possible writing the message is script Facebook told us 27 seconds. So writing the script and then getting the video within half a day by the doctor colleagues and telling them you cannot wait. You cannot. I only have half a day send them to us as quickly as possible. Then I send them to India to be edited. And I had some because we had an editing company that had worked on the previous study compiled them for Facebook and and there on their side, they were, I think much quicker than usual in getting permission to approve the messages. So I think it kind of came together like the coincidence was the coincidence was good. That's not it. And they were so excited that it worked at Thanksgiving because they told us after that usually this stuff don't work, not have any impact. So they were so excited to find an effect that then they kind of then they again they kind of up the onto for doing it at Christmas again. Right. I think we have run out of time. Thank you again so much, Dr. DuFo for this talk. We really appreciate it. I think we'll close everything out. So have a great day everyone. Thank you so much for having me and thank you for listening to me. Thank you. Thank you.