Beyond the headlines: Examining the methodology of the Johns Hopkins paper that suggests lockdowns barely reduced mortality

Despite being criticized by epidemiologists and economists, a systematic review and meta-analysis from Johns Hopkins University continues to be cited as a case against lockdowns. The authors of the paper conclude that lockdowns had little to no effect on COVID-19 mortality. In their analysis, mortality was only reduced by an average of 0.2% to 2.9%, depending on the type of study. The authors concluded that “lockdown policies are ill-founded and should be rejected as a pandemic policy instrument." The conclusion has proved popular with those who criticize public health measures. However, the paper is only one of a number of studies examining the outcomes of policies to reduce the spread of COVID-19. “I think it is like any other research paper in medicine or epidemiology, there’s no one paper that provides the answer,” said Erin Strumpf, an associate professor in the department of economics and the department of epidemiology, biostatistics and occupational health at McGill University. “This is one piece of evidence, and it has strengths and weaknesses as a study, just like any other study.” The paper, known colloquially as “the Johns Hopkins paper” is a working paper, and has not been formally peer-reviewed. Its senior author is economist Steve Hanke, co-director of the Johns Hopkins Institute for Applied Economics, Global Health and the Study of Business Enterprise and in charge of the Studies in Applied Economics series of papers, where the paper in question is to be published. A preprint of the paper was circulated in January. The other two authors are Jonas Herby, an advisor at the Centre for Political Studies in Copenhagen and Lars Jonung, a professor emeritus from Lund University in Sweden. All three authors are respected economists in their field. A number of other researchers were quick to criticize the paper and its findings. “Smoking causes cancer, the earth is round, and ordering people to stay at home (the correct definition of lockdown) decreases disease transmission. None of this is controversial among scientists. A study purporting to prove the opposite is almost certain to be fundamentally flawed,” wrote Seth Flaxman, associate professor in the department of computer science at the University of Oxford, in a comment to the U.K.'s Science Media Centre. Flaxman is the lead author of a 2020 paper published in Nature that estimated the effects of non-pharmaceutical interventions on COVID-19 transmission in Europe. Read: Swabbing floors to detect COVID-19 could be a useful indicator of the disease’s spread Defining a lockdown The authors conducted a meta-analysis, but the papers selected include different interventions. Strumpf gave the example of a meta-analysis of all treatments for breast cancer that includes everything from chemotherapy to radiation to surgery. “There’s a little bit of that going on in this paper, that all the different policy responses are quite different,” she told the Medical Post. “To call them all lockdowns, I’m not really sure that’s a wise way to go.” The Johns Hopkins paper defines lockdowns as “the imposition of at least one compulsory, non-pharmaceutical intervention (NPI). NPIs are any government mandate that directly restrict peoples’ possibilities, such as policies that limit internal movement, close schools and businesses, and ban international travel.” Yet, “the policies which comprised “lockdown” varied dramatically between countries, meaning defining the term is problematic,” said Neil Ferguson, director of the MRC Centre for Global Infectious Disease Analysis at the Jameel Institute at Imperial College London. Ferguson provided a comment on the paper for the U.K. Science Media Centre. By the definition used in the Johns Hopkins paper requiring people to self-isolate if they have COVID-19 is a lockdown, Ferguson said. Read: New turning point? More countries easing COVID restrictions Studies chosen for the meta-analysis The authors of the Johns Hopkins paper are transparent about their search and exclusion criteria for their systematic review and meta-analysis. Twenty-four studies were included in the meta-analysis, most of which included data up to June 2020. The authors included mostly on difference-in-difference studies. This study design makes the assumption that nothing else is changing at the same time as the policy that is the focus of the study. “I don’t think that’s a very reasonable assumption in the context of COVID-19,” Strumpf said. “Not only were there other policy changes going on besides the lockdown, there is a lot of solitary behaviour change.” For example, people reduced their contacts at the same time as public measures were put in place. The quality of any comparative study depends on the quality of the control group and “here, whether the control group is reasonable or not has to be evaluated on a case-by-case basis,” Strumpf said. There is also an inherent bias in all of the studies, in that it was precisely the countries who were having high case counts or high mortality that imposed strict public health measures, Strumpf said. The papers included in the meta-analysis were mostly economics papers. Flaxman said that the authors “systematically excluded from consideration any study based on the science of disease transmission.” The authors also excluded synthetic control method papers. “Some of these papers find a significant impact of NPIs on mortality so including them might have led to somewhat higher average mortality effects,” said David Paton, chair of Industrial Economics at Nottingham University Business School. “The paper gives a robust defence of their exclusion, but I think you would get people on both sides of that debate.” Read: Fewer LTC deaths during Omicron wave despite high infections Mortality as a blunt measure of outcome The Johns Hopkins paper’s authors are clear that the focus of their analysis is mortality. Some argue that mortality may be a blunt measure of the effects of public health measures, since they were put into place for more reasons than mortality, factors which included transmission or case counts, hospitalization, effect on the healthcare system and healthcare workforce, and testing capacity. Ferguson noted that since non-pharmaceutical interventions affect transmission of the virus, researchers can’t assume that interventions have fixed effects on outcome measures (i.e., deaths) over a certain time window. “Interventions affect transmission rates, and therefore the appropriate outcome measures to consider are growth rates (of cases or deaths) over time, with appropriate time lags—not total cases or deaths." Also, mortality prior to a lockdown will be higher than after it, since lockdowns are introduced in response to growing illness in the population. “Neither is lockdown a single event as some of the studies feeding into this meta-analysis assume; the duration of the intervention needs to be accounted for when assessing its impact,” Ferguson said. What next? As governments lift restrictions across Canada policy-makers, they need to be clear about what factors will prompt public health measures to be reinstated, André Picard wrote in a Feb. 15, 2022 Globe and Mail opinion piece. This means the question of which public health measures are effective remain important to examine. Unfortunately, studying the effects of lockdowns on mortality will continue to be difficult because of the lack of an adequate control group. This study design challenge is present when studying all waves of the COVID-19 pandemic. “This is a question we’d all like to be able to answer but I’m not sure we’re going to be able to do that very convincingly given the tools that we have and the way things played out in real life,” Strumpf said. Paton, the economist, said that both the systematic review and meta-analysis in the Johns Hopkins paper “make a significant contribution to our understanding of lockdown effects.” However, it is only one of numerous paper looking at the utility and effectiveness of public health measures during COVID-19. “There’s never one study that provides the smoking gun kind of answer,” Strumpf said. Read: A deep dive into the OMA's proposed plan to embrace ambulatory surgery centres in Ontario

Beyond the headlines: Examining the methodology of the Johns Hopkins paper that suggests lockdowns barely reduced mortality

Register for free access or login.

Related Topics