When Research Over There Isn't Helpful Here
This post was jointly written by me and Caroline Fry, assistant professor at the University of Hawai’i at Manoa! Learn more about my collaboration policy here.
This article will be updated as the state of the academic literature evolves; you can read the latest version here. You can listen to this post above, or via most podcast apps here.
According to most conventional measures of scientific output, the majority of global research takes place in a handful of countries. In the figure below, we pulled data on three measures of R&D efforts across every country in the world: number of scientific/technical articles published by researchers in a country, number of researchers engaged in R&D in a country, and R&D spending by country. We then combined that data with information on the population of every country to create the following chart, which shows the share of R&D occurring in countries with some share of the earth’s population.
Based on data from Our World in Data - source file here
According to this data, countries with about 12% of the world’s people produce half the world’s research. On the other side of the coin, half the world’s population resides in countries that collectively produce about 9% of scientific articles. The ratios are even more skewed if we rely on data on R&D spending or the number of researchers. Put differently, much of the world’s population lives in countries in which little research happens.
Is this a problem? According to classical economic models of the “ideas production function,” ideas are universal; ideas developed in one place are applicable everywhere. If this is true, then where research takes place shouldn’t be a problem. Indeed, if research benefits from clustering, we would actually prefer to concentrate our research communities into a small number of places.1
This is probably true enough for some contexts. But there are at least two problems here. First, as has been well established in the literature on technology diffusion, there are significant frictions associated with the diffusion of knowledge over geographic distances.2 Second, and what we plan to discuss in this post, research may be less useful in countries where it did not occur – or, nearly as consequential, people may believe this to be so. In this post we’ll look at four domains - agriculture, health, the behavioral sciences, and program evaluation research - where new discoveries do not seem to have universal application across all geographies.3
Thanks for reading What's New Under the Sun! Subscribe for free to receive new posts and support my work.
Different places, different problems
In a previous post we discussed some evidence that researchers tend to focus on problems in their local area. To the extent that the prevalence of problems varies around the world, this could mean that the distribution of researchers influences the levels of research to solve problems in some locations (irrespective of diffusion). If the problems of places with few researchers differ from the problems of places with many, then some problems will be under-researched if researchers focus on what’s happening locally. So the first important question is: do problems vary around the world?
Of course they do.
We can start with Moscona and Sastry (2022), which documents that the prevalence of crops and pests varies around the world. In a first step of their analysis, they use a dataset on international crop pests and pathogens from the Centre for Agriculture and Bioscience International to map the prevalence of crop pests or pathogens around the world, documenting significant variation in where crops and their associated pests and pathogens tend to be found.
From Moscona and Sastry (2022)
Similarly, it is well documented that diseases also vary around the world, due to variations in animal hosts, local climates, demographics, and socioeconomic conditions (see Wilson 2017 for a review).4 For example, the 13 parasitic and bacterial infections that make up ‘neglected tropical diseases’ primarily occur in low-income countries in sub-Saharan Africa, Asia and Latin America (Hotez et al. 2007).
From Hotez et al. 2007
Going beyond differences in the pest and disease burden, a well-known 2010 article by Henrich, Heine, and Norenzayan documents extensive variation in human psychology study results depending on the population under study. In particular, they emphasize that the study populations in behavioral science research are overwhelmingly drawn from Western, Educated, Industrial, Rich, Democracies – so-called “WEIRD” countries (indeed, they point to one study showing nearly 70% of subjects in top psychology journals came from the USA alone!). They show that along many important dimensions, findings that are derived from WEIRD samples do not generalize to the broader human population. To take one example, Henrich and coauthors point to a 1966 cross-cultural study about the Müller-Lyer illusion, presented below. In this study, American undergraduates were more likely to perceive line b to be longer than line a, though the two are actually equal in their length. Others, such as San foragers of the Kalahari, tended to perceive the lines to be of equal length.
From Henrich, Heine, and Norenzayan (2010)
A 2020 retrospective by Apicella, Norenzayan, and Henrich, which looked back on the decade since the 2010 article, found samples drawn from WEIRD countries continued to dominate major journals, even as (infrequent) studies continue to find variation across countries is important.5
Finally, economics presents another domain where results in one country may not generalize to other. For example, Vivalt (2020) assesses the extent to which results from impact evaluations of economic development interventions generalize to new contexts. To do so, the author compiles a dataset of all results across hundreds of impact evaluations covering 20 types of development programs (as an example, one type of development program is conditional cash transfers). Vivalt summarizes the variation by intervention and by intervention-outcome, using meta-analysis methods, and documents that there exists significant variation for the same intervention-outcome across contexts, and that this variation is greater than variation that exists across other types of interventions, such as medical interventions.
Trust of evidence from different places
So the problems related to agriculture, disease, human psychology, and economic development are not universal but vary substantially from region to region. If research done in one region is more likely to be related to the problems of that region (and we argued it is, here), then that means the substantial concentration of research means a lot of problems are receiving very little research effort.
Decision-makers beliefs also matter. If people believe research done elsewhere isn’t applicable to their context, then that research is less likely to inform their decisions. That’s true even if the research actually is applicable, but people don’t believe it. And some papers indicate this potential concern is a real one.
Two recent papers attempt to isolate this mechanism in the context of program evaluation evidence. Vivalt et al (2023)and Nakajima (2021) both investigate how policymakers evaluate potentially relevant research with some experiments where they surveyed policymakers on their views about different hypothetical research papers. In both of these papers, the authors provide policymakers with evidence from sets of hypothetical impact evaluations, and ask them to rank or rate which evaluations they prefer. These hypothetical evaluations vary in their methodologies (RCTs versus observational studies), results, sample size, and, importantly for this post, the location of the study. The two studies find similar results: that policymakers tend to have a preference for studies conducted in similar settings to their own country, preferably their own country (Vivalt et al 2023).
Some related evidence from medical research has similar implications. Alsan et al (forthcoming) use a similar approach, a survey experiment, to assess how doctors and patients interpret the results of clinical trial data. In this study the authors provided profiles of hypothetical diabetes drugs, which included the drug’s mechanism of action and supporting clinical trials. In a supplementary experiment the authors asked respondents in the United States how much they trusted clinical trial results conducted in different countries. They found that respondents tended to be less confident about the effectiveness of a drug tested outside of the United States, and several respondents expressed concerns that the drug would not work in the same way due to biological factors, socioeconomic and environmental factors.
(As an aside, geography is of course not the only factor affecting which kinds of populations are underserved by research. The primary experiment in Alsan et al. (forthcoming) is actually about whether representation of different racial groups in clinical trials influences the likelihood that physicians would recommend that drug to their patients, and whether patients would adhere to the drug regimen. The study randomized the share of Black trial subjects and average drug efficacy in trials across drug profiles. Physicians were asked to indicate their intent to prescribe the drugs, and in a separate experiment, hypertension patients were asked their interest in novel therapies to treat hypertension that had been tested in trial sites with varying shares of Black participants. They found that physicians were more likely to state an intention to prescribe drugs that had been tested on representative samples, and that this effect was driven by doctors who routinely saw Black patients. As for the patients, Black respondents were more likely to state that a drug would work for them if the trial was representative.)
So another rationale fo...