Before Today’s Post, an Announcement: The Institute for Progress is hosting a free 6-week online PhD course titled “The economics of ideas, science and innovation.” I’m teaching one of the sessions, and Pierre Azoulay, Ina Ganguli, Benjamin Jones, and Heidi Williams are teaching the rest. An all-star lineup! The course is aimed at economics PhD students who want to learn more about the economics of innovation, but we’re also open to applications from PhD students in related fields or recent graduates. The course starts November 1, but the deadline to apply is September 6. Learn more here! Now for your regularly scheduled content… Like the rest of New Things Under the Sun, this article will be updated as the state of the academic literature evolves; you can read the latest version here. You can listen to this post above, or via most podcast apps: Apple, Spotify, Google, Amazon, Stitcher. For decades, the office was the default way to organize workers, but that default is being re-examined. Many workers (including me) prefer working remotely, and seem to be at least as productive working remotely as they are in the office. Remote capable organizations can hire from a bigger pool of workers than is available locally. All in all, remote work seems to have been underrated, relative to just a few years ago. But there are tradeoffs. I’ve written before that physical proximity seems to be important for building new relationships, even though those relationships seem to remain productive as people move away from each other. This post narrows the focus down to the office. Does bringing people together in the office actually facilitate meeting new people? (spoiler: yes) But I’ll try and get more specific about how, when, and why this happens too. One aside: this is a rich literature that goes back decades. I’m going to focus on relatively recent research that looks at scientists and startups and uses experimental and quasi-experimental approaches. But a lot of this recent work turns out to echo what earlier studies found using more observational approaches. Allen and Henn (2007) provides one overview of some of the older literature. Subscribe now Academic Collaboration Among Neighbors Let’s start with buildings. Are people more likely to work together on a project if they also work in the same building? Miranda and Claudel (2021) look at what happens to collaboration between MIT-affiliated professors and staff when they start working in the same building (or get separated), due to a series of renovation and new building projects over 2005-2015. Every year they look at each pair of 1,417 MIT authors to see if the authors’ offices are in the same building, and if they were coauthors on a paper. They want to estimate the impact of being in a building together, which presents a bit of a challenge. We might expect people to seek out offices in the same building as their expected collaborators, but they would have ended up working together whether they succeeded in getting colocated offices or not. That could overstate the impact of being in the same building. So Miranda and Claudel try to estimate the impact of being in the same building, after you adjust for a particular pair of author’s underlying propensity to collaborate regardless of location.1 Essentially, pick a random pair of MIT coauthors and identify two years where they had the same number of publications in the previous year. If they were in the same building in one of these comparison years and not in the same building in the other, they tended to publish an extra 0.004 papers together in the year they were in the same building. An extra 0.004 papers might not seem like much, but that’s because most random pairs of MIT scientists do not put out any papers together in a given year. With 1,417 MIT authors, there are over a million possible ways for them to pair off, but they only put out 38,000 papers written by multiple MIT authors collectively over the decade. That works out to about 0.004 papers per pair per year, which implies moving people into the same building about doubles the number of papers they might be expected to put out together.2 That’s about the same order of magnitude found by Catalini (2017). Catalini focuses on the Université Pierre-et-Marie-Curie and it’s 17 year quest to remove asbestos from its buildings. Asbestos removal required moving labs to new locations, typically based on what space was available rather than as a way to make inter-lab collaboration easier. Catalini also finds when labs are moved into the same building, they put out 2.5-3.3x as many joint publications as pairs of labs that are not moved together. Going Inside the Building That’s for two people (or groups) working in the same building. But buildings can be pretty big. What if we look within the building; do we see similar effects for people with offices that are closer or farther away from each other? Roche, Oettl, and Catalini (2022) peers within a US co-working space that hosted 251 different startups over 2014-2017. Whereas Miranda and Claudel (2021) and Catalini (2017) needed to try and convince us that building moves were basically random due to renovation, in this case the startup residents actually were randomly allocated to different places in the co-working hub. Very convenient for the researchers! A difficulty is startups do not typically collaborate on easily observable projects like scientific papers though. Instead, Roche, Oettl, and Catalini look for evidence that the startups trade information using data from BuiltWith that describes which web technologies startups use. For example, NewThingsUnderTheSun.com is in the BuiltWith dataset, and it shows I use CloudFlare for a bunch of stuff, and that I registered the domain name from Tucows. Suppose I moved into a coworking space with a bunch of startups that used a web technology called Mixpanel for A/B testing. Roche and coauthors can see this in their dataset. If I started using Mixpanel myself to do A/B testing for NewThingsUnderTheSun.com after moving into the coworking space, then that suggests I learned about Mixpanel from some of the other startups there. Roche, Oettl, and Catalini measure the shortest walking distance between each pair of startups on the same floor (walking distance is the shortest path you could actually walk, respecting walls, furniture, etc) and then they look at the probability startups adopt each other’s component web technologies. As you might expect, the closer two startups workspaces are, the more likely they are to use each other’s stuff. What’s perhaps a bit surprising though is that the effect of distance is highly nonlinear. Divide the startup pairs into four groups, based on their proximity, and you find only the 25% that are closest exhibit any knowledge sharing. It looks like being in the same building only matters if you are actually really close - like, within 66 meters! Additional probability of adopting another startups web tech, dividing distance into 4 bins. From Roche, Oettl, and Catalini (2022) This echoes a common finding in some of the older literature I alluded to earlier. Proximity matters, but for most people the value of proximity falls off very fast. If you have to walk very far to talk with a colocated coworker, then that coworker might as well not be colocated. Hasan and Koning (2019) get similar results in the context of a startup bootcamp in India. They randomly assign 112 aspiring entrepreneurs to 40 different teams, whose location in a large open co-working space is also randomly assigned. Bootcamp attendees spent their first week developing a project that was later evaluated by the team, and Hasan and Koning study how proximity between teams affected their interactions during this week. To measure interactions, they survey people after a week (do you know this person? Did you ask them for advice?) and also see if they sent each other more messages via email or Facebook. As with Roche and coauthors, the impact of very minor distances seems to matter a lot. The probability bootcamp attendees reported they knew, sought advice from, or frequently messaged people on other teams dropped rapidly as distance increased (focus on the black lines below, for now - we will discuss the dashed ones shortly). Probability of working with members of other teams (vertical axis, black solid line), as a function of walking distance (horizontal axis). From Hasan and Koning (2019) It’s also worth noting that all the teams in this study were as close as the teams in the first quartile of the Roche, Oettl, and Catalini (2022) study, so even among the top 25% closest startups, it seems likely the very closest exchanged most of the information. And note, in both of these studies, the locations of teams was random - it’s not as if people were grouped by the similarity of their work. And yet, proximity seemed to matter quite a bit for information sharing anyway. Communication or Discovery? So far, we’ve found evidence that jamming people together in a building increases the probability that they exchange information and start joint projects, especially if their workplaces are very close within the building. This could be for at least two different reasons though. First, being close might make it easier for people to communicate. We know this is true, in the sense that you literally don’t have to walk so far to talk face-to-face with someone who is nearby. If face-to-face conversation is a much better way to trade information than digital messaging, then we expect close coworkers to trade more information. They might also decide to start more scientific projects together, because they know it’ll be easier to complete those projects when it’s so easy to communicate. Call this the communication advantage of proximity. Second, being close might make it easier to meet new people. You might not march across the room to introduce yourself to someone you...

New Things Under the Sun is a living literature review; as the state of the academic literature evolves, so do we. This post highlights some recent updates. An Internet of Ink and Paper The post “The Internet and Access to Distant Ideas” highlighted three studies from the early days of the US internet to illustrate how access to the internet facilitated innovation. Firms who are connected to each other by the internet are more likely to collaborate on patents or cite each other’s work, and counties that would normally be left behind by rising geographic concentration of patenting were better able to buck the trend if they enjoyed greater internet penetration. Thanks for reading What's New Under the Sun! Subscribe for free to receive new posts and support my work. This post has now been updated to include discussion of a new paper by Hanlon and coauthors, which documents the same kinds of effects for a very different change in the technology of long-distance communication: This isn’t the first time we’ve seen something like the dynamics brought about by the internet. Hanlon et al. (2022) travel even further back in time to 1840 in Great Britain to study what happens to science and invention when the price of the mail drops. Prior to 1840, the cost of posting a letter in Great Britain varied substantially based on the distance the letter needed to travel, as can be seen in the figure below. But in 1840, a greatly simplified pricing system was introduced: posting a domestic letter, of any distance, cost 1 penny. Modified from Hanlon et al. (2022) As with the preceding papers, Hanlon and coauthors want to know how this drop in the price of long-distance communication affected collaboration (in science this time) and invention. Though it may seem a bit niche to contemporary readers based outside the UK, as a natural experiment in the effects of communication, this setting has several virtues. In this era, pretty much the only way to communicate with people at a distance was by personal travel or via the postal system (telegrams at this time were primarily used by the railroads, not the general public). So if long-distance communication is important, this price change should matter. Because prices prior to reform were based on distance, we actually have a lot of variation to work with. Distant towns experienced a big price cut in the costs of communication and nearby towns experienced only a small price cut. We can look to see if the effects of the reform varied across those contexts. The price changes were substantial enough, by the standards of the day, to matter. The price of mailing a one-page letter from London to Edinbourgh fell from 10-20% of a professor’s daily salary to 0.5-1%! Also suggesting the price cuts were material, there was a very large increase in mail posted following the reforms. To track the impacts, Hanlon and coauthors do two analyses. The first is based on the citations made by articles published in the premier scientific journal of the day, the Philosophical Transactions of the Royal Society of London. For the ten years before and after the postal pricing reform, they locate where the scientists publishing in the Royal Transactions live and where the scientists they cite live. This gives them 1,251 citations between scientists in different parts of Great Britain. Analogously to Forman and Zeebroek (2019), they show the postal price cut increased citations between towns, and that this effect was larger for towns where correspondence was previously more expensive. Specifically, the price cuts reduced the “distance” penalty, wherein towns that are farther apart cite each other less, by 70%. Hanlon and coauthor’s second analysis tries to assess the impact of the reform on new patents. For this, they have to take a different approach, because even if a patent is drawing on distant knowledge (obtained through mail correspondence), this isn’t really visible in the patent document. Patent citations in this era was not a big thing, nor was collaboration at a distance. After locating where each inventor resides, Hanlon and coauthors try to estimate, for every town, how much did the postal reform affect that specific town’s access to ideas from the rest of Great Britain. By this measure, a town that is very remote from all others would experience a big increase in its access to distant ideas, since prior to the pricing reform it would have been quite expensive to correspond with most of the people in Great Britain. In contrast, a town that lies within a geographical cluster of several large population centers may have experienced a much smaller increase in its access to distant ideas. There are some other complicating details, but again they find the same flavor of result as earlier papers: patents increased by a larger amount in more remote towns, following the introduction of uniform postal pricing. So in two quite different settings we observe the same general phenomenon: when communication at a distance becomes easier, access to distant ideas is improved and this has a disproportionate benefit to places that are otherwise far from where the inventive action is. It didn’t make it into the update, but reading these history papers I am always impressed by the amount of work that has to go into creating the dataset. It’s no small thing to locate where each inventor lives in every year, which post office is closest, and how much it would cost to correspond with other post offices! You can read the rest of the article (now renamed “The internet, the postal service, and access to distant ideas”) including the pre-existing bits about the early internet, here: Read the whole thing Networking at Academic Conferences The post “Academic Conferences and Collaboration” surveyed a few papers that document how academic conferences can be useful for forging new collaborations. This post has been updated to include discussion of a new paper that tackles this question in a new way: Instead of comparing people who attend a conference to those that do not, you can also look within a conference and see if attendees who interact more often during the conference are more likely to collaborate on new projects. Two papers find that is also the case. Zajdela et al. (2022) examine four recent conferences (around 60 attendees), that mixed large topic discussions of around 10 people with small group discussions of 3-4 people. Zajdela and coauthors estimate how much time people spent interacting at the conferences based on their joint assignment to different sessions (they assume you might have interacted more if the session was longer or if the number of attendees was smaller). At the end of the conference they can see if people spontaneously teamed up to submit a proposal for research funding. Do people who spent more time in the same sessions team up at a greater rate? Yes! But that doesn’t tell us much unless we know how these groups were formed. Maybe the conference organizers tried to match people up who they thought were most likely to want to work together; and maybe these people would have identified each other no matter what, in a conference with just 60 attendees. In that case, time spent in sessions together doesn’t matter - these people would always have collaborated. Fortunately, Zajdela and coauthors also know the algorithm which was used to assign people to small and large group sessions. The conferences tried to optimally place people together according to some seemingly desirable, but possibly conflicting, rules.undefined Because this group assignment problem is very complex, the algorithm doesn’t exactly solve for the “best” outcome by these criteria. Instead, it just tries to get as close as it can, and there is a bit of randomness in where it ends up. Zajdela and coauthors re-run this algorithm a bunch of time to come up alternative conference schedules, each of which might well have been the actual schedule but for a bit of algorithmic luck. Then they look to see if collaboration is highly correlated with the actual time spent interacting, rather than the potential time interacting under alternative plausible conference schedules. And it is: among people who did not previously know each other, collaboration was about 9x more likely for pairs that actually attended a small group session together, as compared to pairs who did not attend a small group session together in the real world but would have in alternative possible conference schedules. The post is also updated with a paragraph discussing some results of Lane et al. (2019), which is a longer run follow-up of one of the other papers discussed in the original post (Lane et al. (2019) has also been covered in more detail here). Read the whole thing Responding to a Good Counterargument to a Recent Post The recent post “How common is independent discovery?” surveyed a few lines of evidence to think through how much redundancy there is in science and invention: if the discoverer of some idea had gotten sidetracked and never made the discovery, how likely is it someone else would have come along to make the discovery instead? An email correspondent responding to that post made a really good counterargument to my interpretation of the evidence. I thought a good response to the counterargument was possible, but it would require drawing on a few additional papers. However, since “How common is independent discovery?” was already about as long as I want posts on New Things Under the Sun to be, rather than adding more discussion to that post, I instead decided to split what used to be one long article into two shorter articles. So now there are two (interrelated) articles related to this topic. The original “How common is independent discovery?” has been reorganized and shortened to focus narrowly on papers about exactly what the title promises: independent discovery. Meanwhile a new post titled “Contingency and Science” is now ...

Like the rest of New Things Under the Sun, this article will be updated as the state of the academic literature evolves; you can read the latest version here. Audio versions of this article will be available in a few days (I’m traveling). A huge quantity of academic research that seeks to understand how science works relies on citation counts to measure the value of knowledge created by scientists. The basic idea is to get around serious challenges in evaluating something as nebulous as knowledge by leverage two norms in science: New discoveries are written up and published in academic journals Journal articles acknowledge related work by citing it If that story works, then if your idea is influential, most of the subsequent knowledge it influences will eventually find its way into journal articles and then those articles will cite you. By counting the number of journal articles citing the idea, we have a rough-and-ready measure of it’s impact.1 This measure of scientific impact is so deeply embedded in the literature, that its absolutely crucial to know if it’s reliable. So today I want to look at a few recent articles that look into this foundational question: are citation counts a good measure of the value of scientific contributions? What is Value? Citations as a measure of academic value is a sensitive topic, so before jumping in, it’s important to clarify what value means in this context. There are at least two things that are bundled together. First, there is what we might call the potential value of some discovery. Did the discovery uncover something true (or directionally true) about the universe that we didn’t know? If widely known, how much would affect what people believe and do? How would it be assessed by an impartial observer with the relevant knowledge set? Second, there is the actual impact of the discovery on the world out there. Did the discovery actually affect what people believe and do? More specifically, did it affect the kinds of research later scientists chose to do? If science is working well, then we would hope the only difference between these two is time. Optimistically, good ideas get recognized, people learn about them, incorporate the insights into their own research trajectories, and then cite them. In that case, potential value is basically the same thing as the actual impact if you let enough time pass. But we have a lot of reasons not to be optimistic. Maybe important new ideas face barriers to getting published and disseminated, because of conservatism in science, or because of bias and discrimination. Or, if those obstacles can be surmounted, maybe there are barriers to changing research directions that prevent scientists from following up on the most interesting new ideas and allowing them to reach their potential. Or maybe low potential ideas garner all the attention because the discoverers are influential in the field. In that case, citations still reflect actual impact, in the sense that they really do capture how ideas affect what people believe and do. But in this less optimistic scenario, impact and potential value have been partially or completely decoupled, because science isn’t very good at translating potential value into realized value. It lets good ideas go to waste and showers disproportionate attention on bad ones. But it’s also possible that citations don’t even reflect actual impact. This would be the case, for example, if citations don’t really reflect acknowledgements of intellectual influence. Maybe people don’t read the stuff they cite; maybe they feel pressured to add citations to curry favor with the right gatekeepers; maybe they just add meaningless citations to make their ideas seem important; maybe they try to pass off other people’s ideas as their own, without citation. If these practices are widespread, then citations may not reflect much of anything at all. I’m going to end up arguing that citations are reasonably well correlated with actual impact, and science works well enough that actual impact is also correlated with potential impact. That’s not to say there are no problems with how science works - I think there are plenty - but the system isn’t hopelessly broken. Finally, bear in mind that my goal here is mostly to assess how useful are citation counts in the context of academic papers that study how science functions. That’s a context where we typically have a lot of data: always hundreds of papers, and usually thousands. With a lot of observations, even a small signal can emerge from a lot of noise. In contexts with many fewer observations though, we shouldn’t be nearly so confident that citations are so valuable. If you are trying to assess the contribution of a single paper, or a single person, you shouldn’t assume citation counts are enough. To get a better sense of value in this context, unfortunately you probably have to have someone with the relevant knowledge base actually read the paper(s). OK, onward to what we find when we look into these questions. Why do people cite papers? The whole point of tracking citations is the assumption that people acknowledge the influence of previous discoveries by citing the relevant papers. Is that really what citations do though? Teplitsky et al. (2022) tries to answer this question (and others) by asking researchers about why they cited specific papers. In a 2018 survey, they get responses from a bit over 9,000 academics from 15 different fields on over 17,000 citations made. Surveys are personalized, so that each respondent is asked about two citations that they made in one of their own papers. Teplitsky and coauthors construct their sample of citations so that they have data on citations to papers published in multiple years, and which span the whole range of citation counts, from barely cited to the top 1% most cited in the field and year. Among other things, the survey asks respondents “how much did this reference influence the research choices in your paper?”, with possible answers ranging from “very minor influence (paper would’ve been very similar without this reference)” to “very major influence (motivated the entire project).” Assessed this way, most citations do not reflect significant influence. Overall, 54% of citations had either a minor or very minor influence. Given the way these options were explained to respondents, that’s consistent with most citations affecting something less than a single sentence in a paper. Only about 18% of citations reflect major or very major influence (for example, they influenced the choice of theory, method, or the whole research topic). That implies citations are a very noisy way of measuring influence. But there’s an interesting twist. It turns out the probability a paper is influential is not random: more highly cited papers are also more likely to be rated as major or very major influences. From Teplitsky et al. (2022) Notice the overall data line says “with citer effects.” That’s intended to control for the possibility that there might be systematic differences among respondents. Maybe the kind of researchers who lazily cite top-cited work are also the kind of people who lazily answer surveys and just say “sure, major influence.” But Teplitsky and coauthors survey is cleverly designed so they can separate out any potential differences among the kind of people who cite highly cited work versus those who do not: they can look at the probability the same person rates a paper as more influential than another if it also has more citations. Overall, when you additionally try to control for other features of papers, so that you are comparing papers papers the survey respondent knows equally well (or poorly), the probability they will rate a paper as influential goes up by 34 percentage points for every 10-fold increase in citations. So I take a few things from this survey. First, there is a ton of noise in citation data; just as not all papers are equal, so too are not all citations equal. A paper with 5 citations is quite plausibly more influential than one with 10. But all else equal, there is a pretty strong relationship between the number of citations a paper gets and how influential it is. This measure is subject to a lot of noise, but among very highly cited papers, the relationship between citations and influence is actually stronger than it is for less cited papers. Not only is a paper with 1000 citations more likely to be influential than one with 500 simply because it has so many more chances to be influential, but additionally because each of those chances has a higher probability of being influential. Uncited Influences Note, however, that Teplitsky and coauthors start with citations: they observe a citation that was made and ask the citer why they made it. But that design means it’s impossible to identify work that is influential but uncited. Fortunately, new natural language processing techniques allow us to start answering that as well. Gerrish and Blei (2010) propose a new method to measure the influence of academic papers by looking at how much they change the language of papers that come later. They then show that, indeed, if you try to identify influential papers merely based on the relationships between their text and the text of later articles, there is a reasonably strong correlation between language influence and citations. Gerrish and Blei start with topic models. These are a way of modeling the words used in a paper as the outcome of a blind statistical process. We pretend there are these things out there called “topics” which are like bags of words, where you reach into the bag and pull out a word. Different topics have different mixes of words and different probabilities of grabbing specific words. Next, we pretend papers are nothing more than a bunch of words we grab out of different topic bags. As an example, if I’m writing a paper on the impact of remote work on innovation, then maybe half ...

Like the rest of New Things Under the Sun, this article will be updated as the state of the academic literature evolves; you can read the latest version here. Audio versions of this and other posts: Substack, Apple, Spotify, Google, Amazon, Stitcher. An old divide in the study of innovation is whether ideas come primarily from individual/group creativity, or whether they are “in the air”, so that anyone with the right set of background knowledge will be able to see them. As evidence of the latter, people have pointed to prominent examples of multiple simultaneous discovery: Isaac Newton and Gottfried Liebnitz developed calculus independent of each other Charles Darwin and Alfred Wallace independently developed versions of the theory of evolution via natural selection Different inventors in different countries claim to have invented the lightbulb (Thomas Edison in the USA, Joseph Swan in the UK, Alexander Lodygin in Russia) Alexander Graham Bell and Elisha Grey submitted nearly simultaneous patent applications for the invention of the telephone In 1922, Ogburn and Thomas compiled a list of nearly 150 examples of multiple independent discovery (often called “twin” discoveries or “multiples); wikipedia provides many more. These exercises are meant to show that once a new invention or discovery is “close” to existing knowledge, then multiple people are likely to have the idea at the same time. It also implies scientific and technological advance have some built in redundancy: if Einstein had died in childhood, someone else would have come up with relativity. But in fact, all these lists of anecdote show is it is possible for multiple people to come up with the same idea. We don’t really know how common it is, because these lists make no attempt to compile a comprehensive population survey of ideas. What do we find if we do try to do that exercise? Simultaneous Discovery in Papers and Patents A number of papers have looked at how common it is for multiple independent discovery to occur in academic papers. An early classic is Hagstrom (1974), which reports on a survey of 1,947 academics in the spring of 1966. Hagstrom’s survey asked mathematicians, physicists, chemists, and biologists if they had ever been “anticipated”; today, we would call this getting scooped. Getting scooped isn’t that uncommon: 63% of respondents said they had been scooped at least once in their career, 16% said they had been scooped more than once. For our purposes, the most illuminating question in Hagstrom’s survey is “how concerned are you that you might be anticipated in your current research?” Fully 1.2% of respondents said they had already been anticipated on their current project! Let’s assume people are, on average, halfway through a research project. If they have a constant probability of being scooped through the life of a project, then that implies the probability of getting scooped on any given project is on the order of 2.5%, at least in 1966. Hill and Stein (2020) get similar results, studying the impact of getting scooped over 1999-2017 for the field of structural biology. Structural biology is a great field for studying how science works because of its unusually good data on the practice of science. Structural biologists try to figure out the 3D structure of proteins and other biological macromolecules using data on the diffraction of x-rays through crystalized proteins. When they have a model that fits the data well, the norm (and often publication requirement) is to submit the model to the Protein Data Bank. This submission is typically confidential until publication, but creates a pre-publication record of completed scientific work, which lets Hill and Stein see when two teams have independently been working on the same thing. Since almost all protein models are submitted to the Protein Data Bank, Hill and Stein really do have something approaching a census of all “ideas” in the field of structural biology, as well as a way of seeing when more than one team “has the same idea” (or more precisely, is working on closely related proteins). Overall, they find 2.9% of proteins involve multiple independent discovery, as defined above, quite close to what Hagstrom reported in 1974. Painter et al. (2020) takes yet another approach to identifying multiple simultaneous invention, this time in the field of evolutionary medicine (2007-2011). Their approach is to identify important new words in the text of evolutionary medicine articles, and then to look for cases where multiple papers introduce the same new word at the roughly the same time. In their context, this usually means an idea has been borrowed from another field (where a word for the concept already exists) and they are looking for cases where multiple people independently realized a concept from another field could be fruitfully applied to evolutionary medicine. To identify important new keywords, they take all the words in evolutionary medicine articles and algorithmically pick out the ones unlikely to be there based their frequency in American English. This gives them a set of technical words that are not common English words. They build up a dictionary of such terms mentioned in papers published between 1991 and 2006; these are words that are “known” to evolutionary biology in 2007. Beginning in 2007, they look for papers that introduce new technical words. Lastly, they consider a word to be important if it is mentioned in subsequent years, rather than once and never again. Over the period they study, there were 3,488 new keywords introduced that went on to appear in at least one subsequent year. Of this set, 197 were introduced by more than one paper in the same year, or 5.6%. As a measure of independent discovery, that’s probably overstated, since it doesn’t correct for the same author publishing more than one paper using the same keywords. Again, I think something in the ballpark of 2-3% sounds plausible. Painter and coauthors go on to focus on a small subset of 5 keywords that were simultaneously introduced by multiple distinct people and which were very important, being mentioned not just again, but in every subsequent year. Bikard (2020) is another attempt to identify instances of multiple independent discovery, though in this case it’s harder to use the data to estimate how common they are. Bikard argues that when the same two papers are frequently cited together in the same parenthetical,1 then that is evidence they refer to the same underlying idea. Bikard algorithmically identifies a set of 10,927 such pairs of papers in the PubMed database and shows they exhibit a lot of other hallmarks of being multiple independent discoveries: they are textually quite similar, published close in time, and frequently published literally back-to-back in the same journal issue, which is one way journals acknowledge co-discovery. Given 29.3 million papers in PubMed, if there are only 10,927 instances of multiple discovery, that would naively suggest something on the order of 0.03% of papers having multiple independent discovery. But while Bikard’s publicly available database of twin discoveries is useful for investigating a lot of questions related to science, it’s less useful for ascertaining the probability of independent discovery. That’s because the algorithm requires articles to have the right mix of characteristics to be identified as simultaneous discoveries. For example, in order to identify if two articles are frequently cited together in the same parenthetical block, Bikard needs each paper to receive at least 5 citations, and he needs at least three papers that jointly cite them to have their full text available, so he can see if those citations happen in sequence inside a parentheses. It’s unclear to me how many of the 29.3mn papers in PubMed meet this criteria. But we can at least say that as long as no less than 1 in 100 papers meet the criteria, then Bikard’s method suggests a rate of simultaneous discovery that is significantly lower than 3%. To close out this section, let’s turn to patents. Until 2013, the US patent system featured an unusual first-to-invent system wherein patent rights were awarded not to the first person to seek a patent but to the first person to invent it (provided certain conditions were met). This meant that if two groups filed patents for substantively the same invention, the US patent office initiated something called a “patent interference” to determine which group was in fact the first to invent. These patent interferences provide one way to assess how common is simultaneous invention at the US patent office. Ganguli, Lin, and Reynolds (2020) have data on all 1,329 patent interference decisions from 1998-2014. Of this set, it’s not totally clear how many represent actual simultaneous invention. In a small number of cases (3.5%), the USPTO ruled there had in fact been no interference, but in some cases one party settles or abandons their claim, or ownership of the patents is transferred to a common owner. In these cases, we don’t know necessarily know if the patents were the same. But it turns out this doesn’t really matter for making the argument that simultaneous invention is very rare. For the sake of argument, let’s assume all 1,329 patent interference decisions correspond to cases of independent discovery. On average, it takes a few years for a patent interference decision to be issued. So let’s assume, for the sake of argument, these decisions come from the set of granted patents whose application was submitted between 1996 and 2012. Some 6.3mn patents applications (ultimately granted) were submitted over this time period, which implies 0.02% of patent applications face simultaneous invention. That’s a lot less than the 2-3% we found in some academic papers! Inferring the Probability of Rediscovery All these approaches suggest simultaneous discovery is in fact not very common. But simultaneous discovery i...

This is an audio read-through of the initial version of How Common is Independent Discovery? Like the rest of New Things Under the Sun, the underlying article upon which this audio recording is based will be updated as the state of the academic literature evolves; you can read the latest version here.

New Things Under the Sun is a living literature review; as the state of the academic literature evolves, so do we. This post highlights three recent updates. How Distortionary is Publish-or-Perish to Science? As I wrote earlier this month, science appears to be getting harder. One possible cause of this is increasing competition and the incentive to publish. Maybe scientists can only keep up in the publishing race by doing increasingly slap-dash work? The article Publish-or-perish and the quality of science looked at some evidence on this in two very specific contexts where we have exceptionally good data. A new update adds in some papers that rely on poorer quality data, but which are able to assess a much wider set of contexts: We can find complementary evidence in two additional papers that have far less precision in their measurement but cover much larger swathes of science. Fanelli, Costas, and Larivière (2015) and Fanelli, Costas, and Ioannidis (2017) each look for statistical correlations between proxies for low quality research and proxies for pressure to publish. When we zoom out like this though, we find only mixed evidence that publication pressures are correlated with lower quality research. Fanelli, Costas, and Larivière (2015) look at the quality of research by focusing on a rare but unambiguous indicator of serious problems: retraction. If we compare authors who end up having to retract their papers to those who do not, do we see signs that the ones who retracted their papers were facing stronger incentives to publish? To answer this, Fanelli, Costas, and Larivière (2015) identify 611 authors with a retracted paper in 2010-2011, and match each of these retracted papers with two papers that were not retracted (the articles published immediately before and after them in the same journal). Fanelli, Costas, and Ioannidis (2017) look at a different indicator of “sloppy science.” Recall in Smaldino and McElreath’s simulation of science, one aspect of a research strategy was the choice of protocols you used in research. Some protocols were more prone to false positives than others, and since positive results are easier to publish, labs that adopt these kinds of protocols accumulate better publication records and tend to reproduce their methods. This form of publication bias leads statistical fingerprints that can be measured.undefined Fanelli, Costas, and Ioannidis (2017) tries to measure the extent of publication bias across a large number of disciplines and we can use this as at least a partial measure of “sloppy science.” Each of these papers then looks at a number of features that, while admittedly crude, are arguably correlated with stronger incentives to publish. Are the authors of retracted papers more likely to face these stronger publication pressures? Are the authors of papers that exhibit stronger signs of publication bias more likely to face them? One plausible factor is the stage of an author’s career. Early career researchers may face stronger pressure to publish than established researchers who are already secure in their jobs (and possibly already tenured). And indeed, each paper finds evidence of this: early career researchers are more likely to have to retract papers and showed more evidence of publication bias, though the impact on publication bias was quite small. Another set of variables is the country in which the author’s home institution is based, since countries differ in how academics climb the career ladder. Some countries offer cash incentives for publishing, others disburse public funds to universities based closely on the publication record of universities, and others have tenure-type systems where promotion is more closely tied to publication record. When you sort authors into groups based on the policies of their country, you do find that authors in countries with cash incentives for publication are more likely to retract papers than those working in countries without cash incentives. But that’s the strongest piece of evidence based on national policy that publication incentives lead to worse science. You don’t observe any statistically significant difference between authors in these cash incentive countries when you look at publication bias. Neither do you see anything when you instead put authors into groups based on whether they work in a country where promotion is more closely tied to individual performance. And if you group authors based on whether they work in a country where publication record plays a large role in how funds are distributed, you actually see the opposite result than expected (authors are less likely to retract and show less signs of publication bias, when publication records matter more for how funds are disbursed). A final piece of suggestive evidence is also interesting. In Smaldino and McElreath, the underlying rationale for engaging in “sloppy science” is to accrue more publications. But in fact, authors who publish more papers per year were less likely to retract and their papers either exhibited less bias or no statistically different amount (depending on whether the first or last author is assigned to a multi-authored paper). There’s certainly room for a lot of interpretations there, but all else equal that’s not the kind of thing we would predict if we thought sloppy science let you accrue more publications quickly. Read the whole thing for my view on how all this literature fits together. But the short version is I think publish-or-perish, on average, probably introduces real distortions, but they aren’t enormous. Read the Whole Thing Measuring the Impact of Strange Combinations of Ideas A classic school of thought in innovation asserts that the process of innovation is fundamentally a process of combining pre-existing concepts in new and novel ways. One claim from this school of thought is that innovations that make particularly surprising combinations should be particularly important in the history of innovation. The article The Best New Ideas Combine Disparate Old Ideas looked at a bunch of evidence consistent with this claim, at least in the context of patents and papers. I’ve updated this article with two papers that provide new ways to measure this, in the context of academic papers. The first is by Carayol, Lahatte, and Llopis (2019): Carayol, Lahatte, and Llopis (2019) investigate this by using the keywords that authors attach to their own manuscripts as proxies for the ideas that are being combined. For a dataset of about 10 million papers published between 1999 and 2013, they look at each pair of keywords used in each paper, comparing how many other papers use the same pair of keywords as compared to what would be expected if keywords were just assigned randomly and independently. Using this metric of novelty, they find the more novel the paper, the more citations it gets and the more likely it is to be among the top 5% most cited. In the figures below, papers are sorted into 100 bins from least novel (left) to most novel (right), and the average citations received within 3 years or the probability of being among the top 5% most cited papers for papers in the same centile is on the vertical axis. From Carayol, Lahatte, and Llopis (2019) The second paper brings in a new way to measure the impact of unusual combinations, rather than a new way of measuring how ideas are combined or not combined. [A]s with patents, it would be nice to have an alternative to the number of citations received as a measure of how important are academic papers that combine disparate ideas. Lin, Evans, and Wu (2022) provide one such alternative by comparing how disruptive a paper is and how unusual are the combinations of cited references. Intuitively, disruption is about how much your contribution renders prior work obsolete, and a new line of papers attempt to measure this with an an index based on how much your work is cited on it’s own, and not in conjunction with the stuff your work cited. This is distinct from simply the number of citations a paper receives. You can be highly cited, but not highly disruptive if you get a lot of citations, but most of them also point to one of your references. And you can also be highly disruptive without being highly cited, if most of the citations you do receive cite you and only you. Lin, Evans, and Wu (2022) measure unusual combinations of ideas in the same way as Uzzi, Mukherjee, Stringer, and Jones and (among other things) compare the extent to which a paper makes unusual combinations to how disruptive it is. They find papers citing conventional combinations of journals are disruptive 36% of the time, whereas papers citing highly atypical combinations of journals are disruptive 61% of the time. In this context, a paper is disruptive if it receives more citations from papers that only cite it than citations from papers that cite both it and one of its references. That suggests unusual combinations are particularly important for forming new platforms upon which subsequent papers build. Read the Whole Thing A Bias Against Novelty Lastly, the article Conservatism in science examined a bit of a puzzle: scientists are curious people, so why would they appear to exhibit a bias against novel research? One strand in that argument was a paper by Wang, Veugelers, and Stephan, which presented evidence that papers doing highly novel work eventually get more citations, but are less likely to be highly cited by people in their own discipline, and take longer to receive citations. But that paper was inevitably based on just one sample of data using one particular measure of novelty. Carayol, Lahatte, and Llopis (2019) (discussed previously) provides an alternative dataset and measure of novelty that we can use to assess these claims. In the updated piece, I integrate their results with Wang, Veugelers, and Stephan. …Suppose we’ve recently published an article on an unusual new idea. How is it recei...

Like the rest of New Things Under the Sun, this article will be updated as the state of the academic literature evolves; you can read the latest version here. Audio versions of this and other posts: Substack, Apple, Spotify, Google, Amazon, Stitcher. One of the most famous recent papers in the economics of innovation is “Are Ideas Getting Harder to Find?” by Bloom, Jones, Van Reenen, and Webb. It showed that more and more R&D effort is necessary to sustain the present rates of technological progress, whether we are talking about Moore’s law, agricultural crop yields, healthcare, or other proxies for progress. Other papers that look into this issue have found similar results. While it is ambiguous whether the rate of technological progress is actually slowing down, it certainly seems to be getting harder and harder to keep up the pace. What about in science? A basket of indicators all seem to document a trend similar to what we see with technology. Even as the number of scientists and publications rises substantially, we do not appear to be seeing a concomitant rise in new discoveries that supplant older ones. Science is getting harder. Before diving into these indicators, I want to head off one potential misunderstanding. My claim is that science is getting harder, in some sense, not that science is ending or that we are on the verge of running out of ideas. Instead, the claim is that discoveries of a given “size” are harder to bring about than in the past. Share Raw paper output We’ll actually start with an indicator that shows no evidence of a slowdown though. Since scientists primarily communicate their discoveries via papers, the first place to look for evidence of increasing difficulty of making discoveries is in the number of papers scientists publish annually. The figure below, drawn from Dashun Wang and Albert-László Barabási’s (free!) book on the Science of Science compares publications to authors over the last century. From Wang and Barabási (2021) At left, we can see the number of papers and authors per year has increased basically in lockstep over the twentieth century. Note, the axis is a log-scale, so that a straight-line indicates exponential growth. Meanwhile, at right, the blue dashed line shows that the number of papers per author has hovered around 2 for a century and rather than falling, it is actually on the rise in recent decades. (As an aside, the solid red line at right is strong evidence for the rise of teams in science, discussed more here) So absolutely no evidence that scientists are struggling to find stuff worth writing up. But that’s not definitive evidence, because scientists are strongly incentivized to publish and what constitutes a publishable discovery is whatever editors and peer reviewers think is publishable. If fewer big discoveries are made, scientists may just publish more papers on small discoveries. So let’s take a more critical look at the papers that get published and see if there are any indicators that they contain smaller discoveries than in the past. Nobel Prizes Let’s start by looking at some discoveries whose importance is universally acknowledged. The Nobel prize for discoveries in physics, chemistry, and medicine is one of the most prestigious scientific prizes and has a history long enough for us to see any long-run trends. Using a publicly available database on Nobel laureates by Li et al. (2019), we can identify the papers describing research that is eventually awarded a Nobel prize, and the year these papers were published. Note several papers might be associated with any given award. For each award year, we can then ask, what share of the papers related to the discovery were published in the preceding twenty years. The results of that are presented below, though I smooth the data by taking the ten-year moving average. Share of papers describing Nobel-prize winning work, published in the preceding 20 years. 10-year moving average.Author calculations, based on data from Li et al. (2019). Prior to the 1970s, on average 90% of the time, awards went to papers published in the last twenty years. But by 2015, the ten-year moving average was closer to 50%. So recent discoveries seem to have a harder time getting recognized as Nobel-worthy, relative to a few decades ago. We can also compare the importance of different discoveries that won Nobel prizes. In 2018, Patrick Collison and Michael Nielsen asked physicists, chemists, and life scientists to pick the more important discovery (in their field) from sets of two Nobel prize winning discoveries. For example, they might ask a physicist to say which is more important, the discovery of Giant Magnetoresistance (awarded the Nobel in 2007) or the discovery of the Compton effect (awarded in 1927). For each decade, they look at the probability a randomly selected discovery made in that decade would be picked by their survey respondents over a randomly selected discovery made in another decade. The results are below:1 Probability a discovery in a given decade is rated more important than discovery in another decadeFrom Collison and Nielsen (2018) A few points are notable from this exercise. First, physicists seem to think the quantum revolution of the 1910s-1930s was the best era for physics and it’s been broadly downhill since then. That’s certainly consistent with discoveries today being in a sense smaller than the ones of the past, at least for physics. In contrast, for chemistry and physiology/medicine, the second half of the twentieth century has outperformed the first half. In the Nobel prize data, within the second half of the century, there is no obvious trend up or down for chemistry and medicine. While that’s better than physics, it remains consistent with the notion that science might be getting harder. As we can see in the first figure here, the number of papers and scientists rose substantially between 1950 and 1980, which naively implies that the number of candidates for Nobel-prize winning discoveries should also have risen substantially. If we are selecting the most important discovery from a bigger pool of candidates, we should expect that discovery to be judged more important than discoveries picked from smaller pools. But that doesn’t seem to be the case. So Nobel prize data is also consistent with the idea that discoveries today aren’t what they used to be. Whereas it used to be quite common for work published in the preceding twenty years to be recognized for a Nobel, that doesn’t happen nearly so much today. That said, an alternative explanation is that the Nobel committee is just trying to work through an enormous backlog of Nobel-worthy work which they want to recognize before the discoverers die. In this explanation, we’ll eventually see just as many awards for the work of today. But it’s not clear to me this is how the committee is actually thinking: recent work is awarded half the time still if the committee thinks the discovery is sufficiently important. For example, Jennifer Doudna and Emmanuelle Charpentier were awarded a Nobel for their work on CRISP-R in 2020, less than a decade after the main discoveries. And when you look specifically at the work performed in the 1980s, it doesn’t seem particularly notable, relative to work in the 40s, 50s, 60s, and 70s, despite the fact that many more papers were published in that decade. Top Cited Papers Still, perhaps the Nobel prize is simply too idiosyncratic for us to learn much from. Next, let’s look at another indicator of big discoveries, one which shouldn’t be biased by the sort of factors peculiar to the Nobel. This is the most top-cited papers in a given field. For example, if we look at the top 0.1% most highly cited papers of all time in a particular field, we could ask how easy is it for a new paper to join their ranks. If that has fallen over time, then that’s further evidence that today’s papers aren’t making the same contributions as yesterday’s. On the other hand though, we might think it should get harder and harder to climb to the top 0.1%, even if discoveries are not getting smaller. After all, if discoveries are of constant size, earlier works have more time to get citations; it may not be possible for later papers to catch up, even if they are just as good. But there are also some factors that lean in the opposite direction. First, if work is only cited when relevant, then newer work should have an easier time being relevant to newer papers. Since the number of new papers grows over time, that gives one advantage to the new; they can be tailored to a bigger audience, in some sense. Second, the most esteemed papers of all time may actually stop being cited at high rates, because their contributions become part of common knowledge: it is no longer necessary to cite Newton when talking about gravity, or even Watson and Crick when asserting DNA has a double-helix shape. So let’s proceed with seeing if there has been any change in how easy or hard it is to become a top cited paper, noting that won’t be the last piece of evidence we look at. The closest paper I know of that looks into this is Chu and Evans (2021), which looks at the probability of a new paper ever becoming one of the top 0.1% most cited, even for just one year. But this paper does not plot this probability against time, like the previous charts: instead, it plots this probability against the size of a field, measured by the number of papers published per year. In the scatterplot below, each point corresponds to a field in a year. On the horizontal axis is the number of papers published in the fields in that year and on the vertical axis the probability a paper in that field and year is ever among the top 0.1% most cited. The colored lines are trends for each of these ten fields. Note this figure only includes papers published in the year 2000 or earlier. Since the analysis is conducted with data from 2014, every paper has more than a deca...

This is an audio read-through of the initial version of Science is getting harder. Like the rest of New Things Under the Sun, the underlying article upon which this audio recording is based will be updated as the state of the academic literature evolves; you can read the latest version here.

Dear Reader, Change is afoot! Since December 2020, I have been splitting my time between writing New Things Under the Sun and teaching economics at Iowa State University. I loved teaching and Iowa State has been fantastic. But, to use some economist lingo, my comparative advantage is in writing New Things Under the Sun and I have believed for awhile that the project could have a bigger impact if I were able to specialize completely in it. Accordingly, this is my last day at Iowa State University. Beginning May 22, I will be joining the Institute for Progress (IFP) as Senior Innovation Economist, where my job will be to work full time on New Things Under the Sun and related projects. You may recall the Institute for Progress has been New Things Under the Sun’s partner since January - they are a new non-partisan think tank with a mission to accelerate scientific, technological, and industrial progress. This new arrangement is possible thanks to them and grant support from OpenPhilanthropy. While I am excited to officially be part of IFP, I will continue to work remotely from Iowa and retain sole editorial control over New Things Under the Sun. I continue to believe IFP is doing great stuff, and being affiliated with an organization that is trying to effect actual change is a good influence on me (and I hope I can be a good influence on them!). Among other things, working with IFP provides a constant nudge to think about how academic work sheds light on questions that matter. So what does it look like to specialize in this synthesizer/communicator role I’ve carved out? I guess we’ll find out! But here is my preliminary sketch. First, the most obvious requirement of this job is knowing the academic work well. Over the last years, despite my best efforts, my to-read list has only gotten longer. So I’m going to read more. Second, part of the job is seeing connections between ideas. This is especially important as one of the things that makes New Things Under the Sun unique is I try to keep articles up-to-date with the academic frontier. That means I can’t write articles and then forget about their content. To keep what I write and read perpetually accessible, I am going to try is to build up a spaced repetition memory system.1 Third, I plan to write more. Well, actually, I plan to at least meet the goal I set for myself in January, after partnering up with IFP, to write three articles per month. I’m a bit embarrassed to say I’ve only hit this goal once, in February. Partly that’s due to covid finally catching up to me and then my kids during April, but it’s also because so far my time has been split and sometimes other deadlines assert priority. I don’t think New Things Under the Sun needs to be a really frequent publication - every original piece is designed to be perpetually relevant, with maintenance - but I at least want to get into a rhythm of producing something every ten days or so. Lastly, I’m going to try and meet with more of the producers and “end-users” of academic research. What do people whose work is related to innovation wish they knew? What do academics studying innovation think about their own field? I think it’s obvious my work would benefit from more of this kind of tacit knowledge. I’ve already had a few conversations like this with readers and academics. But I thought it might be helpful to make this a formal invitation: if you ever want to chat about something innovation related, feel free to drop me an email and we can set up a virtual coffee. I can be reached at mattclancy at hey dot com. Now that I’m working fully remote, hopefully this can be a good substitute for some of that serendipity around the water cooler I’ll be missing. If zoom is not your thing, I also plan to visit Washington DC for a few days every quarter, to work at the IFP offices, and I hope to meet with people in person during those visits. I’m sure most of you are more likely to pass through DC then you are to pass through Des Moines. Beyond that, I have plenty of other ideas for improving New Things Under the Sun, which I will also incrementally work on. But first, I’m taking a break! I’ll be taking off next week. Cheers all and thanks for your interest in New Things Under the Sun. Excited about this next step. Matt 1 This isn’t my first experiment with spaced repetition memory systems. During covid-19, I built an online intermediate microeconomics course using the Orbit platform developed by Andy Matuschak, which implements personalized spaced repetition. Check it out if you’re curious about spaced repetition or if you need to learn calculus-based microeconomics!

New Things Under the Sun is a living literature review; as the state of the academic literature evolves, so do we. This post highlights two recent updates. One of those updates was pretty big, so I will end up copying the entire updated post below, rather than an excerpt. But first, one announcement and one shorter update. Endless Frontier Fellowship First, I wanted to do a quick plug for a new fellowship that’s probably of interest to some readers of this newsletter. It’s a one-year science and tech policy fellowship for talented early career individuals, called the Endless Frontier Fellowship. Fellows spend an immersive year embedded as policy entrepreneurs at EFF’s anchor organizations, the Institute for Progress (New Things Under the Sun’s partner), the Federation of American Scientists, or the Lincoln Network. It’s paid! If you want to apply, the deadline is May 2. More details here. Covid-19 and Innovation Second, the article Medicine and the Limits of Market Driven Innovation has been updated with some discussion of a new paper by Agarwal and Gaule (2022), which describes how the biomedical R&D machine responded to covid-19. It’s a bit hard to excerpt the updates, but two points emphasized are: Agarwal and Gaule provide some additional evidence which confirms work done by other papers using earlier data. Biomedical R&D is responsive to the size of the profit opportunity associated with diseases: they find a 10% increase in the size of the market for a drug is associated with about 4% more clinical trials. Against this benchmark, the response of biomedical R&D to covid-19 was a huge outlier. According to their estimates, the size of the “market” for a covid-19 treatment (based on global mortality from the disease) was bigger than the market for any other disease they considered. Even so the number of new clinical trials was 7-20 times larger than their model would have predicted. Covid-19 was strange in other ways as well. One of the main arguments of Medicine and the Limits of Market Driven Innovation is that private biomedical R&D generally responds to profit opportunity only with projects that do not require much fundamental research. While we have pretty good evidence that this is the case, covid-19 represents a big counter-example. As discussed a bit in the new update, covid-19 did in fact lead to a major shift in the kind of research done throughout science (discussed in more detail here). Data on Combinatorial Innovation Lastly, I’ve written a fairly large update to a post originally called “Innovation as Combination: Data.” That was the fifth New Things Under the Sun I ever wrote, and it wasn’t quite in the style of today’s posts. I now try to make each piece make a specific claim, drawing on a set of related papers, but that piece was more a round up of some related articles. I’ve rewritten it to make a specific claim, which is encapsulated in the new title: “The best new ideas combine disparate old ideas.” It’s about 50% new material, with the set of articles covered going from 4 to 7. Rather than excerpt so much, I reproduce the whole updated post below; enjoy! The Best New Ideas Combine Disparate Old Ideas Where do new ideas and technologies come from? One school of thought says they are born from novel combinations of pre-existing ideas. To some extent that’s true by assumption, since everything can be decomposed into a collection of parts. But this school of thought makes stronger claims. One such claim is that new combinations - those pulling together disparate ideas - should be particularly important in the history of ideas. And it turns out we have some pretty good evidence of that, at least from the realms of patents and academic papers (and also computer programming). To get at the notion that new ideas are combinations of older ideas, these papers all need some kind of proxy for the pre-existing ideas that are out there, waiting to be stitched together. They all ultimately rely on classification systems that either put papers in different journals, or assign patents to different technology categories. These journals or technology classifications are then used as stand-ins for different ideas that can be combined. A paper that cites articles from a monetary policy journal and an international trade journal would be assumed to be combining ideas from these disciplines then. Or a patent classified as both a “rocket” and “monorail” technology would be assumed to combine both ideas into a new package technology. New Combinations in Patents and Citations A classic paper here is Fleming (2001), which uses highly specific patent subclasses to proxy for combining technologies. There were more than 100,000 technology subclasses at the time of the paper’s analysis, each corresponding to a relatively narrow technological concept. Using a sample of ~17,000 patents granted in May and June 1990 Fleming calculates the number of prior patents assigned the exact same set of subclasses. He shows patents assigned combinations without much precedent tend to receive more citations, which suggests patents that combined rarely combined concepts were indeed more important. For example, as we go from a patent assigned a completely original set of subclasses to a patent with the maximum number of prior patents assigned the same set of subclasses, citations fall off by 62%. This flavor of result holds up pretty well to a variety of differing methods. For example, Arts and Veugelers (2015) track new combinations in a slightly different way than Fleming, and use a different slice of the data. Rather than counting the number of prior patents assigned the exact same set of technology classifications, they look at the share of pairs of subclasses assigned to a patent that have never been previously combined. This differs a bit from Fleming because they are only interested in patents that are the first to be assigned two disparate technology subclasses, and also because a patent might be a new combination and still be assigned no new pairs. For example, given subclasses A, B, and C, if the pairs AB, BC, and AC have each been combined before, but the set of all three (ABC) has not, then Fleming will code a patent assigned ABC as highly novel and Arts and Veuglers will not. Arts and Veugelers (2015) look at ~84,000 US biotechnology patents granted between 1976 and 2001 and look at the citations received within the next five years. About 2.2% of patents that forge a new connection between different technology subclasses go on to be one of the most highly cited biomedical patents of all time, compared to just 0.9% of patents that fail to forge new connections. And patents that don’t become these breakthroughs still get more citations if they forge novel links between technology subclasses. Moreover, the direction of this relationship is robust to lots of additional control variables. As a final example, He and Luo (2017) also establish this result, measuring novel combinations in yet another way, and using an even broader set of data. He and Luo look at ~600,000 US patents granted in the 1990s, and which contain 5 or more citations to other patents. Rather than relying on the technology classifications assigned directly to these patents, they look at the classifications assigned to cited references. They assume a patent combines ideas from the classifications of the patents it cites. They also use a much coarser technology classification system, which has just 630 different technology categories, rather than over 100,000 used in the previous two papers. To measure novel combinations, they look at how frequently a pair of technology classifications are cited together relative to what would be expected by chance. That means they end up with lots of measures of novelty for each patent, one for every possible pair of cited references. To collapse down the set of novelty measures for each patent, they order the pairs of cited reference from the least conventional to most and then grab the median and the 5th percentile. As a measure of the importance of these patents, we can look at the probability that they are a highly cited patent for the year they were granted and for their technology class. In the figure below, they divide patents up into deciles and compute the probability a patent whose novelty measure falls into that decile is a hit patent. Because they are adapting some earlier work, they set these indices up in a kind of confusing way. In the left figure below, moving from left to right we get increasingly conventional patents, while in the right figure, moving from left to right we get increasingly more unconventional patents. From He and Luo (2017) The figure above shows that when you focus on the most unusual combination of cited technologies made by a patent (the right figure), then more atypical patents have a significantly higher chance of being a hit patent. When you focus on the median, you find a more complicated relationship: you don’t want all the combinations made to be totally conventional nor totally unconventional and strange. There’s a sweet spot in the middle. Perhaps patents that are completely stuffed with weird combinations are too weird for future inventors to understand and build on? Addressing some potential problems The link between unusual combinations of technology classifications and future citations received is pretty reliable across these papers. But before taking these results too far, there are a few potential issues we need to look into. The first potential issue is a form of selection bias. One challenge from this literature is we typically only ever look at patents that are ultimately granted. But suppose patent examiners are biased against patent applications that make unusual combinations. If that’s the case, then patents making unusual combinations will only make it through if they are so valuable that their merits overcome this deficit. That would, in t...

Like the rest of New Things Under the Sun, this article will be updated as the state of the academic literature evolves; you can read the latest version here. Audio versions of this and other posts: Substack, Apple, Spotify, Google, Amazon, Stitcher. We all know the proverb “Necessity is the mother of invention.” This proverb is overly simplistic, but it gets at something true. One place you can see this really clearly is in global crises, which vividly illustrate the linkage between need and innovation, without the need for any fancy statistical techniques. Let’s look at three examples. Share Crisis #1: Covid-19 Global Pandemic Our first crisis is the one we’re all most familiar with: the covid-19 global pandemic. During 2020-2022, the big thing we suddenly needed was medical treatment for covid-19. Agarwal and Gaule (2022) look at what happened to the number of new clinical trials (for all diseases) in the wake of the pandemic.1 No surprises: the number of new clinical trials shot up as the magnitude of the disease became clear, with essentially all of the increase coming from trials related to covid-19. From Agarwal and Gaule (2022) In the end, these trials succeeded and we got a suite of effective vaccines in record time: necessity was the mother of invention. Covid-19 had other effects too. For one, it forced the world to embark on an unprecedented experiment in remote work. Bloom et al. (2021) is a short paper that looks at the share of patent applications, filed in the USA, that relate to remote work. Bloom and coauthors scan the text of patent applications for words related to remote work, such as “work remotely”, “telework”, “video chat”, and many others. As we can see in the figure below, covid-19 induced a step change in the share of patents related to working remotely. Again, necessity was the mother of invention. Update to Bloom et al. (2021) by Mihai Codreanu Crisis #2: Oil Shocks Our second crisis is the oil price shocks of the 1970s. After a long period of relatively stable and predictable energy prices, the price of oil abruptly shot up due to disruptions to Middle Eastern supply in the 1970s. The energy crisis created an urgent need to pivot away from dependence on suddenly unreliable oil supplies. Suggestive evidence that the US economy managed to do just that comes from the following figure from Hassler, Krusell, and Olovsson (2021). The black line is the share of GDP spent on energy, the dashed line tracks the price of energy in the USA. From Hassler, Krusell, and Olovsson (2021) Around 1985 the link between the share of GDP spent on energy and the price of energy seems to have changed (in the figure, the black line moved from above the dashed one to below). That suggests the economy got better at getting more GDP out of less energy. But it’s still not 100% clear how the timing of this all played out; was this really that closely related to the oil shocks? To more precisely estimate the pace of innovation related to energy, Hassler, Krusell, and Olovsson (2021) use some fairly basic economic modeling. They assume economic output is produced by labor and energy, and that technology comes in two flavors, one for each. If the technology for energy gets twice as good, it’s as if you’ve got twice as much energy to play with (when in fact, better technology allows you to use the energy you’ve got twice as efficiently). Similarly, if labor technology gets twice as good, it’s as if you’ve got twice as much labor to work with. The cool thing is that if you accept their pretty simple model, you end up with a way to measure a concept like “technology”, which is normally so nebulous, with some very simple and readily available data. If you assume the economy uses labor and energy efficiently, you can do some math, move things around, and show that the productivity of the energy technology can be expressed as a function of GDP per capita, the share of spending on energy in the economy, and our ability to substitute labor for energy and vice-versa. That’s almost all stuff we can measure. When Hassler, Kruseell and Olovsson plug data into this equation and make some sensible assumptions about our ability to substitute labor for energy (they assume its quite hard), you get the following striking chart tracking our ability to convert energy into economic output. Here, the blue line is a measure of how technology multiplies the energy supply, so that having one barrel of oil in 2020 is like having 3 in 1950. Estimated productivity of energy. From Hassler, Krusell, and Olovsson (2021). Now it’s crystal clear: the oil shocks knocked productivity of energy technology out of its stagnation and into a steady upward trend. Necessity was the mother of invention. An aside: sometimes, people argue one reason technological progress slowed in the 1970s, because we moved from technological progress that took abundant energy for granted to technological progress that did not. Hassler, Krusell, and Olovsson’s work is broadly supportive of that narrative. This is just three data points, so don’t get too excited, but there does seem to be a negative correlation between the pace of progress in technology that converts energy into output and technology that converts labor into output. In other words, when the oil shocks forced us to expend more effort on reducing demand on fossil fuels, that may have come at the expense of other forms of technological progress that we had become accustomed to. From Hassler, Krusell, and Olovsson (2021) Crisis #3: World War II Our last crisis is World War II. We could point to many innovations born out of the exigencies of World War II: radar to defend against attack from the air; penicillin produced at industrial scale; and the Manhattan project to develop the first atomic bomb. But let’s focus on the need to build a lot of airplanes. When President Roosevelt targeted 50,000 planes over the war in 1940, this goal was viewed as simply impossible by many: contemporary economists Robert Nathan and Simon Kuznetz believed the US simply didn’t have the productive capacity to do it (Ilzetzki 2022). And yet, in reality, the US eventually succeeded in producing 100,000 planes in just one year. During the war, there was a 1,600% increase in the number of aircraft produced, and US spending on aircraft alone reached 10% of 1939 GDP. How did the US manage to do the seemingly impossible? The following figure from Ilzetzki (2022) gives some clues. It shows total US aircraft produced (measured by weight), as well as the capital and labor used to produce aircraft, relative to 1942 levels. From Ilzetski (2022) Initially, the US made more airplanes by using more labor and more capital to make airplanes. But after 1943, something surprising happened: the increase in capital and labor slowed or even stopped, but we kept on increasing how many planes we made! In order to meet their ambitious targets, airplane manufacturers were forced to discover new efficiencies. And they did! Necessity was the mother of invention. Ilzetski actually goes much further, and tracks the productivity of individual airplane manufacturers. He shows that, on average, individual manufacturers became more productive when they received more plane orders, and that this effect was greatest for the manufacturers who were already operating closest to capacity. In other words, the manufacturers who had the least ability to meet their aircraft orders by increasing labor or capital were also the ones who most improved their productivity! Invention Has Two Parents The above examples illustrate how sudden new necessities can indeed drive innovative effort. And I’ve written elsewhere about evidence that demand for new technologies, even in non-crisis settings, can also spur innovative effort. For example, the private sector tends to do more R&D on treatments for diseases that become more profitable to treat, and automobile manufacturers developed more fuel efficient vehicles in response to fuel efficiency standards and high energy prices. But we need to be careful not to take this too far. You cannot will technologies into being, simply because someone needs them (if so, we wouldn’t have waited so long for mRNA vaccines and atomic bombs). Invention has two parents. A truer proverb might be “Necessity and knowledge are the parents of invention.” We can also see this in some of the examples just cited. As discussed in a bit more detail here, most of the new clinical trials for covid-19 were not for fundamentally new kinds of drugs. Instead, they were largely attempts to re-deploy existing drugs to a novel use case. In other words, they were attempts to take what was already known to be safe and see if it had beneficial effects on covid-19. Most of these failed. The covid-19 vaccines that eventually succeeded rested on deep foundations of fundamental research that went back decades. Covid-19 was the impetus to transform this knowledge into effective new treatments (though these efforts were already underway before covid-19), but it didn’t give us the knowledge that made that possible. Most of the radical technologies developed during World War II, such as radar and the atomic bomb, relied on breakthroughs in fundamental science that preceded the war. In a 2020 review of the activities of the US Office of Scientific Research and Development, which oversaw these and many other technological breakthroughs of the war, Gross and Sampat write “the time for basic research is before a crisis, and since time was of the essence, ‘the basic knowledge at hand had to be turned to good account.’” Ilzetski shows much of the improvement in airplane manufacturing came from adopting techniques that had been shown to be effective in other sectors, rather than inventing new processes out of whole cloth. Specifically, airplane manufacturers that faced capacity constraints were more likely to adopt production line processes (instead o...

This is an audio read-through of the initial version of When Extreme Necessity is the Mother of Invention. To read the initial newsletter text version of this piece, click here. Like the rest of New Things Under the Sun, this underlying article upon which this audio recording is based will be updated as the state of the academic literature evolves; you can read the latest version here.

Like the rest of New Things Under the Sun, this article will be updated as the state of the academic literature evolves; you can read the latest version here. Audio versions of this and other posts: Substack, Apple, Spotify, Google, Amazon, Stitcher. Finally, as part of the partnership with the Institute for Progress, the fine folks at And Now have designed a new logo for New Things Under the Sun: New scientific research topics can sometimes face a chicken-and-egg problem. Professional success requires a critical mass of scholars to be active in a field, so that they can serve as open-minded peer reviewers and can validate (or at least cite!) new discoveries. Without that critical mass,1 working on a new topic topic might be professionally risky. But if everyone thinks this way, then how do new research topics emerge? After all, there is usually no shortage of interesting new things to work on; how do groups of people pick which one to focus on? One way is via coordinating mechanisms; a small number of universally recognized markers of promising research topics. The key ideas are that these markers are: Credible, so that seeing one is taken as a genuine signal that a research topic is promising Scarce, so that they do not divide a research community among too many different topics Public, so that everyone knows that everyone knows about the markers Prizes, honors, and other forms of recognition can play this role (in addition to other roles). Prestigious prizes and honors tend to be prestigious precisely because the research community agrees that they are bestowed on deserving researchers. They also tend to be comparatively rare, and followed by much of the profession. So they satisfy all the conditions. This isn’t the only goal of prizes and honors in science. But let’s look at some evidence about how well prizes and other honors work at helping steer researchers towards specific research topics. Share Howard Hughes Medical Institute Investigators We can start with two papers by Pierre Azoulay, Toby Stuart, and various co-authors. Each paper looks at the broader impacts of being named a Howard Hughes Medical Institute (HHMI) investigator, a major honor for a mid-career life scientist that comes bundled with several years of relatively no-strings-attached funding. While the award is given to provide resources to talented researchers, it is also a tacit endorsement of their research topics and could be read by others in the field as a sign that further research along that line is worthwhile. We can then see if the topics elevated in this manner go on to receive more research attention by seeing if they start to receive more citations. In each paper, Azoulay, Stuart, and coauthors focus on the fates of papers published before the HHMI investigatorship has been awarded. That’s because papers written after the appointment might get higher citations for reasons unconnected to the coordinating role of public honors: it could be, for instance, that the increased funding resulted in higher quality papers which resulted in more citations, or that increased prestige allowed the investigator to recruit more talented postdocs, which resulted in higher quality papers and more citations. By restricting our attention to pre-award papers, we don’t have to worry about all that. Among pre-award papers, there are two categories of paper: those written by the (future) HHMI investigator themselves, and those written by their peers working on the same research topic. Azoulay, Stuart, and coauthors look at each separately. Azoulay, Stuart, and Wang (2014) looks at the fate of papers written by an HHMI investigator before their appointment. The idea is to compare papers that of roughly equal quality, but where in one case the author of the paper gets an HHMI investigatorship and in the other case doesn’t. For each pre-award paper by an HHMI winner, they match it with a set of “control” papers of comparable quality. These controls are published in the same year, in the same journal, with the same number of authors, and the same number of citations at the point when the HHMI investigatorship is awarded. Most importantly, the control paper is also written by a talented life scientist, with the same position (for example, first author or last author, which matters in the life sciences), but who did not win an HHMI investigator position. Instead, this life scientist won an early career prize. If people decide what to work on and what to cite simply by reading the literature and evaluating its merits, then whatever happens to the author after the article is published shouldn’t be relevant. But that’s not the case. The figure below shows the extra citations, per year, for the articles of future HHMI investigators, relative to their controls who weren’t so lucky. We can see there is no real difference in the ten years leading up to the award, but then after the award a small but persistent nudge up for the articles written by HHMI winners. From Azoulay, Stuart, and Wang (2014) That bump could arise for a number of different reasons. We’ll dig into what exactly is going on in a minute. But one possibility is that the HHMI award steered more people to work on topics similar enough to the HHMI winner that it was appropriate to cite their work. A simple way to test this hypothesis is to see if other papers in the same topic also enjoy a citation bump after the topic is “endorsed” by the HHMI, even though the author of these articles didn’t get an HHMI appointment themselves. But that’s not what happens! Reschke, Azoulay, and Stuart (2018) looks into the fate of articles written by HHMI losers2 on the same topic as HHMI winners. For each article authored by a future HHMI winner, Reschke, Azoulay, and Stuart use the PubMed Related Articles algorithm to identify articles that are on similar topics. They then compare the citation trajectory of these articles on HHMI-endorsed topics to control articles that belong to a different topic, but were published in the same journal issue. As the figure below shows, in the five years prior to the award, these articles (published in the same journal issue) have the same citation trajectories. But after the HHMI decides someone else’s research on the topic merits an HHMI investigatorship, papers on the same topic fare worse than papers on different topics! From Reschke, Azoulay, and Stuart (2018) Given the contrasting results, it’s hard not to think that the HHMI award has resulted in a redistribution of scientific credit to the HHMI investigator and away from peers working on the same topic. So maybe awards don’t actually redirect research effort. Maybe they just shift who gets credit for ideas? The truth seems to be that it’s a bit of both. To see if both things are going one, we can try to identify cases where the coordination effect of prizes might be expected to be strong, and compare those to cases where we might expect it to be weak. For example, for research topics where there is already a positive consensus on the merit of the topic, prizes might not do much to induce new researchers to enter the field. Everyone already knew the field was good and it may already be crowded by the time HHMI gives an award. In that case, the main impact of a prize might be to give a winner a greater share of the credit in “birthing” the topic. In contrast, for research topics that have been hitherto overlooked, the coordinating effect of a prize should be stronger. In these cases, a prize may prompt outsiders to take a second look at the field, or novice researchers might decide to work on that topic because they think it has a promising future. It’s possible these positive effects are enough so that everyone working on these hitherto overlooked topics benefits, not just the HHMI winner. Azoulay, Stuart, and coauthors get at this in a few different ways. First, among HHMI winners, the citation premium their earlier work receives is strongest precisely for the work where we would expect the coordinating role of prizes to be more important. It turns out most of the citation premium accrues to more recent work (published the year before getting the HHMI appointment), or more novel work, where novelty is defined as being assigned relatively new biomedical keywords, or relatively unusual combinations of existing ones. HHMI winners also get more citations (after their appointment) for work published in less high-impact journals, or if they are themselves relatively less cited overall at the time of their appointment. And these effects appear to benefit HHMI losers too. The following two figures plot the citation impact of someone elsegetting an HHMI appointment for work on the same topic. But these figures estimate the effect separately for many different categories of topic. In the left figure below, topics are sorted into ten different categories, based on the number of citation that have collectively been received by papers published in the topic. At left, we have the topics that collectively received the fewest citations, at right the ones that received the most (up until the HHMI appointment). In the right figure below, topics are instead sorted into ten different categories based on the impact factor of the typical journal where the topic is published. At left, topics typically published in journals with a low impact factor (meaning the articles of these journals usually get fewer citations), at right the ones typically published in journals with high impact factors. From Reschke, Azoulay, and Stuart (2018) The effect of the HHMI award on other people working on the same topic varies substantially across these categories. For topics that have not been well cited at the time of the HHMI appointment, or which do not typically publish well, the impact of the HHMI appointment is actually positive! That is, if you are working on a topic that isn’t getting cited and isn’t placing in good journals,...

This is an audio read-through of the initial version of Steering Science with Prizes. To read the initial newsletter text version of this piece, click here. Like the rest of New Things Under the Sun, this underlying article upon which this audio recording is based will be updated as the state of the academic literature evolves; you can read the latest version here.

Like the rest of New Things Under the Sun, this article will be updated as the state of the academic literature evolves; you can read the latest version here. Audio versions of this and other posts: Substack, Apple, Spotify, Google, Amazon, Stitcher. Evolution via natural selection is a really good explanation for how we gradually got successively more complex biological organisms. Perhaps unsurprisingly, there have long been efforts to apply the same general mechanism to the development of ever more complex technologies. One domain where this has been studied a bit is in computer programming. Let’s take a look at that literature to see how well the framework of biological evolution maps to (one form of) technological progress. Share Simulating Technological Evolution We’ll start with Arthur and Polak (2006), who look at how ever more sophisticated logic circuits can, in principle, evolve via a blind process process of mutation, selection, and recombination. The paper reports the results of a large number of digital simulations that do precisely that. These simulations have three main components. First if you’re going to simulate evolution, you need your organism, or in this case, your technology. Arthur and Polak start with a very elementary logic gate, in most simulations a Not-And (NAND) gate. This is a circuit with two binary inputs and one binary output. If every input is 1, then it spits out 0; otherwise it spits out 1. From this seed, much more sophisticated circuits will digitally evolve. From Arthur and Polak (2006) The second thing you need in order to simulate evolution is a way to modify the organism or technology. We might think the natural way to do this is to allow for slight mutations in these circuits, which is how we often think of biological evolution (single base-pairs being switched from one to another). But Arthur and Polak believe recombination is the essence of technological change, rather than mutation. So their model of digital evolution is much more explicitly combinatorial. In every period, sets of 2-12 technologies are picked and randomly wired together in sequence, though any individual circuit is also allowed to mutate a bit on its own. Third, to model evolution you need a way to evaluate the fitness of your organisms, or circuits in this case. If we’re trying to understand technological evolution, then fitness should be related to whether or not humans find technologies to be useful. Arthur and Polak come up with a list of desired functions it is reasonable for people to want circuits to fulfill. These range from very simple to very complex. For example, one simple function is just a NOT gate: it just returns the opposite of its input (1 for 0, 0 for 1). A more complex function is a 15-bit adder: if you put in two 15-bit numbers, it outputs their sum. Arthur and Polak next come up with a way to score circuits based on how close they get to giving the right answer: every time the circuit gives the right answer for a set of inputs, it scores better, every time it gives the wrong answer, it scores worse. And if two circuits perform equally well, the one that does it with fewer components scores better. In every period, the highest scoring circuits and their components gets retained. Next period, the simulation draws components from this basket of retained circuits and wires them together to see if any of the resulting combinations do a better job fulfilling the desired tasks. Finally, Arthur and Polak let this system run for 250,000 periods, 20 different times, and watch what happens. We learn a few things from the results of this exercise. First, the experiment is an existence proof that you don’t need inventors with reasoning minds to get sophisticated technologies; this blind recombinant evolution can also do the job. In 250,000 period these simulations don’t discover everything Arthur and Polak define as desirable, but it does go well beyond the simplest circuits. For example, the simulation successfully discovered circuits that can add 4-bit numbers and and circuits that can indicate if one (and only one) of 8 inputs is 1.  Second, in the experiment, technological advance tends to be lumpy. Desirable circuits tend to be discovered in clusters, after key component pieces are discovered which unlock lots of new functionality. But in between these sprints can be long periods of technological stagnation, even as under the surface the ferment of experimentation and “R&D” is going on invisible to us. Third, their simulations give a nuanced picture about the importance of path dependency. This is the idea that where our technologies start has a big impact on where they finish. If we start along one technological trajectory, we’re more likely to continue on it, and end up with a completely different basket of technologies, than if we started elsewhere. In Arthur and Polak’s experiments, one way they can investigate this is to see how different simulations evolve, when different circuits are discovered first. For example, most of the time, a “not” circuit is found before an “imply” circuit. But not always. In the rarer cases when “imply” circuits are found first, many subsequent technologies build on the imply circuit than the “not” circuit. Over time, however, the program still sniffs out the best overall approaches for different functions, and this begins to chip away at the initially atypical dominance of “imply” components. The importance of where you start matters for a time, but then begins to fade. Fourth, technological innovation, like biological innovation, is red in tooth and claw. Better technologies constantly supplant obsolete ones and sometimes this leads to waves of extinction. For example, suppose some technology x is comprised of 12 other circuits, and each of these component circuits is further comprised of 2-12 subcomponents, which are in turn comprised of sub-subcomponents and so on. If technology x is replaced by a superior technology y, then technology x naturally goes “extinct.” And if the components and subcomponents, and sub-subcomponents that comprised x are not part of any other technology that is the highest scoring on some function (and therefore retained), than they too can go extinct, leading to the collapse of an entire ecosystem of supportive circuits. Lastly, Arthur and Polak’s digital experiment illustrates the importance of intermediate goals in the evolution of technological complexity. In their simulations, if Arthur and Polak remove key desirable circuits of intermediate complexity, the simulations get trapped and unable to advance to more complex designs. Evolution needs stepping stones to get from simple to complex. Evolution in MatLab Contests This is an intriguing experiment, but it’s doesn’t demonstrate that these mechanisms are important in the actual development of technology. For that, I am a big fan of two papers from 2018 and 2020 by Elena Miu, Ned Gully, Kevin Laland, and Luke Rendell. These papers study 19 online programming competitions operated by MathWorks over 1998-2012. This is still an artificial setting, but we now have real people solving real programming problems, and as we’ll see, these contests have some important elements that make them worth studying. In these contests, nearly 2000 participants (average of 136 per contest) competed over the course of a week to write programs in MATLAB that could find the best solution to a problem in which it was impossible to find an exact solution in the time given. For example, in a 2007 contest participants wrote code to play a kind of peg-jumping game, where there is a grid of pegs (all worth different points) and an empty space, and you can remove a peg by jumping over one peg and into an empty space. A program’s score was based on three factors: the number of points it got in the game; how fast it ran, and how complex the code is (with more complex code penalized). Participants could submit their programs at any time and receive a score. They could then modify their code in response to the score they received, and this iterative improvement was an important part of the contest. But there is a catch: programs and their scores were publicly viewable by all participants. So submitting a program and getting feedback on its performance also discloses your program to all the other contest participants, who are free to borrow/steal your ideas. This is a great setting to study technological evolution, for a few reasons. As in the real world, there is robust competition, and inventions can be reverse-engineered and copied. Unlike Arthur and Polak, we have reasoning minds designing and improving programs, rather than blind processes of recombination and selection. But perhaps most importantly, for the purposes of studying technological evolution, we can see the complete “genotype” of computer programs by reading their code. And with standard text-analysis packages, Miu and coauthors can see exactly which lines and blocks of code are copied and how similar programs are to each other. Lastly, because programs are explicitly scored (and players care about these scores; they are actively seeking the highest score), Miu and coauthors also know exactly how “good” a program is. The figure below tracks how scores improve for a sample of 4 contests. In the figure, each dot is a program. The horizontal axis is time (each contest runs 7 days) and the vertical is the score (lower is better). Clearly the best programs improve over time, in fits and starts. From Miu et al. (2018) When Miu and coauthors peer into the underlying dynamics, in their 2018 paper they see that the most common type of program that is submitted is a program that is very similar to the current leader, but with minor tweaks. In their 2020 follow-up, they also document that when two programs have the same score, people are more likely to copy the one submitted by the participant who tends to score higher in...

This is an audio read-through of the initial version of Progress in Programming as Evolution. To read the initial newsletter text version of this piece, click here. Like the rest of New Things Under the Sun, this underlying article upon which this audio recording is based will be updated as the state of the academic literature evolves; you can read the latest version here.

New Things Under the Sun is a living literature review; as the state of the academic literature evolves, so do we. Here are a few recent updates. Importing Knowledge in the Age of Mass Migration The article Importing Knowledge looks at what happens when scientists and inventors immigrate. As might be expected, after receiving migrant inventors, a receiving country tends to do better in the technology fields where the migrant inventors have comparative strength. But what’s perhaps less expected is that the impact on the field exceeds the extra output brought by these talented inventors and seems to spill over to domestic inventors. This article has now been updated to include a discussion of a new paper by Diodato, Morrison, and Petralia, which investigates the same question using a new dataset for the period 1870-1950. Diodato, Morrison, and Petralia (2021) looks across the United States over the period 1870-1950 to see what happens when different US cities receive more migrant scientists and inventors. In particular, they want to know what happens to the inventive activities of US-born inventors when foreign inventors move to town. With quite a lot of creative and tedious work, they are able to construct year-by-year, city-by-city, field-by-field, inventor-country-of-origin data for the USA over 1870-1950. They document several facts that are consistent with the above case studies. First, when a city receives more migrant inventors with expertise in a given technology field (as indicated by how the patents of these migrant-inventors are classified), this is associated with increased patenting by US-born residents of the same city. For their second fact, they restrict their attention to cities and years where US-born inventors have no patents in a given technology field. They then show that when a migrant inventor working in that field shows up, the city is more likely to have patents, in that field, by US-born inventors, in subsequent years. Third, analogous to Bahar, Choudhury, and Rapoport (2020) [note: discussed previously in the article], they show all this holds even when you use some statistical techniques to try and tease out just the migrants who moved to various cities for reasons uncorrelated with that city’s technological opportunity. This should strip out cases, for example, where a town gets a new college or national lab that attracts a lot of inventors, from all backgrounds, who work in a particular technology field. Why does this happen? “Importing Knowledge” argues that’s because the inventors bring more than just their own brainpower - they also bring new knowledge and ideas that spreads through local inventor networks. From the same piece, but later: Diodato, Morrison, and Petralia’s study of migration to different US cities in the first half of the twentieth century provide three additional strands of suggestive evidence. First, they show that most of the “oomph” of having migrant inventors comes from having only a small number of them. Stated more precisely, the increase in patents from US-born inventors, in a given technology, that arises when migrant inventors skilled in that technology move to the city is only slightly larger when many migrant inventors move in, as compared to fewer. Second and closely related, they show the impact of migrant scientists moving to your city fades over time. Both are consistent with the notion that it only takes that first “seed” to get a “garden of knowledge” going - though more seeds can help it grow faster. Finally, Diodata, Morrison, and Petralia provide some evidence that migrant inventors may help connect US born inventors with foreign knowledge, even if the migrant inventor doesn’t personally have that knowledge. To illustrate the idea, suppose Nikola is an inventor who emigrates from France to the USA and takes up residence in New York. Let’s suppose Nikola is an active inventor of technologies related to electricity. Meanwhile, suppose France is renowned for its food processing technologies, even though this is not an area in which Nikola is active. Diodato, Morrison, and Petralia show that having Nikola show up in New York increases the patenting of US-born New Yorkers both in the technologies in which he is directly involved (electricity in this example), as well as the technologies he is not directly involved in, but which are associated with his country of origin (food processing, in this example). The first effect is larger and more robust, but both are there. Read the whole thing for a lot more evidence on these points. Read the whole thing Fresh Perspectives in History The article Gender and What Gets Researched argues that one of the factors that affects researcher’s choice of research topic is what they find personally meaningful. This, in turn, can be affected by different people’s life experiences. One simplistic but well documented place you can see evidence of this is in the different research choices of men and women. “Gender and What Gets Researched” looked at some good evidence on differing research choices related to biomedical science, but a new 2022 paper by Risi and coauthors provides some evidence that this isn’t restricted to just that context. Risi et al. (2022) look at the influence of gender on research topics in history by analyzing a sample of 10,000+ articles from major US history journals over 1951-2014. They use natural language processing algorithms to extract from this sample 90 different “topics”, where topics are defined as sets of words that are usually found together. Once topics are assigned to different papers, it becomes possible to tally up the genders of the authors of each paper to see if topics differ in how much they are studied by men and women. As indicated in the table below, there are some considerable differences across topics. From Risi et al. (2022) Over 1951-2014, women substantially outnumber men in the study of not just the “women and gender” topic, but also “family”, “body history”, and even “consumption and consumerism.” As “Gender and What Gets Researched” points out, we have to be careful in how we analyze data like this; it could also be that differences in the topics favored by men and women does not stem from different preferences, but that various barriers prevent women from working on preferred topics. But “Gender and What Gets Researched” also argues that as the share of women in a field rises, the field more broadly begins to reflect their (initially distinctive) concerns. For example: The left figure below tracks the Jensen-Shannon distance between the topics covered by men and women in history articles. This is an index that measures the difference between two statistical distributions; in this case, the distribution of men and women among the 90 different topics that were identified by Risi and coauthors’ natural language processing algorithms. As this index falls, the difference between these distributions is narrowing; knowing someone’s gender is increasingly less useful for predicting what topics they work on. Meanwhile, at right below, we can see the rise of women in the field of history. As with patents, as more women enter the field, the dissimilarity of the topics studied appears to be dropping. From Risi et al. (2022) That said, while this is consistent with the idea that the ideas and perspectives of women have become mainstreamed, it is also possible that the Jensen-Shannon Distance fell merely because women came to study the same subjects as men, not because men began to do research in topics that used to be distinctive to women. However, Rishi et al. show the share of articles mentioning words like “women” or “gender” has grown substantially over the 1951-2014, whether the authors are men or women, and that these terms are less and less confined to a small niche subset. That suggests the gender-difference between topics is falling at least partially because men are taking up the topics that used to be distinctive to women. Read the whole thing for a lot more discussion. Read the whole thing The Adjacent Knowledge of Teammates Adjacent Knowledge is Useful looks at three different setting that let us say something about what kind of knowledge is most useful - knowledge that’s really close, really far, or somewhere in between. One of those setting was an experiment where life scientists attended a symposium where they were divided up into rooms and then talked about research with a random subset of people. Among other things, the experiment monitored which of these randomized conversations resulted in subsequent collaborations. The authors found collaborations were most likely to emerge among life scientists who worked on some overlapping topics, but not a lot. The updated article discusses a subsequent article that largely confirmed that finding, in a broader non-experimental study, again using natural language techniques to extract topics from text. A 2021 paper by Smith, Vacca, Krenz, and McCarty largely confirms Lane and coauthors work in a broader non-experimental context. Smith and coauthors look at what factors are correlated with researchers choosing to begin collaborating with each other during 2015-2018 for a sample of 3400 researchers at the University of Florida (all the faculty they could match to enough data to run their analysis). Specifically, they are interested in seeing whether faculty are more likely to begin collaborating if they work on more similar or dissimilar topics. Doing so requires a measure of how similar is the research expertise of different faculty at the University of Florida. They use a text analysis approach, based on the abstracts of the 14,000+ articles authored by faculty at the university in the three years prior to 2015. Specifically, they use an algorithm to create 5,000 different topics, each of which is defined by a cluster of words that are commonly used together (where more unusual words count for ...

Like the rest of New Things Under the Sun, this article will be updated as the state of the academic literature evolves; you can read the latest version here. Note: An audio version of New Things Under the Sun is now available on all major podcast platforms. Apple, Spotify, Google, Amazon, Stitcher Think of new technologies as proceeding through a set of stages: Basic scientific research that explores phenomena Applied research to better understand how to harness certain phenomena Technology development to capture and orchestrate phenomena for a purpose Marketing and diffusion of the new technology The real world can be more complicated with back-and-forth interplay between the stages, but this is a fine place to start.1 If you want to shape the direction of technology, you can intervene early in this process and try to push the kinds of technology you want onto the market, by subsidizing research.2 Or you can intervene at the end of the process and try to pull the kinds of technology you want into existence by shaping how markets will receive different kinds of technology. One specific context where we have some really nice evidence about the efficacy of pull policies is the automobile market. Making fuel more expensive or just flat out mandating carmakers meet certain emissions standards seems to pretty reliably nudge automakers into developing cleaner and more fuel efficient vehicles. We’ve got two complementary lines of evidence here: patents and measures of progress in fuel economy. In this post, first I’ll go over the evidence, and then I’ll talk a bit what I think we should take away from it. In my view, we have strong evidence that pull policies work well for incremental progress, but the case for their efficacy at promoting radical innovation is a lot shakier. Share Patents, Progress, and Pull Policies One pull policy is a tax on undesirable technologies, since, all else equal, that makes the taxed technology less profitable to develop and alternatives more profitable to develop. A carbon tax is the most famous example of this kind of policy. Aghion et al. (2016) are interested in how a carbon tax might change innovation in the auto sector, but given how rarely anyone actually tries to implement a carbon tax, they can’t directly study the question. Instead, they do the next best thing. From the perspective of a carmaker, one of the main effects of a carbon tax is to raise the price of fuel. So how do carmakers respond to higher fuel prices? To answer that question, they need a way to measure innovation. More specifically, they want to measure the kind of innovation a firm decides to do: do carmakers focus on clean technology (electric, hydrogen fuel cell, and hybrid vehicle) or conventional fossil fuel innovation? Patents are a useful dataset for this kind of problem, since they are correlated with inventive effort and can be easily categorized into different kinds of technology. One problem though is that patents vary tremendously in how valuable they are. Some are very important, but a lot are junk. So Aghion and coauthors focus on the subset of patents for which the patent-holder sought patent protection in three big markets: the USA, the European Union, and Japan. Since it’s costly to apply for a patent in each market, doing so in all these markets is a signal that the inventor thinks the patented invention is sufficiently valuable to be worth protecting in multiple large markets. So in this paper, you can think of them measuring innovation by counting the number of valuable patents for different kinds of automobile technology. In an ideal setting, if we really wanted to assess the impact of fuel prices on the innovation decisions of carmakers, we would want to randomly assign some carmakers to face higher fuel prices than others. Then we could compare the subsequent patenting behavior across groups facing different fuel prices. And if we wanted to establish this robustly, we would want to do this kind of experiment many times. We can’t do that. But Aghion and coauthors do something that gets you closer to this ideal. The price of fuel varies a lot from year-to-year, thanks to fluctuations in the price of oil (see left figure below), but it also varies a lot from country-to-country, because countries vary substantially in the size of their taxes on fuel (see right figure below). From Aghion et al. (2016) Moreover, carmakers typically sell to multiple countries, but have different footprints in different countries. Aghion and coauthors reason that carmakers are more sensitive to (tax-inclusive) fuel prices in the countries where they have a larger share of their total sales. For example, in the figure above it’s clear that the UK raised taxes pretty substantially over the 1990s, while taxes remained flat in the USA. In other words, if we have two carmakers, one with most of its sales in the USA and some in the UK, and another with most of its sales in the UK rather than the USA, then these carmakers are effectively facing different fuel prices. The carmaker selling primarily to the UK sees an increase in effective fuel prices, and we can compare their behavior to the one selling primarily in the USA. For every carmaker, Aghion and coauthors construct an “effective” fuel price that is specific to that carmaker, by weighting the fuel price in each country by the carmaker’s exposure to that country. They estimate this exposure from the share of patents the carmaker seeks protection for in that country over 1965-1985, because generally you don’t bother seeking patent protection in countries where you don’t plan to operate in the future. They then look to see how the patents of carmakers differ in the subsequent 20 years (1985-2005) as each carmaker faces a different effective fuel price. We can then compare the behavior of firms that were established in markets that went on to have higher fuel prices to the behavior of firms that were established in markets that went on to have lower fuel prices. (Note that they estimate the markets where a firm is established over 1965-1985, but look at the effect of fuel prices over 1985-2005; this prevents their results from being driven by innovative fuel efficient companies entering markets when they raise taxes, or from their measure of exposure to different markets from being whipped around by subsequent patenting activity) Aghion and coauthors find fuel prices exert a powerful impact on innovation. A 10% increase in the effective price of fuel (that a specific carmaker is exposed to) is associated with roughly 5% fewer patents of the conventional “fossil fuel” type, and a 10% increase in clean energy patents. Rozendaal and Vollebergh (2021) adapt this strategy to study the impact of emissions standards on auto innovation (in addition to fuel prices). Emissions standards are essentially a requirement that a carmaker’s average CO2 emissions per mile fall below some target by some date. It’s not actually quite that simple; but that’s the gist of the idea. Just as fuel prices differ across time and space, so to do regulatory standards. And just as a carmaker is more likely to care about the fuel prices in markets where it has a lot of sales, so too is a carmaker more likely to care about the emissions standards of markets where they have a lot of sales. Rozendaal and Vollebergh make one of those kinds of observations that is obvious in retrospect but which some previous papers apparently missed. In terms of its impact on the rate and direction of innovation, what matters is not whether a country has an emissions target, or even if this target is high or low. What matters is (1) how high or low is this target relative to today’s average emissions and (2) how long do carmakers have to meet the target? If the standard is high, but you’ve already cleared it, then even though it’s high it imposes no extra incentive to innovate. On the other hand, if the standard is high and you have not met it yet, but actually have a long way to go to meet it, then it matters whether you’ve got ten years or one year to get there. If you don’t account for this kind of thing, it might look like standards don’t have much of an impact on innovation. Rozendaal and Vollebergh construct a measure of standards that takes all this into account: it’s basically the difference between the average emissions of a country and the target, divided by the number of years left to meet the target. When the number is large, it means the average car in the country has a long way to go before it meets the target, and not a lot of time to get there. Here’s how their measure looks for the big three markets. From Rozendaal and Vollebergh (2021) As with Aghion and coauthors, Rozendaal and Vollebergh construct estimates of each carmaker’s exposure to these three major markets, as of the year 2000. Those heavily exposed to Japan, for example, faced stronger incentives to innovate in the 2000s, compared to those heavily exposed to US and EU markets. But after 2010, the situation has largely reversed. Again, Rozendaal and Vollebergh are going to look at valuable patenting of clean and dirty technology in response to these measures of emission standards stringency (in addition to fuel prices). And they find these pull policies work. A 10% increase in the stringency standards is associated with 2% more clean patents. Also importantly, Rozendaal and Vollebergh confirm Aghion and coauthor’s finding that fuel prices matter, albeit the strength of the relationship is weaker than Aghion and coauthors find. This might be because they are looking at a very different time frame, 2000-2016 compared to 1985-2005. What I like about these studies is that they have thousands of different inventive entities, and each of these entities is, at least in principle, facing a “pull policy”of a different strength (based on the mix of markets they operate in). This lets you estimate pretty precisely how effect...

Like the rest of New Things Under the Sun, this article will be updated as the state of the academic literature evolves; you can read the latest version here. Note: An audio version of New Things Under the Sun is now available on all major podcast platforms. Apple, Spotify, Google, Amazon, Stitcher There’s this idea that technology is characterized by path dependency: once you start going down one technology trajectory, you kind of get locked in and it’s hard to switch to another, possibly better trajectory. That can happen for lots of reasons, but one possibility is that it’s something about the nature of knowledge itself. The more you know, the more you can learn: knowledge begets more knowledge. So whichever technology trajectory we start on becomes the one we know the most about, and therefore the one it makes most sense to stick with. One line of evidence about this comes from dynamics of patenting. You know what’s a pretty good predictor of patent activity in the future? Patent activity in the recent past. In this post, I want to see what we can learn from a literature that directly or indirectly looks at this dynamic. But while I think this line of evidence is useful, I want to be up front that it also has significant limitations. Most importantly, in this literature we almost never get anything like an experiment. Instead, we’re reading the tea leaves in observational data. Share Patent Stocks Specifically, we’re going to look at the conceptual category of a “patent stock” (also frequently called a “knowledge stock”). To illustrate just what a patent stock is, let’s start with a practical implementation of the notion in a paper. Aghion et al. (2016) is a paper interested in three kinds of technological progress among automobiles: clean tech (think electric cars, hydrogen fuel cells, and hybrid vehicles), fossil fuel technology (think internal combustion engine), and “grey” tech (think more fuel efficient fossil fuel technology). The paper measures innovation in these different technologies by counting valuable patents.1 There are about 2,500 companies and 1,000 individuals who hold one of these patents, and Aghion and coauthors construct three patent stocks for each of these inventive entities, one for each of these three flavors of technological progress. (Constructing patent stocks is hardly the only thing this paper does, but it’s what I’m focusing on today) Constructing a patent stock basically means adding up all the patents that an inventive entity has taken out in the past, giving more weight to the more recent patents. To be very explicit, suppose Ford Motor Company had a clean technology patent stock of 1,000 last year and obtains 250 new patents for clean technology this year. Then, to construct the patent stock for current year, we take last years’ patent stock, multiply it by 80%, and then add to that the number of new patents. So this year’s patent stock is 0.8 x 1000 + 250 = 1050. Suppose we get 150 patents next year. Then the patent stock next year is 0.8 x 1050 + 150 = 990. The exact value of 80% isn’t that important and people use different numbers, though always less than or equal to 100%. The key idea is it’s telling us, in a single number, something about the prior patent activity of the Ford Motor Company. Note that last year’s patents are worth less than this year’s patents (in this case, 80% as much). And since we apply the 80% discount each year, patents from two years ago are worth even less (80% x 80% = 64%). For the data that Aghion et al. (2016) have, for clean technology, a 10% increase in last year’s clean tech patent stock is associated with a 3% increase in this year’s new clean tech patents (for that firm). For fossil fuel technology, the link is even better: if a firm has a 10% higher patent stock in fossil fuels last year, that’s associated with 5% more new fossil fuel patents this year. This is quite a robust (though not universal) finding. Rozendaal and Vollebergh (2021) look at the same context (clean and dirty innovation in automobiles), but with a more recent slice of data (2000-2016, instead of Aghion and coauthors’ 1986-2005). In their sample, they find an even stronger link: roughly speaking, a 10% increase in last year’s patent stock is associated with a 10% increase in patenting this year. Noailly and Smeets (2015) do something similar for innovation in renewable energy and fossil fuel energy (not automobiles). They also find that firms sought many more clean or fossil fuel patents when last year’s patent stock (of the appropriate type) was higher. You can also go beyond firms and look at whole countries. Looking specifically at US patents, a famous 2002 paper by David Popp uses a slightly different approach to create patent stocks for 11 different technologies related to energy. He finds that a 10% increase in last year’s patent stock for a particular technology was associated with 7% more patents for that technology this year. And Porter and Stern (2000) compute patent stocks for 17 different OECD countries (looking at patents they seek in the USA, to hold consistent the definition of a patent). Again, a 10% increase in a country’s patent stock last year is associated with 8-11% more patents this year. This isn’t a universal finding. Lazkano, Nøstbakken, and Pelli (2017) look at innovation in renewable energy, conventional energy, and storage (battery) technology. Unlike the work discussed above, they typically find the opposite result: a higher patent stock for energy storage technology last year is associated with less energy storage patenting today. Similarly for renewable energy. That said, this paper is the outlier (I’ll return to it briefly later). For now, let’s proceed with the understanding that a positive link between yesterday’s patent stock and today’s patenting is a pretty robust correlation. But before jumping from a correlation to a conclusion, we need to think a bit harder about what’s really going on here. Why is there this correlation? (And if you feel like saying “patents don’t measure innovation!” I hear you, but bear with me for a bit) What’s Going on Under the Hood? One potential explanation is quite interesting. Suppose: Patent stocks measure how much knowledge we have about a technology When we have more knowledge, it’s easier to discover new things The new knowledge we discover gets added to our existing knowledge If this is what’s “really” going on, then it explains why we have a positive link between last year’s patent stock and this year’s patenting. Knowledge begets more knowledge! And this is especially true of more recent knowledge, for which we have not already wrung out all the possible implications (hence the higher weight on recent patents). If we really believe this story, we can even use the estimated statistical models to make neat little forecasts of how technologies will develop. For every year, we can use last year’s patent stock to predict how many new patents will be discovered. We then use that to predict what next year’s patent stock will be. Rinse and repeat. There’s also a policy implication. If we can temporarily accelerate the accumulation of knowledge in a field, it can pay huge dividends. That’s because the benefits will compound, since they’ll enable more knowledge creation next year, which will enable more knowledge creation in the following year and so on. Moreover, this model implies path dependence is a powerful force in technology. If one technology gets a minor head start, it might keep it’s lead for a very long time. In fact, if the relationship is strong enough (a 10% increase in the patent stock increases patenting by 10% or more), then all else equal a technology that is a bit behind can never catch up! But before we get too far ahead of ourselves, we need to strongly consider some other potentially important explanations. Let’s jettison the conceptual category “patent stock” for a minute. So far all we have really shown is that patenting in the recent past is correlated with patenting today. To think through why that might be the case, we need to think through what kinds of factors determine the number of patents in a given year (for a firm, technology, or country). Since I’m an economist, I’m going to divide the potential factors into two categories: supply and demand. Supply factors are anything that affects the cost of getting patents (where cost is broadly construed). Demand factors are anything that affects the value of getting a patent. If it patents get cheaper or more valuable, then we should expect inventors to seek more patents. And if the things that made them cheaper or more valuable persist over time, then patenting in the recent past predicts patenting today. Yes, it’s true that knowledge is one kind of things that makes it cheaper to discover new things and get patents. But many other non-knowledge factors might also be important. On the supply side, that might include things like more scientists/inventors; more physical capital (computers, laboratories, etc); improved access to financing; more patent lawyers; and so on. On the demand side that might include demand from consumers; new regulations favoring certain kinds of technology; or even just oddball idiosyncratic stuff like demand from a new CEO who thinks the firm should be patenting more of its existing inventions. If we get any of these factors, we’ll get more patenting today and tomorrow (if the factor sticks around), which will deliver the correlation that patenting today predicts patenting tomorrow. But unlike the interesting “knowledge begets knowledge” theory, this doesn’t necessarily have the same policy implications, nor does it necessarily mean we have strong path dependency in technologies. If these other factors are driving the correlation, then if we increase patenting by hiring more scientists, subsidizing R&D, passing some new regulation, or whatever, the boost in patenting will...

New Things Under the Sun is a living literature review; as the state of the academic literature evolves, so do we. Here are three recent updates. Proximity, who you know, and knowledge transfer: Facebook edition The article Why proximity matters: who you know is about why cities seem to do a disproportionate amount of innovating. The article argues that an important reason is that proximity facilitates meeting new people, especially people who work on topics different from our own. These social ties are a channel through which new ideas and knowledge flows. The article goes on to argue that, once you know someone, it’s no longer very important that you remain geographically close. Distance matters for who you know, but isn’t so important for keeping those channels of information working, once a relationship has been formed. The article looks at a few lines of evidence on this. Diemer and Regan (2022) is a new article that tackles the same issue with a novel measure of “who you know.” Below is the new material I’ve added to my article, to bring in Diemer and Regan’s new work. The discussion picks up right after describing some evidence from Agrawal, Cockburn, and McHale that inventors who worked together in the past continue to cite each other work at an elevated level after they move far away from each other. While professional connections are probably the most likely to be useful for inventing, they are not the only kind of connection people have. If I make friends with people at a party, these friendships might also be a vehicle for the transmission of useful information. Diemer and Regan (2022) begins to address this gap with a novel measure of friendships: Facebook data. They have an index based on the number of friendships between Facebook users in different US counties, over a one-month snapshot in April 2016. Unfortunately, this measure of informal ties isn’t as granular as what Agrawal, Cockburn and McHale were able to come up with. If you’re an inventor with a patent, this Facebook dataset doesn’t tell the authors who your friends are and where they live; instead, it tells them something like, on average, how strong are friendship linkages between people in your county and other counties. Still, its one of the first large-scale datasets that lets us look at these kinds of social ties. Diemer and Regan want to see if these informal ties facilitate the transfer of ideas and knowledge by once again looking at patent citations. But this is challenging, because there are a whole host of possible confounding variables. To take one example, suppose: you’re more likely to be friends with people in your industry everyone in your industry lives in the same set of counties you’re also more likely to cite patents that belong to your industry That would create a correlation between friendly counties and citations, but it would be driven by the fact that these counties share a common industry, not informal knowledge exchange between friends. Diemer and Regan approach this by leveraging the massive scale of patenting data to really tighten down the comparison groups. Their main idea (which they borrowed from a 2006 paper by Peter Thompson) is to take advantage of the fact that about 40% of patent citations are added by the patent examiner, not the inventor. Instead of using cross-county friendships to predict whether patent x cites patent y, which would suffer from the kinds of problems discussed above, they use cross-county friendships to predict whether a given citation was added by the inventor, instead of the examiner. The idea is that both the patent examiner and the inventor will want to add relevant patent citations (for example, if both patents belong to the same industry, as discussed above). But a key difference is that only the inventor can add citations that the inventor knows about, and one way the inventor learns about patents is through their informal ties. So if patent x cites patent y, no matter who added the citation, we know x and y are probably technologically related, or there wouldn’t be a citation between them. But that doesn’t mean the inventor learned anything from patent y (or was even aware of it). But if patents from friendly counties are systematically more likely to be added by inventors, instead of otherwise equally relevant citations added by examiners, that’s evidence that friendship is facilitating knowledge transfer. Diemer and Regan actually look at three predictors of who added the citation: cross-county friendships, geographic distance between counties, and the presence of a professional network tie between the cited and citing patent (for example, is the patent by a former co-inventor or once-removed co-inventor). And at first glance, it does look like geographic distance matters: it turns out that if there is a citation crossing two counties, the citation is more likely to have been added by the inventor if the counties are close to each other. But when you combine all three measures, it turns out the effect of distance is entirely mediated by the other two factors. In other words, once you take into account who you know, distance doesn’t matter. Distance only appears to matter (in isolation) because we have more nearby professional ties and friendships, and we are more likely to cite patents linked to us by professional ties and friendships. Consistent with Agrawal, Cockburn, and McHale’s finding that 80% of excess citations from movers comes from people who are professionally connected, Diemer and Regan find professional network connections are a much stronger predictor of who added the citation than friendliness of counties, though both matter. Lastly, as with Agrawal, Cockburn, and McHale, when patent citations flow between more technologically dissimilar patents, the predictive power of how friendly two counties are looms larger. That’s consistent with friendships being especially useful for learning about things outside your normal professional network. But the bottom line is this - distance only matters, in this paper, because it affects who you know. Read the whole thing Even more knowledge transfer: reading edition Let’s stick with the theme of knowledge transfer for a moment. The article Free Knowledge and Innovation looked at three studies that document improving access to knowledge - via the Carnegie libraries, patent depository libraries, or wikipedia - has a measurable impact on innovation. Of these three studies, one by Furman, Nagler, and Watzinger looked at the impact of getting a local patent depository library, by comparing patent rates in nearby regions to the patent rate in other regions that were qualified to get a library but did not (for plausibly random factors). When I first wrote about it, the study was a working paper. It’s now been published and the new published version includes a new analysis that strengthens the case that increased access to patents leads to more knowledge transfer, and more patents. Below is some discussion of this new analysis. Furman, Nagler, and Watzinger … also look at the words in patents. After all, a lot of what we learn from patents we learn by reading the words. Furman, Nagler, and Watzinger try to tease out evidence that inventors learn by reading patents by breaking patents down into four categories: Patents that feature globally new words; words that never appeared before in any other patent Patents that feature regionally new words; words new to any patents of inventors who reside within 15 miles of the patent library or its control, but not new in the wider world Patents that feature regionally learned words; words that aren’t necessarily new to the patents of inventors who live within 15 miles of the library, but which were not used on any patents before the library showed up Patents that feature regionally familiar words; those that were already present in patents of inventors residing within 15 miles of the library, even prior to its opening. To take an example, the word “internet” first appeared in the title of patent 5309437, which was filed in 1990 by inventors residing in Maine and New Hampshire. So patent 5309437 features a global new word (Furman, Nagler, and Watzinger actually look at more than just the patent title, but this is just to illustrate the idea). I live in Des Moines, Iowa, where a patent depository library opened in the late 1980s. The first patent (title) mentioning the word “internet” with a Des Moines based inventor was filed in 2011. We would say that patent features a regionally new word, since no other Des Moines patents had the word “internet” in their title prior to 2011 but patents outside Des Moines did. If, in 2012, another Des Moines based-inventor later used the word “internet” in their patent we would classify that patent as a regionally learned word, since the word “internet” did not appear before our patent library was founded. Finally, a Des Moines based patent without the word “internet” or any other words that are new to the local patent corpus since we got our library would be classified as a familiar words patent. We would expect patent libraries to be especially helpful with regionally new and regionally learned words. These are signals that inventors in, say, Des Moines, are reading about patents from outside Des Moines and adopting new ideas they learn from them. And indeed, when you break patents down in this way, you see more patents of precisely the type you would expect, if people are reading patents and using what they learn to invent new things. From Furman, Nagler, and Watzinger (2021) On the other hand, we wouldn’t necessarily expect patent libraries to be as much help for globally new words, since those words are not found in any library - they are completely new to the world of patenting. Nor would we expect them to be much help for regionally familiar words, since those pertain to knowledge that was already available before the ...

To readers of New Things Under the Sun, Back in August, I wrote: This project began as something I did in my spare time, until November 2021, when Emergent Ventures generously took a chance and gave me a grant that let me spend 25% of my work time on this project. Going forward, I am working with an organization that will provide funding for me to work on it closer to 50% of my time, while keeping the site free. I’ll have more to say on that when everything is finalized. That day is here! Institute for Progress - About I’m pleased to announce that the Institute for Progress has named me a senior fellow and is partnering with New Things Under the Sun. Institute for Progress (IFP) is a new nonpartisan think tank, founded by Caleb Watney and Alec Stapp, with a mission to accelerate scientific, technological, and industrial progress. You can read their (great, in my view) launch document here. Obviously I’m biased, but I think New Things Under the Sun is an important public good. That said, it’s a weird thing and doesn’t neatly fall into a traditional academic role. It’s not the kind of academic research you publish in peer-reviewed journals and get tenure for; neither is it teaching to tuition-paying students. Iowa State University has been quite willing to let me work on this weird project, for which I’m grateful, but I’ve always thought the project would probably best be served, in the long-run, with external support. That’s what the partnership with IFP provides. To be clear; I’m still an assistant teaching professor at Iowa State University. But now a larger part of my time will be carved out for building and maintaining New Things Under the Sun. And even though I’m partnering with IFP, I remain the sole writer of New Things Under the Sun and I retain complete editorial control. The main differences will be more frequent updates to New Things Under the Sun, as well as an updated design to the newsletter and website. Thanks to IFP support, for 2022, I’m targeting about three posts per month, at least on average. My rough goal is that two of those will be new articles, and the third will be a bundle of updates to existing articles (look for just such a bundle in your inbox today). At the same time, I’m very much aligned with the goals of IFP. As a senior fellow, I plan to do additional work specifically for them, helping them ground science policy proposals in rigorous evidence. That will influence the kinds of topics I prioritize writing about in New Things Under the Sun, since it will influence the kinds of things I read and think about. I view this as a feature, not a bug though. One of the goals of New Things Under the Sun has always been to try and highlight academic work that is relevant for understanding how the world works. That’s why posts are written around advancing a specific claim, rather than as a tour of thematically connected papers. It’s my hope that being in dialogue with people trying to find concrete ways to accelerate scientific and technological progress will keep me focused on the ways research can shed light on the world we actually live in. Cheers all, Matt

Like the rest of New Things Under the Sun, this article will be updated as the state of the academic literature evolves; you can read the latest version here. Suppose we think there should be more research on some topic: asteroid deflection, the efficacy of social distancing, building safe artificial intelligence, etc. How do we get scientists to work more on the topic? Buy it One approach is to just pay people to work on the topic. Capitalism! The trouble is, this kind of approach can be expensive. To estimate just how expensive, Myers (2020) looks at the cost of inducing life scientists to apply for grants they would not normally apply for. His research context is the NIH, the US’ biggest funder of biomedical science. Normally, scientists seek NIH Funding by proposing their own research ideas. But sometimes the NIH wants researchers to work on some kind of specific project, and in those cases it uses a “request for applications” grant. Myers wants to see how big those grants need to be to induce people to change their research topics to fit the NIH’s preferences. Myers has data on all NIH “request for applications” (RFA) grant applications from 2002 to 2009, as well as the publication history of every applicant. RFA grants are ones where NIH solicits proposals related to a prespecified kind of research, instead of letting investigators propose their own topics (which is the bulk of what NIH does). Myers tries to measure how much of a stretch it is for a scientist to do research related to the RFA by measuring the similarity of the text between the RFA description and the abstract of each scientist’s most similar previously published article (more similar texts contain more of the same uncommon words). When we line up scientists left to right from least to most similar to a given RFA, we can see the probability they apply for the grant is higher the more similar they are (figure below). No surprise there. From Myers (2020) Myers can also do the same thing with the size of the award. As shown below, scientists are more likely to apply for grants when the money on offer is larger. Again, no surprise there. From Myers (2020) The interesting thing Myers does is combine all this information to estimate a tradeoff. How much do you need to increase the size of the grant in order to get someone with less similarity to apply for the grant at the same rate as someone with higher similarity? In other words, how much does it cost to get someone to change their research focus? This is a tricky problem for a couple reasons. First, you have to think about where these RFAs come from in the first place. For example, if some new disease attracts a lot of attention from both NIH administrators and scientists, maybe the scientists would have been eager to work on the topic anyway. That would overstate the willingness of scientists to change their research for grant funding, since they might not be willing to change absent this new and interesting disease. Another important nuance is that bigger funds attract more applicants, which lowers the probability any one of them wins. That would tend to understate the willingness of scientists to change their research for more funding. For instance, if the value of a grant increases ten-fold, but the number of applicants increases five-fold, then the effective increase in the expected value of the grant has only doubled (I win only a fifth as often, but when I do I get ten times as much). Myers provides some evidence that the first concern is not really an issue and explicitly models the second one. The upshot of all this work is that it’s quite expensive to get researchers to change their research focus. In general, Myers estimates getting one more scientist to apply (i.e., getting one whose research is typically more dissimilar than any of the current applicants, but more similar than those who didn’t apply) requires increasing the size of the grant by 40% or nearly half a million dollars over the life of a grant! Sell it Given that price tag, maybe a better approach is to try and sell scientists on the importance of the topic you think is understudied. Academic scientists do have a lot of discretion in what they choose to study; convince them to use it on the topic you think is important! The article “Gender and what gets researched” looked at some evidence that personal views on what’s important do affect what scientists choose to research: women are a bit more likely to do female-centric research then men, and men who are exposed to more women (when their schools go coed) are more likely to do gender-related research. But we also have a bit of evidence from other domains that scientists do shift priorities to work on what they think is important. Perhaps the cleanest evidence comes from Hill et al. (2021), which looks at how scientists responded to the covid-19 pandemic. In March 2020, it became clear to practically everyone in the world that more information on covid-19 and related topics was the most important thing in the world for scientists to work on. The scientific community responded: by May 2020 and through the rest of the year, about 1 in every 20-25 papers published was related to covid-19. And I don’t mean 1 in every 20-25 biomedical papers - I mean all papers! From Hill et al. (2021) This was a stunning shift by the standards of academia. For comparison, consider Packalen and Bhattacharya (2011), which looks at how biomedical research changed over the second-half of the twentieth century. Packalen and Bhattacharya classify 16 million biomedical publications, all the way back to 1950 and look at the gradual changes in disease burden that arise due to the aging of the US population and the growing obesity crisis. As diseases associated with being older and more obese became more prevalent in the USA, surely it was clear that those diseases were more important to research. Did the scientific establishment respond by doing more research related to those diseases? Sort of. As diseases related to the aging population become more common, the number of articles related to those diseases does increase. But the effect is a bit fragile - it disappears under some statistical models and reappears in others. Meanwhile, there seems to be no discernible link between the rise of obesity and research related to diseases more prevalent in a heavier population. Further emphasizing the extraordinary pivot into covid-related research, most of this pivot preceded changes in grant funding. The NIH did shift to issuing grants related to covid, but with a considerable lag, leaving most scientists to do their work without grant support. As illustrated below, the bulk of covid related grants arrived in September, months after the peak of covid publications (the NSF seems to have moved faster). From Hill et al. (2021) On the one hand, I think these studies do illustrate the common-sense idea that if you can change scientists beliefs about what research questions are important, then you can change the kind of research that gets done. But on the other hand, the weak results in Packalen and Bhattacharya (2011) are a bit concerning. Why isn’t there a stronger response to changing research needs, outside of global catastrophes? It’s hard I would point to two challenges to swift responses in science; these are also likely reasons why Myers (2020) finds it so expensive to induce scientists to apply for grants they would not normally apply for. Both reasons stem from the fact that a scientific contribution isn’t worth much unless you can convince other scientists it is, in fact, a contribution. The first challenge with convincing scientists to work on a new topic is there need to be other scientists around who care about the topic. This is related to the model presented in Akerlof and Michaillat (2018). Akerlof and Michaillat present a model where scientists’ work is evaluated by peers who are biased towards their own research paradigms. They show that if favorable evaluations are necessary to stay in your career (and transmit your paradigm to a new generation), then new paradigms can only survive when the number of adherents passes a critical threshold. Intuitively, even if you would like to study some specific obesity-related disease because you think it’s important, if you believe few other scientists agree, then you might choose not to study it, since it will be such a slog getting recognition. There’s a coordination challenge - without enough scholars working in a field, scholars might not want to work in a field. (This paper is also discussed in more detail here) The second challenge is that, even if there is a critical mass of scientists working on the topic, it may be hard for outsiders to make a significant contribution. That might make outsiders reluctant to join a field, and hence slow its growth. We have a few pieces of evidence that this is the case. Hill et al. (2021) quantify research “pivots” by looking at the distribution of journals cited in a scientists career and then measuring the similarity of journals cited in a new article to the journals cited in the scientist’s last three years. For example, my own research has been in the field of economics of innovation and if I write another paper in that vein, it’s likely to cite broadly the same mix of journals I’ve been citing (e.g., Research Policy, Management Science, and various general economics journals). Hill and coauthors’ measure would classify this as being a minimum pivot of close to 0. I also have written about remote work, and that was a bit of a pivot for me; the work cited a lot of journals in fields I didn’t normally cite up until this point (Journal of Labor Economics, Journal of Computer-Mediated Communication, but also plenty of economics journals). Hill and coauthors’ measure would classify this as an intermediate pivot, greater than 0 but a lot less than 1. But if I were to completely leave ec...

Matt, Dara, Jerusalem, and German use Matt’s last Tuesday episode to discuss life expectancy in the US. They explore paternalistic policy decisions, the misnomer of “deaths of despair,” and the longevity of The Weeds. US life expectancy is compared to that of European and Asian nations, and the US numbers are disaggregated and examined up close.

One dose of vaccine is not enough to protect against the Delta and Beta variants, new research suggests, and even people who have had Covid-19 should be immunized.

Israel reported that the Pfizer vaccine was 64 percent effective against the Delta variant. Other studies suggest a higher rate.