Natural Experiments in Labor Economics and Beyond

a) The introduction of natural experiments to economic analysis of the labor market and related areas has led to a more precise understanding of cause and effect.

b) The ‘credibility revolution’ in empirical economics has improved our understanding of a number of public policy issues, including education, immigration and the minimum wage.

c) In pursuit of credible research designs, researchers often seek good answers instead of good questions.

Background on the first two questions has been provided in much of the coverage of the new Nobels, including a VoxEU column by Jörn-Steffen Pischke, who has co-authored books and papers with Joshua Angrist. The third was stimulated by their 2010 Journal of Economic Perspectives paper on the credibility revolution, which closes by discussing ‘the claim that the experimentalist paradigm leads researchers to look for good experiments, regardless of whether the questions they address are important.’

Of our 43 US experts, 41 participated in this survey; of our 48 European experts, 39 participated – for a total of 80 expert reactions.

Understanding cause and effect

On the first statement about natural experiments leading to a more precise understanding of cause and effect, a big majority of respondents on both panels agree. Weighted by each expert’s confidence in their response, 72% of the US panel strongly agree, 24% agree, 0% are uncertain, and 3% disagree (the totals don’t always sum to 100 because of rounding). Among the European panel (again weighted by each expert’s confidence in their response), 66% strongly agree, 31% agree, 3% are uncertain and 0% disagree.

Overall, across both panels, 69% strongly agree, 27% agree, 2% are uncertain, and 2% disagree.

Among the short comments that the experts are able to include in their responses, Larry Samuelson at Yale says: ‘Natural experiments are a welcome complement to other tools.’ Jan Pieter Krahnen at Goethe University Frankfurt comments: ‘Using natural experiments is an intelligent way to build causal inference on the diversity of institutions, shocks and behavior.’ And Franklin Allen at Imperial remarks: ‘As long as the natural experiment is well designed, these studies have the potential to increase our knowledge.’

Others mention applications beyond labor economics. Christian Leuz at Chicago replies:
‘Agree but not only in labor and micro, but also many other applied fields like finance and accounting.’ Pete Klenow at Stanford provides a reference to a survey of natural experiments in macroeconomics by European panel member Nicola Fuchs-Schundeln and a colleague. And Christopher Udry at Northwestern links to two experimental studies in development economics.

A few panelists express reservations. Anil Kashyap at Chicago notes: ‘Can’t answer all questions with these approaches, but we have definitely learned a lot.’ Patrick Honohan at Trinity College Dublin observes: ‘Although, as with randomized control trials, the evidence often comes from particular contexts that may not be generalizable.’ And Costas Meghir at Yale, who votes uncertain, adds: ‘Use of natural experiments without models relies on behavioral assumptions that are often unstated. May learn little about mechanisms.’

The ‘credibility revolution’ in empirical economics

On the second statement about the impact of the credibility revolution on policy debates in education, immigration and the minimum wage, again there is a big majority in agreement on both panels. Weighted by each expert’s confidence in their response, 60% of the US panel strongly agree, 32% agree, 7% are uncertain, and 0% disagree. Among the European panel (again weighted by each expert’s confidence in their response), 48% strongly agree, 52% agree, and 0% are uncertain or disagree.

Overall, across both panels, 54% strongly agree, 42% agree, 4% are uncertain, and 0% disagree.

Among the comments, Nicholas Bloom at Stanford says: ‘The research on the minimum wage has been pathbreaking – I absolutely changed my mind based on the evidence, and it has driven policy.’ Ricardo Reis at LSE adds: ‘Definitely improved. Of course, one would hope that progress had been even larger and more decisive. But no doubts on the sign of progress.’ And Costas Meghir notes: ‘There has been increased emphasis on justifying sources of exogenous variation in both structural and quasi-experimental studies.’

Several panelists refer to where natural experiments sit relative to other methods in research and policy evaluation. Larry Samuelson comments: ‘I view it as credibility evolution, with techniques that have built upon and evolved alongside other methods.’ Antoinette Schoar at MIT suggests: ‘Policy evaluation is complex and we need to draw on as many methods as possible for progress. But the credibility revolution is a key tool.’ Christian Leuz adds: ‘For policy, causal inferences are critical so need credible designs.’

Robert Shimer at Chicago, who votes uncertain, comments: ‘The devil is in the details with identification; and researchers often explore limited outcomes, e.g. only the short run.’ Also voting uncertain is Angus Deaton at Princeton, who says: Some plusses lots of minuses.’

Good answers and good questions

The third question – inviting views on whether in pursuit of credible research designs, researchers often seek good answers instead of good questions – generated considerably more differences of opinion. Weighted by each expert’s confidence in their response, 4% of the US panel strongly agree, 63% agree, 23% are uncertain, 7% disagree and 4% strongly disagree. Among the European panel (again weighted by each expert’s confidence in their response), 17% strongly agree, 36% agree, 37% are uncertain, and 10% disagree.

Overall, across both panels, 10% strongly agree, 49% agree, 30% are uncertain, 9% disagree, and 2% strongly disagree.

Among the comments of those who agree, Franklin Allen notes: ‘Unfortunately, there is some truth in this statement. It’s difficult to identify good natural experiments for many questions.’ Larry Samuelson adds: ‘This is the case with much of economics, and the social sciences more generally.’ And Anil Kashyap responds: ‘Running joke in the faculty lounge after some shock: “we will see a diff-in-diff paper on the job market using that variation next year…”’.

Others comment on incentives within economic research. Aaron Edlin at Berkeley states: ‘The profession continues to place a premium on clever identification strategies.’ Kenneth Judd at Stanford argues: ‘Too many economists focus on tractability, and even demand that one should know what the results will be before doing the analysis.’ And Luigi Guiso at the Einaudi Institute for Economics and Finance observes: ‘There is indeed a risk that the method drives the question and the risk is already real.’

Several panelists demur at ‘often’ in the statement. David Autor at MIT says: ‘This sometimes happens, for sure. But “often” is too high a bar for me. So, I disagree’; and Barry Eichengreen at Berkeley states: ‘If the question had been worded “sometimes” rather than “often” I would have agreed.’ David Cutler at Harvard concurs: ‘There are certainly some studies like this. “Often” is the key word here’; as does Daron Acemoglu at MIT: ‘I would say “sometimes” rather than “often”. Credibility revolution is fantastic for economics but we can/should not sacrifice big questions.’

Other panelists go further. Oriana Bandiera at LSE argues: ‘I agree that some researchers do this but you can’t blame the method, it’s like saying knives are bad because people use them to hurt others.’ Similarly, Antoinette Schoar comments: ‘I believe that methods can indeed shape the questions researchers ask. But this is true for all tools, including structural estimation, etc.’

Still others are critical of the way the question is framed. Richard Thaler at Chicago says: ‘Don’t love the wording here. Researchers try to answer some question, often it is not the most interesting one.’ Pol Antras at Harvard comments: ‘I don’t know what’s meant by “good”. I see a trade-off between credibility and generality (or external validity), but questions aren’t “bad”.’ And Abhjit Banerjee at MIT protests: ‘I really don’t know what that means. Should we pursue good questions which can’t be answered? Wittgenstein had it right.’

In similar vein, Richard Schmalensee at MIT argues: ‘Working on good questions that can’t be given good answers is a waste of effort; researchers aren’t wrong to consider quality of answers.’ And Michael Greenstone at Chicago objects: ‘This is a straw man debating style criticism. Is the alternative bad answers to good questions? Count me out for that!’

A final set of comments portray a trade-off. Ricardo Reis remarks: ‘Within set of good answers, they pick the better questions. Researchers trade off the two, unclear that they do so sub-optimally.’ Nicholas Bloom adds: ‘There is a trade-off between the quality of the question and the quality of the answer, but as long as we are on the frontier, all is good.’ And Olivier Blanchard at the Peterson Institute concludes: ‘The initial phase was indeed good answers. But we have moved over time to good questions.’

All comments made by the experts are in the full survey results.

Romesh Vaitilingam
@econromesh
October 2021