Tackling randomised trials in economics

The 2019 Nobel Prize has been awarded to three scholars for pioneering recent attempts to answer microeconomic issues in development using randomised experiments.

Over the last three decades randomised trials have become an increasingly popular way of testing interventions designed to address developmental challenges.

But they are controversial. A range of scholars have criticised the use of the approach in development research. Criticism has touched on a number of dimensions. These include questions of ethics, methodological limitations and the danger that policy efforts get reoriented to small interventions. There is also no evidence that the approach leads to better development outcomes.

Academic work we have been involved in has pointed to problems of informed consent and the dangers of a conflict of interest in experiments with high political and economic stakes. And shown that there are fundamental methodological contradictions at the heart of the emphasis on randomised trials for policy.

Methodological problems

The argument for using randomised experiments is that they provide reliable estimates of the causal effects of potential policy interventions. And that they are therefore also the best source of evidence for policy.

An important criticism is that setting up an intervention can itself affect the outcome.

But scholars in economics have objected to both of these claims. These criticisms date back to the mid-1990s.

First, to the experiments. An important criticism is that setting up an intervention can itself affect the outcome. For example, individual behaviour can affect who participates in an experiment. It can also affect the way participants and non-participants react to the intervention.

Take an experiment where randomly selected school children are given extra tuition. One consequence could be that parents of non-selected students will compensate by paying for tutoring. Or they could spend more time helping with homework. For their part, parents of selected students might reduce such efforts.

In situations like this the idea of establishing a simple causal effect has been shown to be highly questionable.

In addition, most experiments only allow researchers to calculate average effects across groups. But for policy purposes, it is often necessary to have a sense of how interventions affect different people.

Linked to this is the fact that the actual effect of an intervention could change substantially (for better or worse) when it’s implemented at scale.

Experiments are usually implemented by research teams or their non-governmental organisation partner. But scaled-up policies are implemented by governments. This introduces a further set of dynamics that can affect implementation.

Perhaps the biggest problem is that there are many other factors that can affect the outcomes of interventions: researchers often do not know what these are and do not measure them. So deciding in advance whether an intervention that seemed to have positive outcomes in one place will do the same elsewhere becomes a matter of guesswork.

This undermines claims by proponents of these methods that randomised experiments lead to more ‘rigorous’ policy decisions than other approaches.

Ethical problems

Among the ethical problems, one concern is that social experiments in developing countries face serious problems of informed consent. Many experiments randomly allocate interventions to entire clusters, such as schools or hospitals, which makes it very difficult for participants to opt out.

And most experiments also involve people who are very poor. They are more likely to be unable to make meaningful choices about participating, particularly if it’s in exchange for desperately needed income or services.

Many experiments randomly allocate interventions to entire clusters, such as schools or hospitals, which makes it very difficult for participants to opt out.

The lack of informed consent also increases the risk of unintentional harm. If participants are aware that they are in an experiment, then they can alert experimenters to unintended negative consequences. This is important when experiments allocate critical resources, such as income or healthcare, to impoverished people. Withholding or providing resources to particular groups can harm vulnerable groups or lead to contestations that are socially destabilising.

These and other ethical concerns have prompted one respected economist to call for a moratorium on social experiments until effective ethical safeguards are put in place.

A distraction

The pioneers of development economics understood development to mean fundamental transformation at the societal level. This required going beyond marginal improvements to the status quo. In this conception, development was largely the outcome of sustainable increases in income levels in society. Through the detailed study of the history of the now developed countries, the pioneers concluded that such transformation was the result of industrialisation.

A body of research over the last 20 to 30 years also points to the primacy of industrialisation in the East Asian and Chinese development ‘miracles’.

The experimental turn in development economics has unfortunately distracted research and policy work from such age-old priorities of development. And in any case, some of the favoured micro-interventions of the new development economists (pricing of mosquito nets, provision of school flip charts, and so forth) would be outcomes – rather than causes – of transformative development.

Angus Deaton, the 2015 economics Nobel Laureate, has argued that in the case of Britain spectacular improvements in well-being in the late 18th and early 19th centuries followed on the heels of increases in the general level of incomes in the economy. Increased societal incomes allowed British society to marshal the resources needed to invest in, for example, large scale public sanitation infrastructure.

The proponents of randomised experiments in development are, therefore, arguably guilty of putting the cart before the horse.

A final critical concern is that there is no historical evidence to support the claim, repeated in this year’s economics Nobel award, that the experimental approach to development policy actually yields faster growth or development.

Many countries have grown their economies and developed in various ways without policy decisions being reliant on, or prioritising, randomised experiments.

These and other ethical concerns have prompted one respected economist to call for a moratorium on social experiments until effective ethical safeguards are put in place.

As we have noted, such experiments only address a limited set of possible development mechanisms that are susceptible to randomised experiments and coincide with researchers’ own pre-existing views. For example, teacher absenteeism is reduced to a crude question of incentives, rather than a set of complex systemic factors, and made the subject of an experiment where attendance is monitored. And these experiments rarely have any basis, either in theory or in practice, for being scaled up or applied in other contexts.

One area of particular concern, which has been the subject of much of the work by the Nobel awardees, is education. Our previous and forthcoming work raises numerous concerns with education experiments. The methodological basis for claiming policy relevance of even popular experiments like randomising school class size is deeply suspect. Some such experiments have been found to be illegal and unconstitutional. And a trickle down effect of this approach to local research has been to ignore or deny the role played by insufficient aggregate public resources for education.

Given all these factors, we suggest that instead of advancing development and poverty reduction, the approach advocated by this year’s economics Nobel winners could actually hold back progress in developing countries.

Seán Mfundza Muller, Senior Lecturer in Economics, Research Associate at the Public and Environmental Economics Research Centre (PEERC) and Visiting Fellow at the Johannesburg Institute of Advanced Study (JIAS), University of Johannesburg; Grieve Chelwa, Senior Lecturer -- Economics, University of Cape Town, and Nimi Hoffmann, Lecturer in International Education, University of Sussex.