Beyond the metrics: why care about how we assess research impact?

UCT has been grappling with the challenges of how to measure the impact of science beyond bibliometrics – in particular, the effect of the current system on the progress of young researchers and on encouraging socially responsive, interdisciplinary research that addresses South Africa’s and the continent’s problems. At a recent symposium intended to bring together thought-leaders to challenge our thinking and help us start to develop alternative tools, Professor Mamokgethi Phakeng introduced the event during the week before she assumed her position as UCT’s vice-chancellor. What follows are her thought-provoking words.

Why do you do the research that you do? What has been the impact of your research so far? What has it informed? What has it changed? What difference has it made?

How different would our society be without it? If you hadn’t done the research that you have done so far, what is it that we would have missed? Most importantly, how do you know that your research is making an impact? What is it that tells you that your research has had impact?

These questions are as relevant to individual researchers as they are to university management, funders and government, and this is the reason why we need to consider the important issue of moving beyond metrics to measure research impact.

Measuring the impact of our research is about considering what happens as a result of our research. That’s the tricky part, because it means a researcher can’t actually “do impact”. You can only undertake activities that enable impact to happen. The questions we ask, the theories, approaches and methodologies we use, as well as how we do our analysis and present our findings are all important and can enable impact. And of course, impact will happen when others take-up and use our research to change something.

I want to offer three provocations with a hope that they filter into our conversations about this important topic.

What are the shared values between sound research and sound democracy?

The first provocation is that research impact is a result of not only the knowledge that is produced. We should look at the impact of research also from the perspective of the values that the practice and process of research inculcates, especially in a young democracy such as ours.

But does research have a place in building democracy?

There are important similarities between research and democracy: the sound conduct of research and the sound conduct of democracy both depend on the same shared values. The very virtues that make democracy work are also those that make research work: a commitment to reason and transparency, an openness to critical scrutiny, a scepticism towards claims that too neatly support reigning values, a willingness to listen to countervailing opinions, a readiness to admit uncertainty and ignorance, and a respect for evidence gathered according to the sanctioned best practices of the moment.

Looking at research impact from this perspective not only elevates research, but it also elevates democracy. Of course, we can argue about whether research has a unique claim on these shared values. That is not important at this stage: what is important is that these values are critical, especially in our country where we must build a culture of democracy.

In strengthening democratic values, we also renew the preconditions for scientific discovery and technological innovation, and thus, high-impact research. The converse of this is also true – research with impact can serve as a precondition for building a vibrant democracy.

Should all of our research be for the public good?

The second provocation is that questions about research impact are often about the contestation of resources: where we invest them and why. We spend billions in public money on research annually, so we have to be accountable and consider its social, economic and environmental impact, as well as its impact on health and well-being and technological developments.

Questions about research impact also force us to consider whether our research spending is the best way to use our very limited resources. Considering the source of our research funding, it is very easy to argue that research should be for public good. But the question is, should all of our research be for the good of the public?

My view is that it is important to have a balance between curiosity-driven and mission-oriented research; research that tackles fundamental questions and research that serves corporate interests; and research for pleasure and research for pay.

Despite the need to engage in research for the public good, it is also necessary to create safe spaces where smart minds can tackle hard questions without any expectation of immediate applications. Like democracy, research is also a value to pursue for its own sake. The argument for engaging in curiosity-driven research that tackles fundamental questions or research for pleasure is always going to be difficult to sell because in a developing country, such as ours, research has a big responsibility to respond to the triple-challenge of unemployment, inequality and poverty. But scientific knowledge is a public good, therefore courageous policy-makers and funders should be prepared to pay for that resource without imposing a utilitarian approach on all publicly funded acquisition of knowledge.

How does the way we measure impact shape our research?

The third provocation is that the way we measure impact has implications for how we recognise and reward performance. It will therefore shape our research activity, research output and research training. If metrics drive research, the danger is that research can become formulaic – focused only on getting citations and impact factors right for purposes of career advancement and winning grants. This will encourage unethical behaviour and destroy our scholarship. We can already see this in the increase in the number of predatory journals as well as people who publish in them.

This is why we need to be clear about what counts as research impact and how we measure it.
 

This story was published in the second issue of Umthombo, a magazine featuring research stories from across the University of Cape Town. 

Umthombo is the isiXhosa word for a natural spring of water or fountain. The most notable features of a fountain are its natural occurrence and limitlessness. Umthombo as a name positions the University of Cape Town, and this publication in particular, as a non-depletable well of knowledge.

Read the complete second issue online or subscribe and receive new issues in your inbox every few months.