DHET publication count: research excellence is more than a numbers game

Dear colleagues and students

As we are concluding our audit of the University of Cape Town (UCT)’s 2018 publication count, the Department of Higher Education and Training (DHET) has finalised its report on South African universities’ 2017 research outputs. Its results have drawn media attention, and there have been some claims that UCT’s research productivity is on the decline. This is, in fact, a simplistic and not entirely accurate reading of the report.

The fact that UCT was ranked sixth in the 2017 DHET report does not mean that UCT is the sixth most productive university in South Africa. It simply means that UCT was ranked sixth in South Africa for subsidies received from the department for research outputs (journal articles, books and conference proceedings), according to the specific methodologies and measurements used by the DHET (see discussion below).

Understanding the DHET Report

For every journal article, conference proceeding or book published, the DHET provides a financial subsidy as a way of distributing government funding to public universities. This subsidy income is what the DHET report is based on. Significantly, the subsidy income is weighted differently for each of these outputs and does not consider the quality of the journal or the citation impact of the output. It is, therefore, a fairly blunt tool – certainly compared to other research-quality indexes.

DHET subsidy income also depends on the proportion of an institution’s authors on any publication: The greater the number of collaborators outside the institution, the lower the subsidy income for that paper. This means that the DHET inadvertently penalises collaboration. There is, however, a global research movement towards greater collaboration to allow many perspectives to work together to solve the world’s wicked problems. The value of collaboration, especially global collaboration, has been shown to increase the impact of the findings of research and is so significant that some global university rankings factor international collaboration into their metrics.

A high number of UCT’s outputs are the result of collaborations – often large-scale, international collaborations – and this reduces its DHET subsidy income per output. For instance, in 2017, around 94% of UCT’s journal publications were in international indexes – substantially more than those of the top DHET subsidy earners, the University of KwaZulu-Natal (82%), University of Pretoria (86%), University of the Witwatersrand (86%), Stellenbosch University (85%) and University of Johannesburg (85%).

Subsidy income should therefore not be used as an accurate measure for quantity of publication output.

There are better tools for this purpose. An analysis of publication metrics extracted from SciVal (an analysis tool based on all international publications indexed by one of the world’s biggest databases, Scopus) reveals that UCT continues to produce the most publications per year compared to the five top-subsidy-income universities in South Africa. In other words, during 2017, UCT produced the most publications overall, as well as the most articles and reviews, specifically.

The DHET report also does not take into consideration the citation impact of papers: in other words, how useful those papers are to the academy. A SciVal analysis reveals that UCT’s publications have the highest field-weighted citation impact. This is the ratio of citations received relative to the expected world average for the subject field, publication type and year. (Although it’s worth noting that citation impact itself is a problematic measurement; as with almost all these measures, it is vulnerable to ‘gaming’.)

These numbers show that, by the measures used above, UCT’s productivity remains consistently high, as does the quality of our research.

The dangers of skewed incentives in research

Although the DHET distributes research income to universities according to their measure of research output, it is up to the individual institutions to decide how that subsidy should be disbursed. Many universities in South Africa pay the bulk of that money directly to researchers in proportion to their publication output. However, aware of the risks of perverse incentives, UCT does not give that subsidy directly to the relevant researchers. Instead, the bulk of the money is allocated to faculties as operational budget in direct relation to the proportion of units produced; each faculty chooses how to use that money to advance its research.

UCT’s position is that publishing papers simply for the sake of publishing does not advance research excellence, and incentivising publication alone – irrespective of journal or research quality – can detract from the quality of the research itself and from goals such as positive societal impact. As further evidence that we value a proxy of different indicators and not just numbers, it’s worth noting that at the most recent University Research Committee (URC) meeting the committee supported the adoption of the principles of the San Francisco Declaration on Research Assessment. UCT’s policy, we believe, contributes to UCT’s high-quality, high-impact research output demonstrated by the SciVal numbers above.

What it does not do is incentivise individual researchers to submit every publication to the central Research Office for the annual DHET publication count, as they likely would if they received the financial subsidy themselves.

Publication count challenges

The Research Office therefore relies heavily on already overloaded faculty administrators to help capture and submit correct journal titles for the publication count on which the DHET report is based. To catch journal titles that have not been captured and submitted, the Research Office extracts lists of journal titles from a number of databases – Scopus, Web of Science, International Bibliography of the Social Sciences and Sabinet – and painstakingly cross-checks them. However, we cannot be sure that this method captures every single title, despite our best, time-consuming efforts. The situation is even worse for books and book chapters, because there is only the Scopus database (which has more recently started indexing book titles, and may not yet be complete) from which we can extract a list and cross-check. As it happens, there appears to have been a decline in the number of books and book chapters published by UCT.

In addition to the existing service level agreement with faculties, which sets out the respective roles and responsibilities of all involved in the publication count, we are looking into ways of bolstering capacity to maximise the audit results.

What about eRA?

Many of you will be aware that UCT has moved much of its research administration over to the electronic Research Administration (eRA) systems, including for the publication count. You may be wondering if this is behind the perception of reduced output by our research community, but this is not the case.

eRA was first used for the publication count cycle in 2016, and if any problems had cropped up, it would have been during that year. By the time the 2017 publication cycle began, the problems in the system around publication count had been ironed out and the data was captured smoothly.

Moving forward

We are investigating the drop in the number of book outputs and undertaking a thorough analysis at faculty and departmental level to discover where there have been declines and the reasons for those, including whether there is a gap in reporting or a genuine decline.

Governments do not fund research to ensure a healthy academic publishing industry – they provide funding because of the very clear social benefits of scientific research. At UCT, that is our focus. This may not reflect in the DHET report, but it does reflect in the quality of our research outputs and our global rankings.

The UCT research community continues to strive for, and achieve, research excellence, and as an institution, we stand by our commitment to focus on research impact rather than high publication output numbers.

Sincerely

Professor Mamokgethi Phakeng
Vice-Chancellor

Read previous communications:

• From the UCT Executive
• Campus Announcements