University rankings under scrutiny

Each year the release of the Times Higher Education World University Rankings generates a great deal of interest, excitement – and debate. This year all eyes are on China which, according to the 2018 rankings, “steps up its ascent” as “East closes in on West”. China’s Tsinghua University in 22nd place has overtaken all of its other Asian competitors.

Only one African institution, the University of Cape Town, made it to the top 200. Other good performers include University of Witwatersrand and University of Stellenbosch, which are both in the top 350.

But what does this mean? Parents, guardians and potential students may want to know what, if any, relationship is there between the rankings and an institution’s quality of teaching? Can we assume that highly ranked universities also have high quality teaching? And conversely, do we assume that low-ranked universities have poor quality teaching?

Ranking and quality of teaching has been subject of much controversy. The Times Higher Education rankings and similar ones are heavily weighted for research. Teaching, which “assesses the learning environment”, accounts for only 30% of a university’s overall ranking.

This is then broken down into the following key performance indicators: reputation survey; staff-to-student ratio; doctorate-to-bachelor’s ratio; the ratio of doctorates awarded to academic staff and institutional income. All of these are related to input factors – what is required to teach, like academics and money. None of the indicators have anything to do with outputs – the results of teaching, for example, course success rates, time to completion and graduate employment.

Essentially, the rankings heavily favour research-intensive, well-resourced universities. They say nothing about the actual quality of teaching as experienced by students or academics.

So, is there a relationship between rankings and quality of teaching? It depends what is meant by quality of teaching.

Quality of teaching as access to powerful knowledge

A recent book by three British academics, Quality in Undergraduate Education, explores the relationship between the quality of education and institutional status.

At stake is the role of higher education in relation to inequality: do universities simply reproduce inequality or can they disrupt it? The authors investigated this vexed and complex issue through a three-year longitudinal study of four higher education institutions in the UK.

Two were “high status” institutions and two were “low status”. High status institutions were the older, research-intensive, typically wealthier universities and the low status were more vocationally-oriented universities established from 1992. The trio analysed interviews with academics and students, observations of classrooms and curriculum documentation, including assessment.

In contrast to the Times Higher Education rankings which focus on inputs and reputation, their definition of quality focused on outputs or products. Quality being the extent to which teaching gives students’ access to “powerful knowledge”.

Powerful knowledge, they argue, is when theory and everyday common sense knowledge align. Quality teaching enables students to meaningfully traverse the gap between theory and lived experience – their own and others’.

The trio’s findings showed no clear relationship between university status and quality teaching. In fact, one of the low-status universities best showcased the powerful and life-changing nature of knowledge.

Redefining quality and inequality

The study should cause us to pause and challenge the assumptions we make about rankings, status and the quality of the student experience. Is there a relationship between the rankings and the actual quality of teaching? It depends on how quality of teaching is defined.

Assessing the quality of teaching in ways that can be standardised and compared isn’t simple. That’s why we often end up valuing what we can measure, instead of measuring what we value. Measuring the quality of teaching on the basis of input factors as the Times Higher Education rankings, is not enough. There are “output” indicators that can be used – for example, dropout and retention, student experience surveys and graduate employability.

The study also raises another question, what is the relationship between quality and addressing inequality?

In South Africa we know this kind of output data when separated out by race reveals persistent inequalities with racial differentiations in academic performance. Tackling these challenges to ensure parity of participation is a hallmark of quality teaching.

As for the Times Higher Education rankings, there may be changes on the horizon. New ways of measuring teaching, such as student surveys, are being piloted. Duncan Ross, data analytics director for the Times Higher Education rankings, announced a review of the rankings’ methodology for 2020. As an example, he raised the issue of whether universities’ gender balance should be assessed and asked:

Can a university that isn’t adequately serving half the population be said to be world-leading?

The same question can be asked, how and to what extent are first generation university, black and minority students being served? With inequality as one of the greatest challenges of the 21st century, should this not also be a feature of world-leading universities?

It would be a significant advance if these increasingly influential rankings could showcase those institutions as “world-leading” who are making a contribution to social justice through the quality of their teaching.

Suellen Shay, Associate Professor, University of Cape Town.