TLC2023: Classroom things – the ups and downs of ChatGPT

As the digital revolution gains ground globally, role players in higher education converged at the University of Cape Town (UCT) to discuss, debate and interrogate how best to reimagine teaching and learning methods using new ways of working, while addressing inequality, embracing innovation and promoting academic integrity.

These discussions took place against the backdrop of UCT’s 2023 Teaching and Learning Conference, which took place at the School of Education on Tuesday, 21 November, and Wednesday, 22 November, under the umbrella theme: “Assessment Entangled: Rethinking for Excellence, Transformation and Sustainability”.

One pertinent morning discussion explored the benefits and drawbacks of ChatGPT in the context of accounting. ChatGPT is an artificial intelligence (AI) chatbot that uses natural language processing to create humanlike conversational dialogue. The language model can respond to questions and compose various written content, including articles, social media posts, essays and emails.

“How can we think about using ChatGPT in the process of learning and teaching?”

Titled “Co-creating knowing for learning: What is the educational value of ChatGPT?”, the talk was presented by UCT’s Professor Ilse Lubbe from the College of Accounting and was based on a research project carried out in partnership with colleagues at the University of Australia in Perth.

“How can we think about using ChatGPT in the process of learning and teaching?” Lubbe asked.

Testing the bot

Professor Lubbe’s presentation shifted slightly from the conference theme that was centred on reimagining assessments amid the global digital revolution and pivoted towards the advantages and disadvantages of using ChatGPT in the classroom.

To test the chatbot’s accuracy levels, Lubbe used fundamental accounting threshold concepts that second-year accounting students should easily grasp. These concepts – control, consolidation and business combination – are specific to the field of accounting yet could easily be misinterpreted or confused with other disciplines. 

“Using these threshold concepts, we thought: What would ChatGPT tell us? Will a student be able to find [the meaning] of these threshold concepts using AI? What answers will ChatGPT give us? How reliable [will] ChatGPT be?” she asked.

Asking the right questions  

To help guide the research process, Lubbe established a simple research question: “How successful can ChatGPT be in articulating the application of these threshold concepts in preparation of consolidated financial statements?”

Thereafter, she formulated three key questions she needed the chatbot to answer. She said the type of questions users ask are crucial and often informs the response they receive. Her mix of questions were straightforward: What is a business combination and what is control? How would the chatbot prepare a consolidation? Finally, she asked ChatGPT to prepare a consolidation.

In its response to Lubbe’s first question, ChatGPT said: “Control has various meanings depending on the context in which it is used.” Further, the bot also said control relates to “authority or power” which, Lubbe added, is an important component in accounting, and refers to the concept of control within entities before consolidations are performed. The bot also indicated that control relates to the ability to influence and manage individuals.

“The description being provided here doesn’t make it context specific.”

“For [an accounting] student, when they ask ChatGPT: ‘What is control?’ The description being provided here doesn’t make it context specific to accounting. [It provides] a more general answer. I’m not saying that answer is wrong. What I am saying is that it’s a general answer,” Lubbe said.

Back to the drawing board

After its generic response, Lubbe said she reframed her question slightly to include: What is control in a business combination? And the answer ChatGPT then provided was in line with what an accounting student needed to hear.

“In the context of business combinations, control has a specific meaning relating to accounting and financial reporting. Control in this context refers to the power to govern the financial and operating policies of an entity in a way that allows the entity to obtain benefits from its activities,” ChatGPT said.

This explanation, Lubbe said, was correct. And what was doubly useful was the bot provided a succinct explanation in plain language that was simple for a student to understand, especially those students whose first language isn’t English.

The gaps

Interestingly, Lubbe said the bot also provided an accurate response to her question on the meaning of a consolidation in accounting. However, when it came down to the nitty gritty and the bot was asked to perform a consolidation, the financial statements it prepared contained several errors. Some of which, she said, included no elimination of intergroup transactions, and no goodwill recognion. 

“There are definitely some errors in there, and for me this is where the experiential value [of ChatGPT] lies. If I give my students this [ChatGPT set of financial statements] it’s an excellent opportunity for them to spot the errors,” she said.

So, like with most things, ChatGPT has its benefits and drawbacks.

“Our preliminary results [indicate] that ChatGPT can give us the theoretical knowledge that can help our students.”

“Our preliminary results [indicate] that ChatGPT can give us the theoretical knowledge that can help our students. Where it will be great [to use] in the classroom is to give students the opportunity to work through an answer and to see whether they are able to identify some of the significant, fundamental errors ChatGPT currently makes. And I say currently because I don’t know if ChatGPT will become cleverer as we feed it more questions,” she said.