Investigating the Potential for LLMs for Inductive and Deductive Coding

Investigating the Potential of LLMs for Inductive and Deductive Coding Simon Buckingham Shum (Connected Intelligence Centre), Antonette Shibani (TD School), Lisa Lim (Connected Intelligence Centre) & Ram Ramanathan (Connected Intelligence Centre)

Until recently, qualitative data analysis (QDA), such as the deductive and inductive coding of textual data, was considered the preserve of human researchers. The nuanced judgements required to apply a complex coding scheme, or to discern themes that evolve into a coding scheme, were beyond algorithms. However, the emergence and mainstream availability of large language models (LLMs: e.g., GPT, Gemini, Claude, Llama) has catalysed rigorous research into their ability to perform such QDA in minutes. This is accompanied by healthy debate on whether this could lead to the full automation of certain kinds of analysis, or the augmentation of their work through productive, hybrid analysis with a new generation of interactive QDA tools. Using LLMs hosted by privacy-respecting, secure, university instances, we have been testing LLMs for both inductive and deductive coding, and welcome your thoughts on how we address important considerations including: 

• How can we translate a theory-grounded codebook into a system prompt guiding the LLM? 
• How do we evaluate the quality of the coding compared to human researchers?
• Since (like humans) LLMs are intrinsically variable in their coding, how do we understand and manage this variability?
• How can an LLM provide a transparent account of its inductive coding of a corpus so humans can understand it?
• How will human and machine analysts work together in the future, harnessing their respective strengths?
• What concerns do researchers have about automated coding, and can these be addressed?

Events in the 2025 Qualitative Data Analysis series

• Thematic Analysis - Big Q and little q – Tuesday 25 February 2025, 12:30pm – 1:30pm
• Autoethnography - The possibilities – Tuesday 18 March 2025, 12:30pm – 1:30pm
• Investigating the Potential for LLMs for Inductive and Deductive Coding – Tuesday 22 April 2025, 12:30pm – 1:30pm
• Autoethnography - Triangulations – Tuesday 29 April, 12:30pm – 1:30pm
• Inductive Content Analysis - Tuesday 13 May, 12:30pm - 1:30pm
• Using mixed methods to bring together big data, and qualitative research in the Wikihistories project - Tuesday 10 June, 12:30pm - 1:30pm
• Mess in QDA – Tuesday 8 July, 12:30pm – 1:30pm
• Applying concepts and underlying 'ologies' – Tuesday 9 September, 12:30pm – 1:30pm
• Deep Dive into Thematic Analysis - From Concepts to Practice – Tuesday 7 October, 12:30pm- 1:30pm
• Non-word-based techniques in qualitative research and analysis – Tuesday 11 November, 12:30pm – 1:30pm   

This is a UTS Aspire event. UTS Aspire is a program of pan-university research development opportunities led by the Research Capability and Development Team.