Qualitative Data Analysis: Investigating the Potential for LLMs for Inductive and Deductive Coding

Investigating the Potential of LLMs for Inductive and Deductive Coding Simon Buckingham Shum (Connected Intelligence Centre), Antonette Shibani (TD School), Lisa Lim (Connected Intelligence Centre) & Ram Ramanathan (Connected Intelligence Centre)

Until recently, qualitative data analysis (QDA), such as the deductive and inductive coding of textual data, was considered the preserve of human researchers. The nuanced judgements required to apply a complex coding scheme, or to discern themes that evolve into a coding scheme, were beyond algorithms. However, the emergence and mainstream availability of large language models (LLMs: e.g., GPT, Gemini, Claude, Llama) has catalysed rigorous research into their ability to perform such QDA in minutes. This is accompanied by healthy debate on whether this could lead to the full automation of certain kinds of analysis, or the augmentation of their work through productive, hybrid analysis with a new generation of interactive QDA tools. Using LLMs hosted by privacy-respecting, secure, university instances, we have been testing LLMs for both inductive and deductive coding, and welcome your thoughts on how we address important considerations including: 

• How can we translate a theory-grounded codebook into a system prompt guiding the LLM? 
• How do we evaluate the quality of the coding compared to human researchers?
• Since (like humans) LLMs are intrinsically variable in their coding, how do we understand and manage this variability?
• How can an LLM provide a transparent account of its inductive coding of a corpus so humans can understand it?
• How will human and machine analysts work together in the future, harnessing their respective strengths?
• What concerns do researchers have about automated coding, and can these be addressed?

This is a UTS Aspire event. UTS Aspire is a program of pan-university research development opportunities led by the Research Capability and Development Team.