Qualitative synthesis with ChatGPT: Better or worse than human intelligence?
Summary
- Effectiveness of ChatGPT in thematic analysis: Learn about my process of training ChatGPT to conduct precise thematic analysis. You’ll gain insights into the capabilities and limitations of ChatGPT in providing accurate and comprehensive analysis for framework construction
- Value of human potential: We’ll address the value of human self-reflection and the ability of interpreting organizational context for crafting engaging frameworks
- Comparison between human and ChatGPT: By comparing the human-driven outcomes against ChatGPT for qualitative analysis, you’ll see how effective the synthesized frameworks are generated by the researcher and ChatGPT separately.
- Collaboration between human and ChatGPT: You’ll also learn when and how to incorporate human interpretation with ChatGPT to achieve the best practice in qualitative analysis and synthesis
Key Insights
-
•
ChatGPT is more thorough and comprehensive than a fatigued human analyst, capturing details humans may miss.
-
•
ChatGPT struggles to create proper hierarchical organizing themes, often equating them in granularity with basic themes.
-
•
Human-generated frameworks excel in illustrating connections and flows between concepts, which ChatGPT tends to present as siloed, overlapping components.
-
•
AI-generated analogies, while existent, lack the coherence and nuance of human metaphors that effectively communicate complex relationships.
-
•
Qualitative synthesis requires subjective first-person experience and curiosity, qualities AI inherently lacks.
-
•
Users can use ChatGPT effectively as a tool to validate, triangulate, and summarize qualitative data, supplementing human analysis.
-
•
Thematic networks benefit from a visual and iterative human process, facilitating mental models that AI's bullet-point summaries do not provide.
-
•
Prompt engineering and role specification in ChatGPT do not currently resolve fundamental limitations in framework generation.
-
•
AI synthesis results should be viewed as stimuli for human curiosity and interpretation, not as definitive conclusions.
-
•
The most advanced qualitative synthesis goals, such as creating engaging analogies, remain a largely human prerogative due to AI's lack of experiential learning.
Notable Quotes
"ChatGPT has a big advantage in being thorough and comprehensive, especially compared to myself."
"Many of ChatGPT's organizing themes are at the same level as the basic themes, which is confusing and inappropriate."
"ChatGPT's frameworks generate siloed insights but fail to show the connections and flows essential for storytelling."
"Using analogy is very difficult for AI because it requires a unique human ability to connect and engage through storytelling."
"Qualitative synthesis is like watching a movie in a cinema; AI synthesis feels like watching a recap on YouTube—fast but missing the immersive experience."
"AI can’t experience data subjectively or fulfill curiosity, which drives human insight and discovery in qualitative research."
"We should not replace our curiosity and experience with AI but should instead use AI to speed up summarization and then slow down for human sensemaking."
"I tried different prompts and roles, but they didn’t fundamentally improve ChatGPT’s ability to generate meaningful frameworks."
"ChatGPT’s approach in analogy generation segmented the narrative into unrelated metaphors, which confused the overall story."
"AI synthesis should be considered a human-machine mutual learning process, with AI stimulating new perspectives rather than providing conclusions."
Or choose a question:
More Videos
"Accessibility ownership should never fall on just one person but be understood and shared across entire product teams."
Saara Kamppari-MillerDesignOps for Inclusive Design and Accessibility
May 26, 2022
"Biases out of sight are biases out of mind — facial recognition misidentifies Black faces leading to false arrests."
Dr. Jamika D. BurgeBroad Strokes: Connecting Design, Research, and AI to the World Around Us
June 7, 2023
"Integrate additional innovation skills like data analytics and behavioral economics to amplify the power of design."
Stephanie WadeBuilding and Sustaining Design in Government
December 8, 2021
"Sharing our playbook with engineering helped push design diversity thinking beyond our own team up to the C-suite level."
Michelle MorrisonCulture Design
May 21, 2020
"The best solutions come from good questions and sitting with the problem, not rushing to fix."
Joshua GravesWe Need To Talk: Addressing Unmet Expectations (Part 2 of 3)
April 28, 2025
"Culture often makes the biggest difference in how research ops works across industries."
Kate TowseyAsk Me Anything (AMA) with Kate Towsey
April 2, 2025
"Agile can feel like a cult, but the data helps show it’s working and not just a belief system."
Mackenzie Cockram Sara Branco Cunha Ian FranklinIntegrating Qualitative and Quantitative Research from Discovery to Live
December 16, 2022
"If your talk isn’t accepted, it doesn’t mean it’s bad—it might just not fit the program’s narrative arc or available slots."
Louis Rosenfeld Jemma Ahmed Christian Crumlish Uday Gajendar Chris GeisonCoffee with Lou #3: What Makes for a Successful UX Conference Presentation?
May 2, 2024
"We stopped talking about patterns and consistency as it made me the ‘patterns guy’—instead, I shifted language to what different roles cared about."
Eniola OluwoleLessons From the DesignOps Journey of the World's Largest Travel Site
October 24, 2019
