Human vs. machine: Testing AI’s ability to synthesize and analyze research

Key Insights

• •

  AI tools frequently produce insight-shaped outputs but often lack the rigor and accuracy of trained human researchers.

• •

  AI moderators cannot currently assess user behavior beyond spoken words, missing key usability observations like failed or inefficient tasks.

• •

  Contextual elements such as environmental interruptions are critical in research but are invisible to AI tools.

• •

  Synthetic users generated by AI tend to produce overly positive, unrealistic feedback that can mislead product teams.

• •

  AI excels at finding semantic connections and grouping codes in large, already coded qualitative datasets quickly.

• •

  Meta-analysis of large repositories using AI can uncover recurring user themes, like change aversion, much faster than manual methods.

• •

  Integrating AI with organizational systems to pull in diverse data sources improves context but requires expert setup and is not yet simple.

• •

  AI’s context window limitations cause it to forget earlier input, affecting the accuracy of multi-turn interactions.

• •

  Even trained researchers must use AI outputs cautiously, vetting insights to maintain research quality.

• •

  Effective user research depends on human synthesis, collaboration, and contextual understanding, areas where AI currently fails.

Notable Quotes