Experimental research: techniques for deep, psychology-driven insights
Summary
In this session, we’ll unlock the transformative power of experimental research methods, inspired by best practices in the social sciences. We’ll explore what these techniques are and how they can be used to reveal powerful insights, drive innovation. You’ll also learn how to get started, how to apply them in ethical and rigorous ways, and why these methods are essential for deep, psychology-driven insights.
Key Insights
-
•
Experimental research goes beyond AB testing by revealing why changes work, revealing causation instead of just correlation.
-
•
Testing 20 trust signals simultaneously in one experiment is far more resource-efficient than running 20 separate AB tests.
-
•
Trust is multi-dimensional; measuring it at both platform and individual pro levels offers a more holistic understanding.
-
•
Random assignment and control groups are essential for isolating effects and attributing results to manipulated variables.
-
•
Half of the 20 trust signals tested had statistically significant positive effects on overall trust in Thumbtack.
-
•
Ten trust signals demonstrated broad impact across all eight trust metrics, providing a clear prioritization roadmap for product teams.
-
•
Experimental research can help product teams prioritize smartly by focusing on hypotheses likely to succeed before costly AB tests or full launches.
-
•
Ethics in experimentation require transparency, informed consent, and minimization of harm to participants.
-
•
Hypothetical scenarios in experiments improve control but have limitations in ecological validity, which can be addressed by validating in live products.
-
•
Understanding that correlation does not equal causation is crucial for UX researchers to avoid misleading conclusions and to rely on experiments.
Notable Quotes
"When we hear experiment, most think AB testing, but experimental research goes beyond to uncover the why behind outcomes."
"The independent variable is what you manipulate, dependent variable is what you measure to see if there’s an effect."
"Randomization ensures differences between groups come from the variable tested, not external factors."
"Half of the trust signals had a statistically significant positive effect on overall trust in Thumbtack, which is a big deal."
"Testing 20 trust signals simultaneously saved significant time and resources compared to running individual AB tests."
"Trust is complex, so we measured eight different trust-related outcomes at both the platform and professional level."
"Ethics are foundational in experiments since manipulating variables can impact participants' perceptions and emotions."
"Correlation does not equal causation. To say X causes Y, you’ve gotta do an experiment."
"Experimental research bridges the gap between curiosity and action by turning ideas into evidence and evidence into impact."
"Start small with one clear testable hypothesis and build from there to make experimental research approachable."
Or choose a question:
More Videos
"Embracing uncertainty means diving in without knowing the outcome and creating something new every time."
Jim KalbachJazz Improvisation as a Model for Team Collaboration
November 6, 2017
"A really strong book has to be designed as a journey with a consistent voice guiding the reader."
Louis RosenfeldCoffee with Lou: Should You Write a (UX) Book?
March 7, 2024
"If you’re sitting there thinking I need to build out this project and nobody else is saying the same thing, it means you have to do it, period."
Catt Small Micah Bennett Brian Carr Jessica HarlleeWhat's Next for ICs: Exploring Staff and Principal Designer Roles
February 22, 2024
"I invited Mark to a user interview — he had no idea interviewing wasn’t for him but respected that skill deeply."
Marieke McCloskeyUser Science: Product Analytics & User Research
March 11, 2021
"Blurring user faces loses context and doesn’t address voice privacy, making it an inadequate solution."
Llewyn Paine[Demo] Deploying AI doppelgangers to de-identify user research recordings
June 5, 2024
"Correlation doesn’t imply causation, but sometimes two things happening together do imply a causal mechanism."
Joshua NobleCasual Inference
October 6, 2023
"Not all learnings lead to crucial conversations, but big insights often do, requiring emotional awareness."
Sara LogelYour Colleagues are Your Users Too
March 29, 2023
"This is the same conference in so many good ways — good ways are that it’s resilient and there’s a certain steadiness."
Bria Alexander Louis RosenfeldWelcome
January 8, 2024
"I’ve nearly bought products in my sleep because I memorized the key presses from consistent checkout flows."
Sam ProulxOnline Shopping: Designing an Accessible Experience
June 7, 2023
Latest Books All books
Dig deeper with the Rosenbot
Why is design education’s emphasis on critical making important, and how does it interrelate with critique and theory?
What are the benefits of linking AI-powered research repositories with tools like Slack for organizational adoption?
Why is the emerging AI integration in healthcare seen as a major opportunity for UX professionals?