Experimental research: techniques for deep, psychology-driven insights
Summary
In this session, we’ll unlock the transformative power of experimental research methods, inspired by best practices in the social sciences. We’ll explore what these techniques are and how they can be used to reveal powerful insights, drive innovation. You’ll also learn how to get started, how to apply them in ethical and rigorous ways, and why these methods are essential for deep, psychology-driven insights.
Key Insights
-
•
Experimental research goes beyond AB testing by revealing why changes work, revealing causation instead of just correlation.
-
•
Testing 20 trust signals simultaneously in one experiment is far more resource-efficient than running 20 separate AB tests.
-
•
Trust is multi-dimensional; measuring it at both platform and individual pro levels offers a more holistic understanding.
-
•
Random assignment and control groups are essential for isolating effects and attributing results to manipulated variables.
-
•
Half of the 20 trust signals tested had statistically significant positive effects on overall trust in Thumbtack.
-
•
Ten trust signals demonstrated broad impact across all eight trust metrics, providing a clear prioritization roadmap for product teams.
-
•
Experimental research can help product teams prioritize smartly by focusing on hypotheses likely to succeed before costly AB tests or full launches.
-
•
Ethics in experimentation require transparency, informed consent, and minimization of harm to participants.
-
•
Hypothetical scenarios in experiments improve control but have limitations in ecological validity, which can be addressed by validating in live products.
-
•
Understanding that correlation does not equal causation is crucial for UX researchers to avoid misleading conclusions and to rely on experiments.
Notable Quotes
"When we hear experiment, most think AB testing, but experimental research goes beyond to uncover the why behind outcomes."
"The independent variable is what you manipulate, dependent variable is what you measure to see if there’s an effect."
"Randomization ensures differences between groups come from the variable tested, not external factors."
"Half of the trust signals had a statistically significant positive effect on overall trust in Thumbtack, which is a big deal."
"Testing 20 trust signals simultaneously saved significant time and resources compared to running individual AB tests."
"Trust is complex, so we measured eight different trust-related outcomes at both the platform and professional level."
"Ethics are foundational in experiments since manipulating variables can impact participants' perceptions and emotions."
"Correlation does not equal causation. To say X causes Y, you’ve gotta do an experiment."
"Experimental research bridges the gap between curiosity and action by turning ideas into evidence and evidence into impact."
"Start small with one clear testable hypothesis and build from there to make experimental research approachable."
Or choose a question:
More Videos
"The current wave of AI has disrupted practice so much, so quickly, that it's difficult to tell what practice is or what it will be in three years."
Nathan Shedroff Hugh Dubberly Thomas J. McLeishHow Will Design be Taught When the Schools Shut Down?
May 8, 2026
"We replaced the exhausting time estimation game with data-driven models that give us more reliable results."
John Paul de Guzman10k Screens Later: How We Became a Data-Driven Design Organization
September 24, 2024
"Advanced logic, matrix questioning, AI survey design creation — these are baseline capabilities you should expect from any modern survey tool."
Aleksandra Korczynska Caroline Jarrett Justyna ParmeeSurvey Tools
March 12, 2026
"One hallmark of a healthy team is how well team members feel cared for so they can shine."
Kristin SkinnerTheme 2: Introduction and Provocation
January 8, 2024
"The community is a big tent - welcoming diverse perspectives and geographies to define and shape civic design."
Rachael Dietkus, LCSW Victor Udoewa Jennifer StricklandEverything You Need to Know about the Civic Design 2022 Call for Presentations
May 17, 2022
"Enterprises are places where one hand doesn’t know what the other hand is doing and sometimes despises it."
Uday GajendarThe Rise of Meta-Design: A Starter Playbook
May 19, 2022
"Feedback is a lot like holding up a mirror. You're reflecting what's already working and like using a flashlight to hide in the corners."
Vanessa VarinFeedback: The Other F-Word
September 10, 2025
"Horst Rittel introduced wicked problems to describe problems where consensus on definition is difficult, and calling them problems can be misleading—maybe they’re better called messes or tangles."
Hugh DubberlyProblems with Problems: Reconsidering the Frame of Designing as Problem-Solving
June 19, 2019
"Balancing risk and confidence helps teams move at the right speed for the right decisions."
Sean McKayCoexisting with non-researchers: Practical strategies for a democratized research future
March 11, 2025
