Experimental research: techniques for deep, psychology-driven insights
Summary
In this session, we’ll unlock the transformative power of experimental research methods, inspired by best practices in the social sciences. We’ll explore what these techniques are and how they can be used to reveal powerful insights, drive innovation. You’ll also learn how to get started, how to apply them in ethical and rigorous ways, and why these methods are essential for deep, psychology-driven insights.
Key Insights
-
•
Experimental research goes beyond AB testing by revealing why changes work, revealing causation instead of just correlation.
-
•
Testing 20 trust signals simultaneously in one experiment is far more resource-efficient than running 20 separate AB tests.
-
•
Trust is multi-dimensional; measuring it at both platform and individual pro levels offers a more holistic understanding.
-
•
Random assignment and control groups are essential for isolating effects and attributing results to manipulated variables.
-
•
Half of the 20 trust signals tested had statistically significant positive effects on overall trust in Thumbtack.
-
•
Ten trust signals demonstrated broad impact across all eight trust metrics, providing a clear prioritization roadmap for product teams.
-
•
Experimental research can help product teams prioritize smartly by focusing on hypotheses likely to succeed before costly AB tests or full launches.
-
•
Ethics in experimentation require transparency, informed consent, and minimization of harm to participants.
-
•
Hypothetical scenarios in experiments improve control but have limitations in ecological validity, which can be addressed by validating in live products.
-
•
Understanding that correlation does not equal causation is crucial for UX researchers to avoid misleading conclusions and to rely on experiments.
Notable Quotes
"When we hear experiment, most think AB testing, but experimental research goes beyond to uncover the why behind outcomes."
"The independent variable is what you manipulate, dependent variable is what you measure to see if there’s an effect."
"Randomization ensures differences between groups come from the variable tested, not external factors."
"Half of the trust signals had a statistically significant positive effect on overall trust in Thumbtack, which is a big deal."
"Testing 20 trust signals simultaneously saved significant time and resources compared to running individual AB tests."
"Trust is complex, so we measured eight different trust-related outcomes at both the platform and professional level."
"Ethics are foundational in experiments since manipulating variables can impact participants' perceptions and emotions."
"Correlation does not equal causation. To say X causes Y, you’ve gotta do an experiment."
"Experimental research bridges the gap between curiosity and action by turning ideas into evidence and evidence into impact."
"Start small with one clear testable hypothesis and build from there to make experimental research approachable."
Or choose a question:
More Videos
"Lack of research before or after the workshop can be recovered from but at some cost to the concept’s validity."
Anne MamaghaniHow Your Organization's Generative Workshops Are Probably Going Wrong and How to Get Them Right
March 28, 2023
"No amount of clearer presentation or communications is going to solve this; it’s a cooperation problem."
Kurt McCullochFaster alone, further together: Rebuilding collaboration in the age of AI research
March 10, 2026
"Every project and initiative we launched has been a Trojan horse navigating through dark matter."
Sofía Delsordo Kassim VeraPublic Policy for Jalisco's Designers to Make Design Matter
December 8, 2021
"This is for me an amazing opportunity to see these ideas explored by brilliant people with a wonderful community."
Christian CrumlishIntroduction by our Conference Chair
December 6, 2022
"We created AI OKRs allowing our team members to experiment and then share their findings with the team."
Ebru NamaldiDesigning the Designer’s Journey: Scaling Teams, Culture, and Growth Through DesignOps
September 11, 2025
"Designers are uniquely entitled to critique—it’s a privilege to rigorously improve craft while others might envy that process."
Joseph MeersmanSweating the Pixel: Scaling Quality through Critique
June 10, 2021
"Our clients all have innovation portfolios, but if you zoom out, it's many micro decisions over time that are often opportunistic, lacking strategic connection."
Milan Guenther Benjamin KumpfThe $212 billion ‘so what?’: unlocking impact in development cooperation
November 20, 2025
"Allowing others to experience research firsthand can convert them into advocates for hiring more researchers."
Kathleen AsjesResearch Democratization: the Good, the Bad and the Ugly
March 10, 2022
"Machine intelligence can elevate experiences by easing frictions and amplifying human judgment and agency."
Josh Clark Veronika KindredSentient Design, AI, and the Radically Adaptive Experience (1st of 3 seminars)
January 15, 2025
Latest Books All books
Dig deeper with the Rosenbot
What are the benefits of linking AI-powered research repositories with tools like Slack for organizational adoption?
What are effective participatory engagement methods besides traditional surveys?
How can research repositories score or show the impact of research recommendations to enhance organizational learning?