Experimental research: techniques for deep, psychology-driven insights
Summary
Key Insights
-
•
Experimental research goes beyond AB testing by revealing why changes work, revealing causation instead of just correlation.
-
•
Testing 20 trust signals simultaneously in one experiment is far more resource-efficient than running 20 separate AB tests.
-
•
Trust is multi-dimensional; measuring it at both platform and individual pro levels offers a more holistic understanding.
-
•
Random assignment and control groups are essential for isolating effects and attributing results to manipulated variables.
-
•
Half of the 20 trust signals tested had statistically significant positive effects on overall trust in Thumbtack.
-
•
Ten trust signals demonstrated broad impact across all eight trust metrics, providing a clear prioritization roadmap for product teams.
-
•
Experimental research can help product teams prioritize smartly by focusing on hypotheses likely to succeed before costly AB tests or full launches.
-
•
Ethics in experimentation require transparency, informed consent, and minimization of harm to participants.
-
•
Hypothetical scenarios in experiments improve control but have limitations in ecological validity, which can be addressed by validating in live products.
-
•
Understanding that correlation does not equal causation is crucial for UX researchers to avoid misleading conclusions and to rely on experiments.
Notable Quotes
"When we hear experiment, most think AB testing, but experimental research goes beyond to uncover the why behind outcomes."
"The independent variable is what you manipulate, dependent variable is what you measure to see if there’s an effect."
"Randomization ensures differences between groups come from the variable tested, not external factors."
"Half of the trust signals had a statistically significant positive effect on overall trust in Thumbtack, which is a big deal."
"Testing 20 trust signals simultaneously saved significant time and resources compared to running individual AB tests."
"Trust is complex, so we measured eight different trust-related outcomes at both the platform and professional level."
"Ethics are foundational in experiments since manipulating variables can impact participants' perceptions and emotions."
"Correlation does not equal causation. To say X causes Y, you’ve gotta do an experiment."
"Experimental research bridges the gap between curiosity and action by turning ideas into evidence and evidence into impact."
"Start small with one clear testable hypothesis and build from there to make experimental research approachable."
Or choose a question:
More Videos
"Many organizations silo data inputs and prioritize within categories before combining results."
Mark Interrante Harry MaxAI for Prioritization (3rd of 3 seminars)
July 11, 2024
"If you are not speaking to the CEO or C-executives about compliance and transformation, someone else will."
Patrizia BertiniThe (r)evolution of designOps: It’s Time to Think (really) BIG
September 11, 2025
"The phrase card sort may not require translation itself, but the activity it represents certainly does."
Shan ShenTranslating UX Terms into Business Contexts
November 29, 2023
"Stop, pivot, or invest early—these are the survival metric decisions that save companies."
Adam ThomasSurvival Metrics – Making Change in a Fast, Data-Informed, and Politically Safe Way
December 6, 2022
"Research is inherently a silo-busting activity because customers experience the whole, not just parts divided by departments."
Robin BeersResearch as a Catalyst for Organizational Transformation
March 12, 2021
"Accessibility at scale can put you into the VUCA world – volatile, uncertain, complex, ambiguous."
Sheri Byrne-HaberAccessibility at Scale
June 9, 2021
"Critiques maintain focus on the quality of our outcomes and sweat every pixel we put in the hands of our users."
Joseph MeersmanSweating the Pixel: Scaling Quality through Critique
June 10, 2021
"It’s not just about design; build relationships with HR, product management, sales, and more."
Deanna Washington Bria Alexander Jon Fukuda Saara Kamppari-Miller Farid SabitovConnecting the Ops: Plenary Panel and Closing Circle
September 9, 2022
"Government isn’t always trusted, so we worked with local partners to connect with underserved communities."
Aaron Stienstra Lashanda HodgeLeveraging Civic Design to Advance Equity and Rebuild Trust in the US Federal Government
December 8, 2021
Latest Books All books
Dig deeper with the Rosenbot
How can smaller or solo research teams scale insight sharing in large organizations?
How can measuring team behaviors instead of just results improve trust and performance in design ops?
Who are natural allies for healthcare UX professionals inside complex healthcare systems, and how do you engage them?