Are My Research Findings Actually Meaningful?
Summary
You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.
Key Insights
-
•
Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.
-
•
Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.
-
•
Statistical significance depends on sample size, variability, and magnitude of change.
-
•
A statistically significant change may not always be meaningful to the business context.
-
•
Meaningfulness is determined by alignment with organizational goals and market conditions.
-
•
Increasing sample size alone can make every change appear significant, which is misleading.
-
•
Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.
-
•
Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.
-
•
Effective communication involves framing findings differently for executives, managers, and individual contributors.
-
•
Frequent measurement without changes made leads to wasted effort and misinterpretation.
Notable Quotes
"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."
"Every board member can interpret an NPS score differently depending on their role and responsibilities."
"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."
"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."
"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."
"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."
"Customers benchmark your company to their last best experience, often outside of your industry."
"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."
"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."
"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."
Or choose a question:
More Videos
"Assistive technology has to communicate with the destination, that’s your website or product."
Sheri Byrne-HaberAccessibility at Scale
June 9, 2021
"Feature bloat kills usability because it clouds the core value proposition."
Prayag Narula Hannah HudsonEmpowering Designers to do Good Research
March 11, 2022
"Being open and flexible to what's most helpful for the team at that moment is key."
Janelle EstesUX Research Trends
January 28, 2021
"If you are spinning trying to make words work over and over, you likely have an interaction problem, not a word problem."
Craig Brookes Andreas Huebner Morgan Quinn"Just Make it Look Good" and Other Ways We're Misunderstood
June 11, 2021
"We encourage one-to-one video calls to build rapport and feel comfortable reaching out for advice."
Marc Fonteijn Ru ButlerIncrease your confidence, influence, and impact (through a Professional Community)
December 3, 2024
"Managing vendors is a full-time job, and research ops teams often coordinate numerous vendors for tools and recruitment."
Kate TowseyThe State of ResearchOps: More Than Just Theory
June 20, 2019
"Co-regulation means someone who is safe can help bring another out of a triggered state."
Alla WeinbergDesign Teams Need Psychological Safety: Here’s How to Create It
September 9, 2022
"We have over 30 contracting teams spread across roughly 10 vendors all working on parts of the veteran experience landscape."
Shawna Hein Kevin HoffmanCreate a Cohesive Civic Design Practice Across Agency, Vendors, and Contracts
November 17, 2022
"You have to look two or three steps ahead so you're not chasing what's right in front of you — avoid target fixation."
Tricia WangSCALE: Discussion
June 15, 2018
Latest Books All books
Dig deeper with the Rosenbot
What benefits come from integrating apprenticeship or internship models into design education?
What strategies help speed up the procurement and legal review of UX research platforms in organizations like LinkedIn?
What are the economic and demographic challenges threatening independent design colleges like CCA?