Are My Research Findings Actually Meaningful?
Summary
Key Insights
-
•
Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.
-
•
Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.
-
•
Statistical significance depends on sample size, variability, and magnitude of change.
-
•
A statistically significant change may not always be meaningful to the business context.
-
•
Meaningfulness is determined by alignment with organizational goals and market conditions.
-
•
Increasing sample size alone can make every change appear significant, which is misleading.
-
•
Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.
-
•
Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.
-
•
Effective communication involves framing findings differently for executives, managers, and individual contributors.
-
•
Frequent measurement without changes made leads to wasted effort and misinterpretation.
Notable Quotes
"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."
"Every board member can interpret an NPS score differently depending on their role and responsibilities."
"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."
"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."
"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."
"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."
"Customers benchmark your company to their last best experience, often outside of your industry."
"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."
"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."
"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."
Or choose a question:
More Videos
"Slow and intentional communication allows you to be fully present rather than distracted by constant notifications."
Marc Fonteijn Ru ButlerIncrease your confidence, influence, and impact (through a Professional Community)
December 3, 2024
"Asking stupid questions can lead experts to reframe their thinking and improve the work."
Dem Gerolemou Alexis Oh Neef RehmanClimate technology fundamentals
May 15, 2024
"Doing at least two benchmarking rounds lets you see if optimizations have measurable impacts over time."
Daniela Magaña Flores Ariane Rahn Jeff Ephraim BanderAhead of Competition: Learn What UX Benchmarking Can Do for Your Business Today
March 10, 2022
"Hands-on activities like practicing user interviews and design sprint sketches gave people courage to try UX tactics in the real world."
Abbey Smalley Sylas SouzaScaling UX Past the Size of Your Team
January 8, 2024
"If you’re like us, you’d love to have 100% of the UX talent every project needs, but your team isn’t large enough."
Patrick CommarfordDesign Staffing for Impact
January 8, 2024
"The Researcher in Residence program invites members of affected communities to use our tools over a funded three-month period pro bono."
Xenia Adjoubei Sean BruceEmpowering Communities Through the Researcher in Residence Program
March 29, 2023
"If you’re not paying for the product, you are the product."
Craig VillamorDesign Systems for Ethical Design
January 26, 2023
"This isn’t more work—it's consolidating work that’s already happening scattered across tools into one connected place."
Jon Fukuda Ellie KryslDesign Planning and Management Support
October 3, 2023
"Business language is for communication. We use it to explain why user centricity matters to broader audiences."
Iain McMaster IHan ChengDesign and Product: from Frenemy to Harmony
November 29, 2023
