Are My Research Findings Actually Meaningful?
Summary
You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.
Key Insights
-
•
Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.
-
•
Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.
-
•
Statistical significance depends on sample size, variability, and magnitude of change.
-
•
A statistically significant change may not always be meaningful to the business context.
-
•
Meaningfulness is determined by alignment with organizational goals and market conditions.
-
•
Increasing sample size alone can make every change appear significant, which is misleading.
-
•
Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.
-
•
Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.
-
•
Effective communication involves framing findings differently for executives, managers, and individual contributors.
-
•
Frequent measurement without changes made leads to wasted effort and misinterpretation.
Notable Quotes
"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."
"Every board member can interpret an NPS score differently depending on their role and responsibilities."
"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."
"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."
"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."
"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."
"Customers benchmark your company to their last best experience, often outside of your industry."
"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."
"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."
"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."
Or choose a question:
More Videos
"Hugh Dubberly argues we should stop describing design as problem solving and instead see it as engaged conversation."
Daniel GloydDesigning From the Inside Out: How Method Acting Can Inspire Design Research
February 12, 2026
"Inviting leaders to shadow research sessions helped one department hire a client feedback panel representing marginalized voices."
Deirdre Hirschtritt Cesar Paredes Marie PerrotResearch is Only as Good as the Relationships You Build
November 17, 2022
"After four years we were able to see in a concrete way what happened, the changes that we have been able to do for the organization."
Benjamin RealMaturity Models: A Core Tool for Creating a DesignOps Strategy
October 1, 2021
"Check the people on the board and the advisors because that tells you a lot about the company’s true values."
Max Gadney Andrea Petrucci Joshua Stehr Hannah WickesAssessing UX jobs for impact in climate
August 14, 2024
"Mykelaan’s story is a playbook on dealing with very difficult high-rank stakeholders."
Lada GorlenkoTheme 1: Intro
January 8, 2024
"Within five minutes and two seconds, they just dove right in and started collaborating."
Feyikemi AkinwolemiwaPlay to innovate: How curiosity and experimentation transform UX
March 11, 2026
"Sponsor sessions are optional but offer incredibly high content, and they are not sales pitches."
Bria AlexanderOpening Remarks
March 29, 2023
"Processing limits and licensing terms currently restrict how much video and audio these AI tools can handle."
Llewyn Paine[Demo] Deploying AI doppelgangers to de-identify user research recordings
June 5, 2024
"Accessibility is a great example where OKRs can unify multiple product and design teams to a non-negotiable standard."
Bria Alexander Benson Low Natalya Pemberton Stephanie GoldthorpeOKRs—Helpful or Harmful?
January 20, 2022
