Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Have fun with statistics?

Thursday, December 12, 2024 • Rosenfeld Community
Share the love for this talk
Have fun with statistics?
Speakers: Caroline Jarrett and Erin Weigel
Link:

Summary

Let’s face it, many of us feel daunted by statistics. But we also know that colleagues and clients ask whether our research has “statistically significant” results. Erin’s book Design for Impact helps you to test your hypotheses about improving design, and she guides you through deciding on your effect sizes to help you get those statistically significant results. Caroline’s book Surveys That Work talks about “significance in practice” and she’s not all that convinced about whether it’s worth aiming for statistical significance. Watch this lively session where Erin and Caroline compared and contrasted their ideas and approaches - helped by your questions and contributions.

Key Insights

  • Statistical significance often confuses practitioners because it requires mentally flipping hypotheses and disproving nulls, which is cognitively demanding.

  • Effect size is critical to understanding whether a change detected by statistics is meaningful in practice, a concept often neglected in statistics education.

  • Fast progress isn't necessarily good progress; teams benefit from slowing down and using statistics to ensure they're moving in the right direction.

  • Engineers can be reluctant to implement experiments due to the extra coding load, but they respond well when they understand the learning value gained.

  • Survey results (the numerical outcomes) are often confused with the number of respondents required for statistical significance, leading to misunderstandings.

  • Statistical thresholds like 95% confidence can be adjusted depending on project needs; lower confidence levels are sometimes acceptable.

  • A good hypothesis often starts as an intuitive guess, which gets refined over time through repeated testing and data collection.

  • Qualitative and quantitative research should be viewed as complementary tools in a holistic research approach rather than opposed methods.

  • AI tools can help generate first drafts of survey questions, but human-centered pilot testing is essential to avoid errors and misinterpretations.

  • It's common and acceptable to act on results that are significant in practice but not statistically significant, especially when outcomes clearly affect users.

Notable Quotes

"Statistics is hard because you have to flip flop in your head: think of a hypothesis, then a null hypothesis, then try to disprove the null."

"People confuse statistical significance with significance in practice — they want to know if the change is meaningful, not just mathematically significant."

"Fast is not a virtue in and of itself; moving slower and acting with intention ensures you go in the right direction."

"Engineers hate writing more code, so getting them to buy into experiments means showing the value of the learning on the other side."

"You can have an effect size that matters in practice but isn’t statistically significant, like five users failing a key task in usability testing."

"Most science starts with somebody pulling a number out of their ass — it’s okay to start with a gut instinct or guess."

"Statistics is another tool in our toolbox, part of a hierarchy of evidence that includes qualitative and quantitative methods."

"AI can create first drafts of survey questions, but unless you pilot test with real humans, you won’t know if your audience gets it."

"A lot of people think 95% confidence is the only way, but you can adjust confidence levels based on your situation and needs."

"Start with basics like means, minimums, and ranges — statistics rapidly becomes less mysterious and more useful with practice."

Ask the Rosenbot
Frances Yllana
D.E.A.R.R. Diaries (Discipline, Experience, Architecture, Reflection + Revolution)
2022 • Civic Design 2022
Gold
Daniel Gloyd
Warming the User Experience: Lessons from America's first and most radical human-centered designers
2024 • Rosenfeld Community
Nancy Douyon
We'll Figure That Out in the Next Launch: Enterprise Tech's Nobility Complex
2018 • Enterprise Experience 2018
Gold
Bria Alexander
The Big Question about Resilience: A panel discussion
2024 • DesignOps Summit 2024
Gold
Melinda Belcher
Bridging the Gap: Making the Most of the Differences Between Agency and Enterprise
2024 • Enterprise Experience 2020
Gold
Leah Buley
Closing Plenary: The Crisis of Digital
2020 • Advancing Research 2020
Gold
Sheri Byrne-Haber
The Importance of Accessible Design Systems
2024 • DesignOps Summit 2020
Gold
Bria Alexander
Opening Remarks
2021 • DesignOps Summit 2021
Gold
Sandra Camacho
Creating More Bias-Proof Designs
2025 • Rosenfeld Community
Cennydd Bowles
Exit Interview #2: Rediscovering the ethical heart of design
2025 • Rosenfeld Community
Greg Nudelman
Designing Conversational Interfaces
2019 • Enterprise Community
James Lang
If you can design an app, you can design a community
2025 • Rosenfeld Community
Sam Proulx
Accessibility: An Opportunity to Innovate
2022 • Civic Design 2022
Gold
George Zhang
UX Research Excellence Framework
2021 • Advancing Research 2021
Gold
Andy Polaine
What is the role of service design in product-led organizations?
2024 • Advancing Service Design 2024
Gold
Cassini Nazir
The Dangers of Empathy: Toward More Responsible Design Research
2023 • Advancing Research 2023
Gold

More Videos

Sheri Byrne-Haber

"Getting something accessible is a straight line; keeping something accessible requires process change."

Sheri Byrne-Haber

Accessibility at Scale

June 9, 2021

Prayag Narula

"Empathy isn't just kindness, it's the lens through which we see users' true motivations."

Prayag Narula Hannah Hudson

Empowering Designers to do Good Research

March 11, 2022

Janelle Estes

"The hardest skill to recruit for is someone who can consult and ask the right questions to uncover what the request really is."

Janelle Estes

UX Research Trends

January 28, 2021

Craig Brookes

"Our customers actually live in Siberia and they don't want ice cream, they want hot cookies out of the oven."

Craig Brookes Andreas Huebner Morgan Quinn

"Just Make it Look Good" and Other Ways We're Misunderstood

June 11, 2021

Marc Fonteijn

"We design a community that would be missed if it were gone for our service design leaders."

Marc Fonteijn Ru Butler

Increase your confidence, influence, and impact (through a Professional Community)

December 3, 2024

Kate Towsey

"Engagement and impact roles bring creativity and help maintain team morale in otherwise dry operations work."

Kate Towsey

The State of ResearchOps: More Than Just Theory

June 20, 2019

Alla Weinberg

"When we’re not safe, our thinking brain — the neocortex — shuts down and operating IQ drops by half."

Alla Weinberg

Design Teams Need Psychological Safety: Here’s How to Create It

September 9, 2022

Shawna Hein

"Slack channels and weekly meetings encourage cross-contract collaboration, avoiding duplication and sharing learnings across teams."

Shawna Hein Kevin Hoffman

Create a Cohesive Civic Design Practice Across Agency, Vendors, and Contracts

November 17, 2022

Tricia Wang

"You have to look two or three steps ahead so you're not chasing what's right in front of you — avoid target fixation."

Tricia Wang

SCALE: Discussion

June 15, 2018