Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

UX Lessons from running more than 1,200 A/B Tests

Wednesday, July 10, 2024 • Rosenfeld Community
Share the love for this talk
UX Lessons from running more than 1,200 A/B Tests
Speakers: Erin Weigel
Link:

Summary

Knowing how to solve the right problem is only one small part of the product-success equation. And nailing the execution is harder than most people think. As Erin says in this talk, “There are far more ways to fail than there are to succeed” when it comes to experimentation. To help you learn, Erin shares some of the silly mistakes she made while A/B testing her designs. That way you can avoid those pitfalls as you work to make your digital product better—not just different.

Key Insights

  • Nine out of ten AB tests at Booking.com fail, but the few that succeed compound into massive growth.

  • Execution often fails on good ideas rather than the ideas themselves.

  • Technical details like page load time can negate positive design changes if not carefully managed.

  • Edge case bugs in different languages or currencies can silently kill conversions.

  • Minor typography choices, such as switching from Times New Roman to Arial, have a large impact on conversion.

  • Using system fonts improves legibility and page load speed, positively affecting conversions.

  • Tracking should only include users exposed to the tested change to avoid noisy data.

  • Qualitative user research drives many experiment ideas by revealing real user challenges.

  • Guardrail metrics like customer service calls and loyalty prevent short-term wins from harming long-term value.

  • Products should be retested over time as technology and user behaviors evolve, since what fails today may succeed tomorrow.

Notable Quotes

"I’m actually a really big failure — nine out of ten tests fail with no positive measurable impact."

"Compound effect means good upon good upon good eventually builds to incredibly fast growth."

"If a problem keeps coming up, it’s usually the execution of the idea that’s failing, not the concept."

"Increasing page load time by three seconds can nullify all your design improvements."

"Picking the wrong size image can impact customer experience as much as what you see on screen."

"There are as many versions of the website as stars in the sky because of language, currency, and device variations."

"No tracking is perfect because numbers tell you what likely happened, but not why it happened."

"System fonts load faster, improve legibility, and outperform fancy brand fonts in conversion."

"Guardrail metrics help catch unintended consequences like increased customer service calls despite higher conversions."

"You can’t fail 90 times out of 10 without laughing at yourself to keep going."

Ask the Rosenbot
Dane DeSutter
Keeping the Body in Mind: What Gestures and Embodied Actions Tell You That Users May Not
2024 • Advancing Research 2024
Gold
Mike Oren
Improving Democratized Research with CustomGPTs and Gems
2026 • Rosenfeld Community
Uday Gajendar
Leading through the long tail of trauma
2022 • Advancing Research Community
Devon Powers
Imagining Better Futures
2022 • Advancing Research 2022
Gold
Stephen Anderson
Puzzled? How to Coordinate Humans for Complex Challenges
2021 • Enterprise Community
Saara Kamppari-Miller
Cartography for Design Communities
2025 • DesignOps Summit 2025
Gold
Marc Fonteijn
First Insights from the 2025 Service Design Salary(+) Report
2024 • Advancing Service Design 2024
Gold
Tala Tayebi
Voice and influence in an age of noise
2026 • Advancing Research 2026
Gold
Janelle Estes
UX Research Trends
2021 • Advancing Research Community
Joshua Noble
Casual Inference
2023 • QuantQual Interest Group
Luca Rager
Empowering Gaming at Scale: How Xbox Builds Powerful, Automated, and Distributed Design Systems with Sketch
2021 • DesignOps Summit 2021
Gold
Jemma Ahmed
Theme 2 Intro
2024 • Enterprise Experience 2020
Gold
Angelos Arnis
State of DesignOps: Learnings from the 2021 Global Report
2021 • DesignOps Summit 2021
Gold
Uday Gajendar
Theme Four Intro
2023 • Enterprise UX 2023
Gold
Ryan Matthew
Bridging Design and Code: AI-Powered Design System Integration
2025 • Rosenfeld Community
Frances Yllana
DesignOps–Leading the Path to Parity
2023 • DesignOps Community

More Videos

Rebecca Gimenez

"Cycles of successive approximation help us get closer and closer to great design through iteration."

Rebecca Gimenez

Work in Progress: Service Design at Airbnb

December 3, 2024

Lada Gorlenko

"Mastercard designers grew meaningfully for both people and business despite uncertainty."

Lada Gorlenko

Theme 2 Intro

June 9, 2022

Brigette Metzler

"We partnered with Consent Kit to create and maintain an open source consent form builder that addresses multiple legal frameworks."

Brigette Metzler Dana Chrisfield

Research Repositories: A global project by the ResearchOps Community

August 27, 2020

"Align on what you’re looking for before the interview to evaluate non-traditional candidates fairly."

Discussion

June 9, 2017

Sarah Brooks

"In traditional government projects, policy writers are often gone long before implementations fail or need revising."

Sarah Brooks Jennifer Pahlka

Fireside chat with Sarah Brooks and Jen Pahlka

October 21, 2021

Lisa Spitz

"We constantly analyze our recruitment data and note that we still struggle to reach Hispanic or Latino origin individuals, older adults, and lower-income participants."

Lisa Spitz Nikki Brand

Building Trust Through Equitable Research Practices

November 18, 2022

Theresa Neil

"Designers almost always partner with clinicians who are the domain experts in healthcare projects."

Theresa Neil

Designing for Wellness: Specializing in Healthcare

May 22, 2024

Maverick Chan

"Using tools like 11 Labs, we instantly generated voices with emotion and appropriate cadence for characters."

Maverick Chan Claire Lin

From Doodle to Demo: AI as Our Storytelling Partner

October 23, 2025

Alla Weinberg

"Our nervous system does not know the difference between a tiger and an angry email from a manager—it just senses danger."

Alla Weinberg

Design Teams Need Psychological Safety: Here’s How to Create It

September 8, 2022