Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

UX Lessons from running more than 1,200 A/B Tests

Wednesday, July 10, 2024 • Rosenfeld Community
Share the love for this talk
UX Lessons from running more than 1,200 A/B Tests
Speakers: Erin Weigel
Link:

Summary

Knowing how to solve the right problem is only one small part of the product-success equation. And nailing the execution is harder than most people think. As Erin says in this talk, “There are far more ways to fail than there are to succeed” when it comes to experimentation. To help you learn, Erin shares some of the silly mistakes she made while A/B testing her designs. That way you can avoid those pitfalls as you work to make your digital product better—not just different.

Key Insights

  • Nine out of ten AB tests at Booking.com fail, but the few that succeed compound into massive growth.

  • Execution often fails on good ideas rather than the ideas themselves.

  • Technical details like page load time can negate positive design changes if not carefully managed.

  • Edge case bugs in different languages or currencies can silently kill conversions.

  • Minor typography choices, such as switching from Times New Roman to Arial, have a large impact on conversion.

  • Using system fonts improves legibility and page load speed, positively affecting conversions.

  • Tracking should only include users exposed to the tested change to avoid noisy data.

  • Qualitative user research drives many experiment ideas by revealing real user challenges.

  • Guardrail metrics like customer service calls and loyalty prevent short-term wins from harming long-term value.

  • Products should be retested over time as technology and user behaviors evolve, since what fails today may succeed tomorrow.

Notable Quotes

"I’m actually a really big failure — nine out of ten tests fail with no positive measurable impact."

"Compound effect means good upon good upon good eventually builds to incredibly fast growth."

"If a problem keeps coming up, it’s usually the execution of the idea that’s failing, not the concept."

"Increasing page load time by three seconds can nullify all your design improvements."

"Picking the wrong size image can impact customer experience as much as what you see on screen."

"There are as many versions of the website as stars in the sky because of language, currency, and device variations."

"No tracking is perfect because numbers tell you what likely happened, but not why it happened."

"System fonts load faster, improve legibility, and outperform fancy brand fonts in conversion."

"Guardrail metrics help catch unintended consequences like increased customer service calls despite higher conversions."

"You can’t fail 90 times out of 10 without laughing at yourself to keep going."

Ask the Rosenbot
Dave Hoffer
UX Job Search AMA #3 with Joanne Weaver and Dave Hoffer
2025 • Rosenfeld Community
Jemma Ahmed
Convergent Research Techniques in Customer Journey Mapping
2020 • Advancing Research 2020
Gold
Steve Portigal
Looking Back…to Look Ahead
2024 • Advancing Research 2024
Gold
Gillian Salerno-Rebic
Redefining Speed and Scale: How Accenture’s GrowthOS Uses AI-Simulated Insights to Reduce Risk and Accelerate Innovation
2025 • Designing with AI 2025
Gold
Sarah Barrett
AI in Real Life: Using LLMs to Turbocharge Microsoft Learn
2025 • Rosenfeld Community
JP Allen
Navigating the UX Tools Landscape
2021 • DesignOps Summit 2021
Gold
Peter Boersma
How to Define and Maintain a DesignOps Roadmap
2023 • DesignOps Summit 2023
Gold
Amy Jiménez Márquez
The Atypical UX Manager Path
2020 • Enterprise Community
Dana Chisnell
The Sensemaking Business
2026 • Advancing Research 2026
Gold
Laurent Christoph
Scale the impact of DesignOps in 3D: Diligence, Decision, Discipline
2025 • DesignOps Community
Scher Foord
Turn the Ship Around: How to Apply Design Thinking Across Your Organization
2021 • Design at Scale 2021
Gold
Milan Guenther
A Shared Language for Co-Creating Ambitious Endeavours
2023 • Enterprise UX 2023
Gold
Sam Proulx
Designing For Screen Readers: Understanding the Mental Models and Techniques of Real Users
2021 • DesignOps Summit 2021
Gold
David Cronin
The GE Design System and Thoughts about Craft at Scale
2015 • Enterprise UX 2015
Gold
Renee Bouwens
Landing Product Impact: Aligning Research as a Foundational Driver for Delivering the World’s Best Products
2023 • QuantQual Interest Group
Bria Alexander
Day 2 Welcome
2024 • DesignOps Summit 2024
Gold

More Videos

Jim Kalbach

"Embracing uncertainty means diving in without knowing the outcome and creating something new every time."

Jim Kalbach

Jazz Improvisation as a Model for Team Collaboration

November 6, 2017

Louis Rosenfeld

"Inclusion, collaboration, and iteration are the three pillars of how we work with authors to make a book."

Louis Rosenfeld

Coffee with Lou: Should You Write a (UX) Book?

March 7, 2024

Catt Small

"Levels can mean really different things at different companies, so ask a lot of questions about expectations for each level."

Catt Small Micah Bennett Brian Carr Jessica Harllee

What's Next for ICs: Exploring Staff and Principal Designer Roles

February 22, 2024

Marieke McCloskey

"You might find that spending the most time getting people interested in collaboration is where the work really happens."

Marieke McCloskey

User Science: Product Analytics & User Research

March 11, 2021

Llewyn Paine

"Processing limits and licensing terms currently restrict how much video and audio these AI tools can handle."

Llewyn Paine

[Demo] Deploying AI doppelgangers to de-identify user research recordings

June 5, 2024

Joshua Noble

"Random assignment to treatment or control is essential—if it’s not random, it’s not a true experiment."

Joshua Noble

Casual Inference

October 6, 2023

Sara Logel

"Choosing the right output depends on the desired outcome—from quick iterations to gaining senior leadership buy-in."

Sara Logel

Your Colleagues are Your Users Too

March 29, 2023

Bria Alexander

"If something makes you feel unwelcome, the code of conduct explains how to engage with staff to resolve issues."

Bria Alexander Louis Rosenfeld

Welcome

January 8, 2024

Sam Proulx

"Consistency is so important that sometimes even consistency in failure works if it means I only have to learn the workaround once."

Sam Proulx

Online Shopping: Designing an Accessible Experience

June 7, 2023