Rosenverse
UX Lessons from running more than 1,200 A/B Tests

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

UX Lessons from running more than 1,200 A/B Tests

Wednesday, July 10, 2024 • Rosenfeld Community
Share the love for this talk
UX Lessons from running more than 1,200 A/B Tests
Speakers: Erin Weigel
Link:

Summary

Knowing how to solve the right problem is only one small part of the product-success equation. And nailing the execution is harder than most people think. As Erin says in this talk, “There are far more ways to fail than there are to succeed” when it comes to experimentation. To help you learn, Erin shares some of the silly mistakes she made while A/B testing her designs. That way you can avoid those pitfalls as you work to make your digital product better—not just different.

Key Insights

  • Nine out of ten AB tests at Booking.com fail, but the few that succeed compound into massive growth.

  • Execution often fails on good ideas rather than the ideas themselves.

  • Technical details like page load time can negate positive design changes if not carefully managed.

  • Edge case bugs in different languages or currencies can silently kill conversions.

  • Minor typography choices, such as switching from Times New Roman to Arial, have a large impact on conversion.

  • Using system fonts improves legibility and page load speed, positively affecting conversions.

  • Tracking should only include users exposed to the tested change to avoid noisy data.

  • Qualitative user research drives many experiment ideas by revealing real user challenges.

  • Guardrail metrics like customer service calls and loyalty prevent short-term wins from harming long-term value.

  • Products should be retested over time as technology and user behaviors evolve, since what fails today may succeed tomorrow.

Notable Quotes

"I’m actually a really big failure — nine out of ten tests fail with no positive measurable impact."

"Compound effect means good upon good upon good eventually builds to incredibly fast growth."

"If a problem keeps coming up, it’s usually the execution of the idea that’s failing, not the concept."

"Increasing page load time by three seconds can nullify all your design improvements."

"Picking the wrong size image can impact customer experience as much as what you see on screen."

"There are as many versions of the website as stars in the sky because of language, currency, and device variations."

"No tracking is perfect because numbers tell you what likely happened, but not why it happened."

"System fonts load faster, improve legibility, and outperform fancy brand fonts in conversion."

"Guardrail metrics help catch unintended consequences like increased customer service calls despite higher conversions."

"You can’t fail 90 times out of 10 without laughing at yourself to keep going."

Ask the Rosenbot
Irina Tikhonova
Small Wins, Big Impact: Leveraging and Elevating User Engagement
2021 • Civic Design 2021
Gold
Adam Thomas
Survival Metrics – Making Change in a Fast, Data-Informed, and Politically Safe Way
2022 • Design in Product 2022
Gold
Adam Cutler
Discussion
2016 • Enterprise UX 2016
Gold
Vasileios Xanthopoulos
A Top-Down and Bottom-Up Approach to User-Centric Maturity at Scale
2024 • Enterprise Experience 2020
Gold
Jorge Arango
The Best of Both Worlds: How to Integrate Paper and Digital Notes (1st of 3 seminars)
2024 • Rosenfeld Community
Jon Fukuda
The Big Question about Innovation: A Panel Discussion
2024 • DesignOps Summit 2024
Gold
Ren Pope
Building Experiences for Knowledge Systems
2023 • Enterprise UX 2023
Gold
Savannah Carlin
Don't botch the bot: Designing interactions for AI
2024 • Designing with AI 2024
Gold
Sara Logel
Your Colleagues are Your Users Too
2023 • Advancing Research 2023
Gold
Peter Merholz
Design at Scale is People!
2021 • Design at Scale 2021
Gold
Ian Johnson
Latent Scope: Finding structure in unstructured data
2025 • Designing with AI 2025
Gold
Kit Unger
Theme 2: Introduction
2021 • Design at Scale 2021
Gold
Dr. Jamika D. Burge
The Future of Research: Bridging the Gaps
2021 • Advancing Research Community
Steve Portigal
War Stories LIVE! Steve Portigal
2020 • Advancing Research 2020
Gold
Lisa Gironda
Opener: Chief of Staff–An unexpected journey
2024 • DesignOps Summit 2020
Gold
Aurobinda Pradhan
Introduction to Collaborative DesignOps using Cubyts
2022 • DesignOps Summit 2022
Gold

More Videos

Nathan Shedroff

"What if everyone was accepted? We could rise almost anyone to proficiency if the curriculum and pedagogy are strong enough."

Nathan Shedroff Hugh Dubberly Thomas J. McLeish

How Will Design be Taught When the Schools Shut Down?

May 8, 2026

John Paul de Guzman

"Being design-driven is a means to an end. We aim to be 1% better every day with data-driven obsession."

John Paul de Guzman

10k Screens Later: How We Became a Data-Driven Design Organization

September 24, 2024

Aleksandra Korczynska

"Continuous feedback requires collecting responses tied to the same user across multiple channels with behavioral targeting."

Aleksandra Korczynska Caroline Jarrett Justyna Parmee

Survey Tools

March 12, 2026

Kristin Skinner

"What are some of the metrics that you’ll use to measure the health of your teams and practice?"

Kristin Skinner

Theme 2: Introduction and Provocation

January 8, 2024

Rachael Dietkus, LCSW

"You are allowed to say white supremacy culture here, even if you can’t say that in your day-to-day work."

Rachael Dietkus, LCSW Victor Udoewa Jennifer Strickland

Everything You Need to Know about the Civic Design 2022 Call for Presentations

May 17, 2022

Uday Gajendar

"Agile as trauma is a useful essay to understand how agile can harm design practices."

Uday Gajendar

The Rise of Meta-Design: A Starter Playbook

May 19, 2022

Vanessa Varin

"You can't do one without the other. Design the system, set rituals in the quality bar. They reinforce each other."

Vanessa Varin

Feedback: The Other F-Word

September 10, 2025

Hugh Dubberly

"You don’t think of raising a child or teaching a student as solving a problem, it’s an ongoing process—and so is design today."

Hugh Dubberly

Problems with Problems: Reconsidering the Frame of Designing as Problem-Solving

June 19, 2019

Sean McKay

"Balancing risk and confidence helps teams move at the right speed for the right decisions."

Sean McKay

Coexisting with non-researchers: Practical strategies for a democratized research future

March 11, 2025