Rosenverse
UX Lessons from running more than 1,200 A/B Tests

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

UX Lessons from running more than 1,200 A/B Tests

Wednesday, July 10, 2024 • Rosenfeld Community
Share the love for this talk
UX Lessons from running more than 1,200 A/B Tests
Speakers: Erin Weigel
Link:

Summary

Knowing how to solve the right problem is only one small part of the product-success equation. And nailing the execution is harder than most people think. As Erin says in this talk, “There are far more ways to fail than there are to succeed” when it comes to experimentation. To help you learn, Erin shares some of the silly mistakes she made while A/B testing her designs. That way you can avoid those pitfalls as you work to make your digital product better—not just different.

Key Insights

  • Nine out of ten AB tests at Booking.com fail, but the few that succeed compound into massive growth.

  • Execution often fails on good ideas rather than the ideas themselves.

  • Technical details like page load time can negate positive design changes if not carefully managed.

  • Edge case bugs in different languages or currencies can silently kill conversions.

  • Minor typography choices, such as switching from Times New Roman to Arial, have a large impact on conversion.

  • Using system fonts improves legibility and page load speed, positively affecting conversions.

  • Tracking should only include users exposed to the tested change to avoid noisy data.

  • Qualitative user research drives many experiment ideas by revealing real user challenges.

  • Guardrail metrics like customer service calls and loyalty prevent short-term wins from harming long-term value.

  • Products should be retested over time as technology and user behaviors evolve, since what fails today may succeed tomorrow.

Notable Quotes

"I’m actually a really big failure — nine out of ten tests fail with no positive measurable impact."

"Compound effect means good upon good upon good eventually builds to incredibly fast growth."

"If a problem keeps coming up, it’s usually the execution of the idea that’s failing, not the concept."

"Increasing page load time by three seconds can nullify all your design improvements."

"Picking the wrong size image can impact customer experience as much as what you see on screen."

"There are as many versions of the website as stars in the sky because of language, currency, and device variations."

"No tracking is perfect because numbers tell you what likely happened, but not why it happened."

"System fonts load faster, improve legibility, and outperform fancy brand fonts in conversion."

"Guardrail metrics help catch unintended consequences like increased customer service calls despite higher conversions."

"You can’t fail 90 times out of 10 without laughing at yourself to keep going."

Ask the Rosenbot
Emily Eagle
Can't Rewind: Radio and Retail
2019 • Enterprise Experience 2019
Gold
Emily Danielson
“I mean, I can lift a shovel”: Design Skills in Disaster Response
2022 • Design at Scale 2022
Gold
Allison Sanders
Operating with Purpose
2024 • DesignOps Summit 2020
Gold
Saskia Liebenberg
Start Small for Big Impact
2019 • DesignOps Community
Peter Merholz
The Trials and Tribulations of Directors of UX
2023 • Enterprise Community
Indra Klavins
A Design Ops Girl in a Dev Ops World
2019 • DesignOps Summit 2019
Gold
Elizabeth Churchill
Exploring Cadence: You, Your Team, and Your Enterprise
2017 • Enterprise Experience 2017
Gold
Candace Myers
Standardizing Design at Scale
2022 • DesignOps Summit 2022
Gold
Corey Long
Hiring in DesignOps: A Critical Study on How to Hire and Get Hired
2024 • DesignOps Summit 2024
Gold
Ned Dwyer
The Future of DesignOps is Tool Consolidation
2024 • DesignOps Summit 2024
Gold
Patrizia Bertini
DesignOps + KPIs = Measure your Impact!
2024 • DesignOps Summit 2020
Gold
Amelia Cole
Data-Prompted Interviews
2021 • QuantQual Interest Group
Erika Flowers
AI-Readiness: Preparing NASA for a Data-Driven, Agile Future
2025 • Designing with AI 2025
Gold
Christian Madsbjerg
Influencing Strategy
2020 • Advancing Research 2020
Gold
Joi Freeman
A New Vantage Point: Building a Pipeline for Multifaceted Research(ers)
2020 • Advancing Research 2020
Gold
Kat Vellos
Opener: The Other L Word
2024 • DesignOps Summit 2020
Gold

More Videos

Alexia Cohen

"Relinquishing control over implementation empowered community-driven solutions and fostered lasting trust."

Alexia Cohen Adriane Ackerman

Increasing Health Equity and Improving the Service Experience for Under-Served Latine Communities in Arizona

December 4, 2024

Marc Majers

"Generally everyone is just cutting right through because that’s the fastest route."

Marc Majers Tony Turner

Interrupted UX - Add A Dose of Reality To Usability Testing

March 11, 2022

Jorge Arango

"Design thinking has helped raise awareness but often lacks the depth of design practice which requires dedicated craft."

Jorge Arango

Design as an Antidote to VUCA

May 9, 2019

Brenna Fallon

"If you get a perfect OKR score, it means you didn’t set your sights high enough."

Brenna Fallon

Learning Over Outcomes

October 24, 2019

Daniel Gloyd

"The Shakers’ artifacts allowed moments of silence and moments of song—balancing rationality with spirit in design."

Daniel Gloyd

Warming the User Experience: Lessons from America's first and most radical human-centered designers

May 9, 2024

Sam Proulx

"Multiple payment providers aren’t just a feature; they’re an accessibility lifeline for people who can’t read or reach their credit card easily."

Sam Proulx

Online Shopping: Designing an Accessible Experience

October 3, 2023

Kristin Skinner

"Every student has unique strengths that should be recognized and nurtured."

Kristin Skinner

Five Years of DesignOps

September 29, 2021

Denise Jacobs

"It can be so tiring to be the only one constantly saying this has to be done differently."

Denise Jacobs Nancy Douyon Renee Reid Lisa Welchman

Interactive Keynote: Social Change by Design

January 8, 2024

Roberta Dombrowski

"Fraud detection automatically refunds you if a participant is found to be fraudulent and helps find a replacement."

Roberta Dombrowski Lianna Aduana

5 Reasons to Bring your Recruiting in House

September 30, 2021