Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
UX Lessons from running more than 1,200 A/B Tests
Summary
Key Insights
-
•
Nine out of ten AB tests at Booking.com fail, but the few that succeed compound into massive growth.
-
•
Execution often fails on good ideas rather than the ideas themselves.
-
•
Technical details like page load time can negate positive design changes if not carefully managed.
-
•
Edge case bugs in different languages or currencies can silently kill conversions.
-
•
Minor typography choices, such as switching from Times New Roman to Arial, have a large impact on conversion.
-
•
Using system fonts improves legibility and page load speed, positively affecting conversions.
-
•
Tracking should only include users exposed to the tested change to avoid noisy data.
-
•
Qualitative user research drives many experiment ideas by revealing real user challenges.
-
•
Guardrail metrics like customer service calls and loyalty prevent short-term wins from harming long-term value.
-
•
Products should be retested over time as technology and user behaviors evolve, since what fails today may succeed tomorrow.
Notable Quotes
"I’m actually a really big failure — nine out of ten tests fail with no positive measurable impact."
"Compound effect means good upon good upon good eventually builds to incredibly fast growth."
"If a problem keeps coming up, it’s usually the execution of the idea that’s failing, not the concept."
"Increasing page load time by three seconds can nullify all your design improvements."
"Picking the wrong size image can impact customer experience as much as what you see on screen."
"There are as many versions of the website as stars in the sky because of language, currency, and device variations."
"No tracking is perfect because numbers tell you what likely happened, but not why it happened."
"System fonts load faster, improve legibility, and outperform fancy brand fonts in conversion."
"Guardrail metrics help catch unintended consequences like increased customer service calls despite higher conversions."
"You can’t fail 90 times out of 10 without laughing at yourself to keep going."
Or choose a question:
More Videos
"Icons are not just decoration, they convey information and must meet WCAG color and size guidelines to be usable."
Sheri Byrne-HaberThe Importance of Accessible Design Systems
January 8, 2024
"Hands-on activities like practicing user interviews and design sprint sketches gave people courage to try UX tactics in the real world."
Abbey Smalley Sylas SouzaScaling UX Past the Size of Your Team
January 8, 2024
"Large language models don’t think like us but have superpowers in pattern matching that help surface relationships in linked notes."
Jorge ArangoExploding the Notebook: How to Unlock the Power of Linked Notes (2nd of 3 seminars)
April 19, 2024
"Customer experience refers to the collection of interactions someone has with the brand, agnostic of channel or touchpoint."
Sarah WilliamsA Framework for CX Transformation
June 11, 2021
"People put stressful topics like money at the back of their head, so self-reported surveys on money stress are often inaccurate."
Yasmine KhanChecking Bias and Listening to Financially Vulnerable Americans
March 30, 2020
"Do we have the feedback loops to feel our way through paths of emergence rather than pre-carved routes?"
Dave MaloufTheme 3: Introduction and Provocation
January 8, 2024
"Our goal ultimately is not to deploy a static solution but to make sure humans in the system achieve their goals successfully as often as possible."
Jeff GothelfWho does what by how much?
November 20, 2025
"It's almost impossible to do a survey without using some kind of tool nowadays."
Aleksandra Korczynska Caroline Jarrett Justyna ParmeeSurvey Tools
March 12, 2026
"The absence of feedback during design demos was a red flag — we missed information and had to realign quickly."
Frank DuranPartnership Playbook: Lessons Learned in Effective Partnership
January 8, 2024
