Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Garbage in, garbage out? Measuring error rates to get ready for AI

Thursday, January 8, 2026 • Rosenfeld Community
Share the love for this talk
Garbage in, garbage out? Measuring error rates to get ready for AI
Speakers: Caroline Jarrett
Link:

Summary

We’re all aware of a big push to implement AI everywhere, including in the services that many of us are working on. It seems only fair to try to give the AI some good quality input in the hope of getting decent output from it. Or, being more pessimistic: we probably expect to get some level of errors from the AI, but what do we know about the error rates in what we’re putting into the AI? In this session, we will compare our ideas on identifying errors and measuring error rates, including thinking about errors in six ways: 1) Problems along the way 2) Wrong result 3) Unnecessary action 4) Delayed-impact problem 5) Non-uptake or over-uptake 6) Technology problem We’ll wrap up with “tips and next steps”: an opportunity to consider what we now need to find out or do differently.

Key Insights

  • Errors in data collection and user input are foundational issues that compromise AI and service outcomes.

  • Users often 'fudge' answers due to ambiguous questions, privacy concerns, or to achieve a desired outcome.

  • Non-uptake, where users abandon a form or process, is a major source of error but is rarely published or measured.

  • Mistakes can be categorized as problems along the way, wrong results, unnecessary actions, and delayed impact issues.

  • Multiple accounts creation often occurs due to users forgetting existing accounts, leading to data duplicates and service inefficiencies.

  • Measuring error rates is complex; different metrics (per person, per attempt, completion vs. start) yield different perspectives.

  • Elections provide a useful model for measuring data quality, using turnout, participation, and eligibility rates.

  • Data quality deteriorates over time due to changes like moving, name changes, loss of documents, or organizational restructuring.

  • AI initiatives can provide a compelling rationale and funding opportunity for improving longstanding data quality problems.

  • Frameworks like the UK Government Data Quality Framework help organizations systematically assess and address data issues.

Notable Quotes

"If we get garbage in, we get garbage out — this is true for AI as much as for surveys or forms."

"People can make all sorts of inventive mistakes on their forms that AI struggles to interpret."

"Sometimes a form forces you into a wrong answer by giving inappropriate options."

"I’ve seen people fudge their date of birth so their child can attend a summer camp they aren’t technically eligible for."

"A major error in many services is users creating multiple accounts because they can’t find or reuse existing ones."

"An error might not be immediate; data can be fine when collected but deteriorate over time and cause problems later."

"Completion rates (conversion rates) and dropout rates are simple metrics but often not tracked or shared."

"Organizations rarely know their error rates, which limits their ability to improve user experience or data accuracy."

"Linking data quality efforts to AI initiatives can help secure attention and budget for necessary improvements."

"Data quality involves accuracy, completeness, uniqueness, timeliness, and representativeness—not just error reduction."

Ask the Rosenbot
Tamara Hale
War Stories LIVE! Tamara Hale
2020 • Advancing Research 2020
Gold
Deirdre Hirschtritt
Research is Only as Good as the Relationships You Build
2022 • Civic Design 2022
Gold
Sam Proulx
Prototype Reviews, People With Disabilities, and You
2021 • DesignOps Summit 2021
Gold
Kevin M. Hoffman
Theme 2: Enterprise Team Journey
2019 • Enterprise Experience 2019
Gold
Laine Riley Prokay
How DesignOps can Drive Inclusive Career Ladders for All
2021 • DesignOps Summit 2021
Gold
Ben Reason
Making the system visible: The fastest path to better decisions
2025 • Advancing Service Design 2025
Gold
Chris Govias
Perspectives on Civic Design
2021 • Civic Design Community
Shipra Kayan
How we Built a VoC (Voice of the Customer) Practice at Upwork from the Ground Up
2021 • DesignOps Summit 2021
Gold
Natalia Radywyl
Co-Designing New Power in Australia's Public Sector
2022 • Civic Design 2022
Gold
Carla Casariego
DesignOps in Wonderland
2019 • DesignOps Summit 2019
Gold
Lavy Kumar
Future of Work
2021 • Design at Scale 2021
Gold
Taylor Klassman
Shaping the Next Era of UX Research: Collaborative Forum
2025 • Advancing Research 2025
Gold
Bud Caddell
DesignOps Community Sensing Session
2021 • DesignOps Community
Dave Hora
Advice for Establishing Research
2022 • Advancing Research Community
Jake Burghardt
Stop wasting research: Create new value with insight summaries
2025 • Rosenfeld Community
John Cutler
Oxbows, Rivers, and Estuaries: How to navigate the currents of change (without burning out)
2024 • Advancing Service Design 2024
Gold

More Videos

Sheri Byrne-Haber

"You’re not supposed to sell to the US public sector unless you’re WCAG 2.0 AA compliant."

Sheri Byrne-Haber

Accessibility at Scale

June 9, 2021

Prayag Narula

"If you skip iterative testing, you risk building features nobody really wants."

Prayag Narula Hannah Hudson

Empowering Designers to do Good Research

March 11, 2022

Janelle Estes

"Don't just flip the switch and say go talk to customers; you have to set guardrails and train people."

Janelle Estes

UX Research Trends

January 28, 2021

Craig Brookes

"Start with content first wireframing and avoid lorem ipsum to prevent treating words like decoration."

Craig Brookes Andreas Huebner Morgan Quinn

"Just Make it Look Good" and Other Ways We're Misunderstood

June 11, 2021

Marc Fonteijn

"Everyone goes through quite an extensive application process, but it was stress-free and informal."

Marc Fonteijn Ru Butler

Increase your confidence, influence, and impact (through a Professional Community)

December 3, 2024

Kate Towsey

"Managing vendors is a full-time job, and research ops teams often coordinate numerous vendors for tools and recruitment."

Kate Towsey

The State of ResearchOps: More Than Just Theory

June 20, 2019

Alla Weinberg

"Psychological safety is the number one factor determining team effectiveness and performance."

Alla Weinberg

Design Teams Need Psychological Safety: Here’s How to Create It

September 9, 2022

Shawna Hein

"We have over 30 contracting teams spread across roughly 10 vendors all working on parts of the veteran experience landscape."

Shawna Hein Kevin Hoffman

Create a Cohesive Civic Design Practice Across Agency, Vendors, and Contracts

November 17, 2022

Tricia Wang

"When ideas don't make it into the product, I do a post-mortem to share opportunities missed and improve the process."

Tricia Wang

SCALE: Discussion

June 15, 2018