Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Garbage in, garbage out? Measuring error rates to get ready for AI

Thursday, January 8, 2026 • Rosenfeld Community
Share the love for this talk
Garbage in, garbage out? Measuring error rates to get ready for AI
Speakers: Caroline Jarrett
Link:

Summary

We’re all aware of a big push to implement AI everywhere, including in the services that many of us are working on. It seems only fair to try to give the AI some good quality input in the hope of getting decent output from it. Or, being more pessimistic: we probably expect to get some level of errors from the AI, but what do we know about the error rates in what we’re putting into the AI? In this session, we will compare our ideas on identifying errors and measuring error rates, including thinking about errors in six ways: 1) Problems along the way 2) Wrong result 3) Unnecessary action 4) Delayed-impact problem 5) Non-uptake or over-uptake 6) Technology problem We’ll wrap up with “tips and next steps”: an opportunity to consider what we now need to find out or do differently.

Key Insights

  • Errors in data collection and user input are foundational issues that compromise AI and service outcomes.

  • Users often 'fudge' answers due to ambiguous questions, privacy concerns, or to achieve a desired outcome.

  • Non-uptake, where users abandon a form or process, is a major source of error but is rarely published or measured.

  • Mistakes can be categorized as problems along the way, wrong results, unnecessary actions, and delayed impact issues.

  • Multiple accounts creation often occurs due to users forgetting existing accounts, leading to data duplicates and service inefficiencies.

  • Measuring error rates is complex; different metrics (per person, per attempt, completion vs. start) yield different perspectives.

  • Elections provide a useful model for measuring data quality, using turnout, participation, and eligibility rates.

  • Data quality deteriorates over time due to changes like moving, name changes, loss of documents, or organizational restructuring.

  • AI initiatives can provide a compelling rationale and funding opportunity for improving longstanding data quality problems.

  • Frameworks like the UK Government Data Quality Framework help organizations systematically assess and address data issues.

Notable Quotes

"If we get garbage in, we get garbage out — this is true for AI as much as for surveys or forms."

"People can make all sorts of inventive mistakes on their forms that AI struggles to interpret."

"Sometimes a form forces you into a wrong answer by giving inappropriate options."

"I’ve seen people fudge their date of birth so their child can attend a summer camp they aren’t technically eligible for."

"A major error in many services is users creating multiple accounts because they can’t find or reuse existing ones."

"An error might not be immediate; data can be fine when collected but deteriorate over time and cause problems later."

"Completion rates (conversion rates) and dropout rates are simple metrics but often not tracked or shared."

"Organizations rarely know their error rates, which limits their ability to improve user experience or data accuracy."

"Linking data quality efforts to AI initiatives can help secure attention and budget for necessary improvements."

"Data quality involves accuracy, completeness, uniqueness, timeliness, and representativeness—not just error reduction."

Ask the Rosenbot
Jerome “Axle” Brown
How to Use Self-Directed Learning to Ensure Your Research Insights are Heard and Acted Upon
2021 • Advancing Research 2021
Gold
Peter Levin
Solve a Problem Here, Transform a Strategy There: Research as an Occasion for Expanding Organizational Possibility
2024 • Advancing Research 2024
Gold
Tamara Hale
War Stories LIVE! Tamara Hale
2020 • Advancing Research 2020
Gold
Theresa Neil
Just Build Me a Dashboard!
2019 • Enterprise Community
Sarah Gallimore
Inspire Progress with Artifacts from the Future
2022 • Civic Design 2022
Gold
Susan Simon-Daniels
War Stories LIVE! Susan Simon-Daniels
2020 • Advancing Research 2020
Gold
Aleksandra Korczynska
Survey Tools
2026 • Advancing Research 2026
Gold
Kim Holt
A Salesforce Panel Discussion on Values-Driven DesignOps
2022 • DesignOps Summit 2022
Gold
Andrew Custage
The Digital Journey: Research on Consumer Frustration and Loyalty
2023 • Advancing Research 2023
Gold
Saara Kamppari-Miller
Theme Three Intro
2022 • DesignOps Summit 2022
Gold
Adam Thomas
Survival Metrics – Making Change in a Fast, Data-Informed, and Politically Safe Way
2022 • Design in Product 2022
Gold
Jen Briselli
Learning Is The Engine: Designing & Adapting in a World We Can’t Predict
2025 • Rosenfeld Community
Rachael Dietkus, LCSW
Trauma-Responsive Design: Reimagining the Future of Design Now
2021 • Civic Design 2021
Gold
Simon Wardley
Maps and Topographical Intelligence
2019 • Enterprise Community
John Cutler
The Alignment Trap
2023 • Design in Product 2023
Gold
Andreas Huebner
What Is It Like To Be Part of The UX Team at Compass?
2021 • Advancing Research 2021
Gold

More Videos

Rebecca Gimenez

"Brian Chesky does what Brian Chesky wants — he tweeted our blueprint, and we had nothing to do with that."

Rebecca Gimenez

Work in Progress: Service Design at Airbnb

December 3, 2024

Lada Gorlenko

"We have a choice in crisis — see it as a loss or see it as an opportunity."

Lada Gorlenko

Theme 2 Intro

June 9, 2022

Brigette Metzler

"Put more in than you take out — that’s the rule of thumb I give for contributing to repositories."

Brigette Metzler Dana Chrisfield

Research Repositories: A global project by the ResearchOps Community

August 27, 2020

"Find an unindicted co-conspirator in your company with a problem you can help solve, and the rest will follow."

Discussion

June 9, 2017

Sarah Brooks

"In traditional government projects, policy writers are often gone long before implementations fail or need revising."

Sarah Brooks Jennifer Pahlka

Fireside chat with Sarah Brooks and Jen Pahlka

October 21, 2021

Lisa Spitz

"Participants can skip any questions they don’t feel comfortable answering; most demographic questions are optional to respect privacy."

Lisa Spitz Nikki Brand

Building Trust Through Equitable Research Practices

November 18, 2022

Theresa Neil

"You aren't going to understand the healthcare ecosystem right off the bat; it’s incredibly complex."

Theresa Neil

Designing for Wellness: Specializing in Healthcare

May 22, 2024

Maverick Chan

"I haven’t gone more than one day without seeing a new gen AI tool that promises to do it all with a single prompt."

Maverick Chan Claire Lin

From Doodle to Demo: AI as Our Storytelling Partner

October 23, 2025

Alla Weinberg

"Leadership usually determines whether psychological safety is built or eroded in a culture."

Alla Weinberg

Design Teams Need Psychological Safety: Here’s How to Create It

September 8, 2022