Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Journeying toward AI-assisted documentation in healthcare

Gold
Wednesday, June 5, 2024 • Designing with AI 2024
Share the love for this talk
Journeying toward AI-assisted documentation in healthcare
Speakers: Jennifer Kong
Link:

Summary

Documentation technology is the foundation of modern healthcare delivery. Convoluted, redundant, and excessive documentation is a pervasive problem that causes inefficiency in all aspects of the industry. At IncludedHealth, we are developing an AI-assisted documentation that summarizes and documents conversations between patients and their care providers. A care provider can push one button and have their entire patient encounter captured in a succinct and standardized format. Upon a pilot launch, the results were staggering. Within 6 months, we demonstrated a 64% reduction in time per encounter! However, despite our promising results, there still remain challenges specific to the demands of the healthcare domain. As our team continues to develop solutions to meet these challenges, we gain even more clarity on what it takes to design a human-backed, AI-powered healthcare system. Takeaways From this session, you can expect to learn the following: Developing AI design in healthcare requires close collaboration between end users and your data science team Piloting GenAI solutions may be more effective than traditional prototyping Trading accuracy for efficiency is a barrier to adopting GenAI tools in healthcare GenAI design in healthcare requires establishing critical boundaries as well as a good understanding of cognitive processing Other factors to consider when designing AI solutions for service-based industries are understanding how training might be impacted, the importance of standardization vs. personalization of data output and the need for more autonomy and control elements due to consequences of unpredictable output errors

Key Insights

  • Generative AI can reduce healthcare documentation time by over 60% but doesn't eliminate the need for manual editing and human review.

  • Focusing AI tools initially on low-risk use cases like chat encounter summaries mitigates potential clinical harm.

  • Large Language Models (LLMs) excel at summarizing text but struggle with capturing exact clinical details and non-verbal cues.

  • Designing AI tools requires managing user expectations to prevent disappointment and frustration over imperfect outputs.

  • A simple one-button UI with options to regenerate and manually edit notes improves user workflow efficiency and error recovery.

  • Implicit metrics such as the 'edit rate'—the fraction of user edits on AI-generated text—help monitor AI output quality unobtrusively.

  • Traditional prototyping methods are less effective for AI; incremental live pilots provide critical learning about AI's unpredictable outputs.

  • Operational factors like quality assurance metrics impact user attitudes when AI-generated notes lower scores despite time savings.

  • Human-centered collaboration between designers, data scientists, and users is essential to shape AI capabilities and discover unforeseen problems.

  • AI models must remain static (non-learning) in healthcare to comply with regulations, creating challenges for ongoing improvements.

Notable Quotes

"We saw documentation time reduced by 64% after six months of using the AI tool."

"Documentation is important for liability and useful for patient handoffs, so notes must be concise yet detailed."

"Healthcare is not a silver bullet for AI; a lot of context comes from non-verbal cues where generative AI doesn’t apply."

"The unpredictable black box nature of LLMs means we had to zoom in on small problems first before seeing the bigger picture."

"We emphasized a simple one-button workflow with manual editing and regeneration to handle inevitable AI errors."

"The edit rate, our metric of human-added characters over total characters, tracks AI output quality without burdening users."

"Users were initially excited but six months later expressed frustration and uncertainty about the tool’s usefulness."

"We believe cognitive biases like frequency bias and expectation bias affected how users perceived AI errors over time."

"Our AI tool doesn’t learn from ongoing use due to regulatory constraints, which led to user frustration when performance didn’t improve."

"People are the plot twist in this journey — despite promising AI, only human connection reveals new problems and solutions."

Ask the Rosenbot
Nick Cochran
Growing in Enterprise Design through Making Connections
2019 • Enterprise Experience 2019
Gold
Ashley Cortez
Shifting Toward Community-Led Innovation in Local Government
2021 • Civic Design 2021
Gold
Sam Proulx
Prototype Reviews, People With Disabilities, and You
2021 • DesignOps Summit 2021
Gold
Megan Blocker
Theme 2 Intro
2025 • Advancing Research 2025
Gold
Peter Van Dijck
Coffee with Lou #4: Taking a Peek Under the Rosenbot's Hood
2024 • Rosenfeld Community
Jen Crim
Culture, DIBS & Recruiting
2021 • Design at Scale 2021
Gold
Michelle Bejian Lotia
Rolling Out a Repository: How Zapier Centralizes Insights from Across their Organization
2023 • Advancing Research 2023
Gold
Melissa Eggleston
Practical People Skills for Building Trust on Teams and with Partners
2021 • Civic Design 2021
Gold
Robin Beers
Research as a Catalyst for Organizational Transformation
2021 • Advancing Research 2021
Gold
Ted Neward
Theme 4: Enterprise Organizational Journey
2019 • Enterprise Experience 2019
Gold
Jennifer Bolduc
What's involved with getting people back to work?: A panel discussion
2021 • DesignOps Community
Kate Towsey
Participant Recruitment and Management Tools
2026 • Advancing Research 2026
Gold
Paula Bach
Improving Legacy Software: How Much Better Does it Have to Be?
2022 • Advancing Research 2022
Gold
Sam Ladner
How Research Can Drive Strategic Foresight
2022 • Advancing Research 2022
Gold
Robert Fabricant
Industry junctures: Paths forwards for UXR and the critical decisions that get us there [Advancing Research Community Workshop Series]
2024 • Advancing Research Community
Jorge Arango
The Best of Both Worlds: How to Integrate Paper and Digital Notes (1st of 3 seminars)
2024 • Rosenfeld Community

More Videos

Jim Kalbach

"Empathy in jazz means the band is in it together—when someone plays a wrong chord, the rest adapt and turn it into an opportunity."

Jim Kalbach

Jazz Improvisation as a Model for Team Collaboration

November 6, 2017

Louis Rosenfeld

"If your topic is already tackled exceptionally by a prior book, go deeper or wait until the market shifts."

Louis Rosenfeld

Coffee with Lou: Should You Write a (UX) Book?

March 7, 2024

Catt Small

"Levels can mean really different things at different companies, so ask a lot of questions about expectations for each level."

Catt Small Micah Bennett Brian Carr Jessica Harllee

What's Next for ICs: Exploring Staff and Principal Designer Roles

February 22, 2024

Marieke McCloskey

"Less than a third of respondents feel like they work somewhere with a clear investment in customer-driven decision making."

Marieke McCloskey

User Science: Product Analytics & User Research

March 11, 2021

Llewyn Paine

"Replacing the entire body with an avatar future-proofs against unanticipated identifiers like tattoos or moles."

Llewyn Paine

[Demo] Deploying AI doppelgangers to de-identify user research recordings

June 5, 2024

Joshua Noble

"If you cannot remove a treatment, it may not be a true cause."

Joshua Noble

Casual Inference

October 6, 2023

Sara Logel

"Storytelling helps gain people’s attention, build empathy, and improve understanding and recall of key points."

Sara Logel

Your Colleagues are Your Users Too

March 29, 2023

Bria Alexander

"Learning is a marathon, not a sprint, especially in the context of online conferences."

Bria Alexander Louis Rosenfeld

Welcome

January 8, 2024

Sam Proulx

"I’ve nearly bought products in my sleep because I memorized the key presses from consistent checkout flows."

Sam Proulx

Online Shopping: Designing an Accessible Experience

June 7, 2023