Designing AI-first products on top of a rapidly evolving technology
This video is featured in the AI and UX playlist.
Summary
This talk addresses the distinct challenges of designing AI-first products when the foundational technology is advancing at an unprecedented pace. We will discuss the practical implications of this rapid evolution on the design process and share our insights into creating effective user experiences in such a dynamic environment.
Key Insights
-
•
AI models evolve faster than previous tech revolutions, requiring adaptive product development.
-
•
Effective AI products use evals—automated tests on real data—to continuously measure quality.
-
•
Retrieval augmented generation and tool use are practical architectural tricks to reduce AI hallucination.
-
•
AI experiences must account for user trust despite AI’s confident but sometimes inaccurate outputs.
-
•
Human factors like worker anxiety about job security are critical to address in AI adoption.
-
•
Cross-disciplinary teams are essential in AI design due to the space’s novelty and uncertainty.
-
•
Prompt engineering skills grow best through extensive real use rather than frozen techniques.
-
•
Planning AI products with expectations of improved capabilities 6-9 months ahead optimizes outcomes.
-
•
Cultural acceptance of AI use inside organizations must be actively encouraged to avoid user hiding.
-
•
User testing of AI products still requires old-school observation combined with analytics and evals.
Notable Quotes
"You can’t have good AI without good information architecture."
"Everything’s changing really quickly—if you don’t revise your plan in a few months, you’re building for a world that’s already outdated."
"Hallucinations are when AI makes up stuff—it's a fundamental property of these models."
"Retrieval augmented generation is just adding results from a search into the model’s context to make it more grounded."
"Agents are models using tools in the loop, calling normal software functions as part of their process."
"People tend to trust AI a lot because it seems very confident—even when it’s not always accurate."
"Workers are often worried about layoffs and their value when AI speeds up their process, while executives like the efficiency gains."
"You should get good at prompt engineering by using AI a lot, not by memorizing outdated tricks."
"Building evals is like building tests to know if your AI system is good enough, and you can rerun them as models improve."
"User testing for AI still means watching people use it, plus collecting analytics and running evaluations."
Or choose a question:
More Videos
"Each team should talk about their core values like integrity, accountability, and respect to work together with the same mindset."
Jon Fukuda Amy Evans Ignacio Martinez Joe MeersmanThe Big Question about Innovation: A Panel Discussion
September 25, 2024
"Thinking about accessibility is really thinking about creating technology that works for all people all the time."
Sam ProulxAccessibility: An Opportunity to Innovate
March 9, 2022
"Design and research people must report to leaders who understand their functions, or else they get assigned irrelevant tasks like social coordinator."
Anna Avrekh Amy Jiménez Márquez Morgan C. Ramsey Catarina TsangDiversity In and For Design: Building Conscious Diversity in Design and Research
June 9, 2021
"We really try to push the idea that you shouldn't pretend to be hard of hearing; just say sorry, I didn’t get that."
Greg NudelmanDesigning Conversational Interfaces
November 14, 2019
"The components you design in Sketch or XD come with matching coded components in the app builder, so what you design is what you get in code."
George Abraham Stefan IvanovDesign Systems To-Go: Reimagining Developer Handoff, and Introducing App Builder (Part 2)
October 1, 2021
"Private mode in Miro lets you get the most variety of input by making contributions anonymous."
Shipra KayanMake your research synthesis speedy and more collaborative using a canvas
January 24, 2025
"Screen readers navigate based on the code and semantics, not on visual layout or appearance."
Sam ProulxDesigning For Screen Readers: Understanding the Mental Models and Techniques of Real Users
December 10, 2021
"We stuck with NPS because it was a pure metric with a long history we could trend, instead of muddying things with loyalty or revenue."
Shipra KayanHow we Built a VoC (Voice of the Customer) Practice at Upwork from the Ground Up
September 30, 2021
"The computer metaphor of mind does not do a very good job of explaining what we see people do with their bodies."
Dane DeSutterKeeping the Body in Mind: What Gestures and Embodied Actions Tell You That Users May Not
March 26, 2024
