Designing AI-first products on top of a rapidly evolving technology
This video is featured in the AI and UX playlist.
Summary
Key Insights
-
•
AI models evolve faster than previous tech revolutions, requiring adaptive product development.
-
•
Effective AI products use evals—automated tests on real data—to continuously measure quality.
-
•
Retrieval augmented generation and tool use are practical architectural tricks to reduce AI hallucination.
-
•
AI experiences must account for user trust despite AI’s confident but sometimes inaccurate outputs.
-
•
Human factors like worker anxiety about job security are critical to address in AI adoption.
-
•
Cross-disciplinary teams are essential in AI design due to the space’s novelty and uncertainty.
-
•
Prompt engineering skills grow best through extensive real use rather than frozen techniques.
-
•
Planning AI products with expectations of improved capabilities 6-9 months ahead optimizes outcomes.
-
•
Cultural acceptance of AI use inside organizations must be actively encouraged to avoid user hiding.
-
•
User testing of AI products still requires old-school observation combined with analytics and evals.
Notable Quotes
"You can’t have good AI without good information architecture."
"Everything’s changing really quickly—if you don’t revise your plan in a few months, you’re building for a world that’s already outdated."
"Hallucinations are when AI makes up stuff—it's a fundamental property of these models."
"Retrieval augmented generation is just adding results from a search into the model’s context to make it more grounded."
"Agents are models using tools in the loop, calling normal software functions as part of their process."
"People tend to trust AI a lot because it seems very confident—even when it’s not always accurate."
"Workers are often worried about layoffs and their value when AI speeds up their process, while executives like the efficiency gains."
"You should get good at prompt engineering by using AI a lot, not by memorizing outdated tricks."
"Building evals is like building tests to know if your AI system is good enough, and you can rerun them as models improve."
"User testing for AI still means watching people use it, plus collecting analytics and running evaluations."
Or choose a question:
More Videos
"Nobody’s gonna listen to you people until you start speaking in the language of business."
Mark Interrante Harry MaxAI for Prioritization (3rd of 3 seminars)
July 11, 2024
"If you are not speaking to the CEO or C-executives about compliance and transformation, someone else will."
Patrizia BertiniThe (r)evolution of designOps: It’s Time to Think (really) BIG
September 11, 2025
"Speaking fluent language that translates into business discussions forces us to think and speak like a business owner as well as a designer."
Shan ShenTranslating UX Terms into Business Contexts
November 29, 2023
"Everyone has a plan until they get punched in the mouth, said Mike Tyson."
Adam ThomasSurvival Metrics – Making Change in a Fast, Data-Informed, and Politically Safe Way
December 6, 2022
"Research is inherently a silo-busting activity because customers experience the whole, not just parts divided by departments."
Robin BeersResearch as a Catalyst for Organizational Transformation
March 12, 2021
"Getting something accessible is a straight line; keeping something accessible requires process change."
Sheri Byrne-HaberAccessibility at Scale
June 9, 2021
"Playbacks are stakeholder-focused meetings that sell the work’s story and drive buy-in, different from critiques focused on craft."
Joseph MeersmanSweating the Pixel: Scaling Quality through Critique
June 10, 2021
"Go meet someone new at this conference and set up a coffee chat next week."
Deanna Washington Bria Alexander Jon Fukuda Saara Kamppari-Miller Farid SabitovConnecting the Ops: Plenary Panel and Closing Circle
September 9, 2022
"We have the responsibility to redistribute our power and include those who have a stake in the work we do."
Aaron Stienstra Lashanda HodgeLeveraging Civic Design to Advance Equity and Rebuild Trust in the US Federal Government
December 8, 2021
