Building the Rosenbot
Summary
- What does strategy look like in an AI world?
- What does eval-first mean?
- What the hell is going on deep inside an LLM?
- And what does all that mean for the future we want to build?
Key Insights
-
•
Generative AI is a new design material requiring fresh approaches distinct from prior UX methods.
-
•
Rose Bot leverages retrieval augmented generation to semantically search Rosenfeld’s extensive UX content.
-
•
Conversation logic beyond the LLM is essential to route, classify intents, and ensure safe responses.
-
•
Evaluation (eval) of AI is fundamentally challenging due to its non-deterministic input and output.
-
•
Effective evaluation requires combining human expert and end-user feedback alongside automated tools.
-
•
A third of a project’s budget and time should be dedicated solely to rigorous AI evaluation.
-
•
‘Prompt deep dive’ usability testing spends extended time on individual prompts to deeply understand interactions.
-
•
New tooling is emerging specifically for tracing conversations, prompt engineering, and observability in AI.
-
•
UX roles remain vital in AI development by inventing new research techniques and ensuring user-centered design.
-
•
Co-creation between users and technology defines how AI applications evolve and succeed or fail.
Notable Quotes
"When GPT came out, my kids immediately adopted it at school and couldn’t pry it from their dead hands."
"This AI stuff is a new design material, just like the internet or mobile was before."
"The Rose Bot has read everything—every piece of Rosenfeld’s knowledge—to help users access it."
"Building generative AI tools is not scary, it’s just very different from the last 20 years of building products."
"There’s a lot of steps behind the scenes to make conversations useful and safe."
"Eval is the engineering word for evaluation, but also what researchers and UX designers naturally do."
"Without proper evaluation, you’re just building a demo – demos are easy, quality production is hard."
"We invented the ‘prompt deep dive’ technique to spend lots of time on one prompt to deeply understand it."
"Co-creation between users and technology determines if this AI evolves to be useful or harmful."
"We should feel okay to throw away old assumptions and tooling and invent new techniques for this new world."
Or choose a question:
More Videos
"If you don’t feel connected and you’re standing alone, the challenge looks much bigger."
Marc Fonteijn Ru ButlerIncrease your confidence, influence, and impact (through a Professional Community)
December 3, 2024
"Design’s value is in asking meaningful questions and steering towards the right outcome, not getting bogged down in technical parameters."
Dem Gerolemou Alexis Oh Neef RehmanClimate technology fundamentals
May 15, 2024
"Users tend to rate a familiar site better than non-users, especially for Amazon, because they feel more secure and familiar."
Daniela Magaña Flores Ariane Rahn Jeff Ephraim BanderAhead of Competition: Learn What UX Benchmarking Can Do for Your Business Today
March 10, 2022
"Think like a designer: approach how you build your team and scale your company’s product design practice like a design problem."
Abbey Smalley Sylas SouzaScaling UX Past the Size of Your Team
January 8, 2024
"We created a tiers of support model so even low priority projects get some level of guidance, even if just a 45-minute consultation."
Patrick CommarfordDesign Staffing for Impact
January 8, 2024
"We were interested in supporting networks of self-organization and mutual aid formed between refugees and host communities."
Xenia Adjoubei Sean BruceEmpowering Communities Through the Researcher in Residence Program
March 29, 2023
"Designers need to speak truth to power and engage their creative abilities to materially change how and what is produced."
Craig VillamorDesign Systems for Ethical Design
January 26, 2023
"User research is really only part of the information schema that we’re talking about."
Jon Fukuda Ellie KryslDesign Planning and Management Support
October 3, 2023
"More frequent, open communication and direct conversations helped us avoid assumptions and deliver something usable quickly."
Iain McMaster IHan ChengDesign and Product: from Frenemy to Harmony
November 29, 2023
