Building the Rosenbot
Summary
A 30-minute deep-dive into the building of the Rosenbot. We’ll get both hands-on practical and likely a bit philosophical. What does it take to build a useful AI assistant? What does it mean for a business, strategically? And how do we make sure we are building the future that we want, while doing all this? Take-aways: What does strategy look like in an AI world? What does eval-first mean? What the hell is going on deep inside an LLM? And what does all that mean for the future we want to build?
Key Insights
-
•
Generative AI is a new design material requiring fresh approaches distinct from prior UX methods.
-
•
Rose Bot leverages retrieval augmented generation to semantically search Rosenfeld’s extensive UX content.
-
•
Conversation logic beyond the LLM is essential to route, classify intents, and ensure safe responses.
-
•
Evaluation (eval) of AI is fundamentally challenging due to its non-deterministic input and output.
-
•
Effective evaluation requires combining human expert and end-user feedback alongside automated tools.
-
•
A third of a project’s budget and time should be dedicated solely to rigorous AI evaluation.
-
•
‘Prompt deep dive’ usability testing spends extended time on individual prompts to deeply understand interactions.
-
•
New tooling is emerging specifically for tracing conversations, prompt engineering, and observability in AI.
-
•
UX roles remain vital in AI development by inventing new research techniques and ensuring user-centered design.
-
•
Co-creation between users and technology defines how AI applications evolve and succeed or fail.
Notable Quotes
"When GPT came out, my kids immediately adopted it at school and couldn’t pry it from their dead hands."
"This AI stuff is a new design material, just like the internet or mobile was before."
"The Rose Bot has read everything—every piece of Rosenfeld’s knowledge—to help users access it."
"Building generative AI tools is not scary, it’s just very different from the last 20 years of building products."
"There’s a lot of steps behind the scenes to make conversations useful and safe."
"Eval is the engineering word for evaluation, but also what researchers and UX designers naturally do."
"Without proper evaluation, you’re just building a demo – demos are easy, quality production is hard."
"We invented the ‘prompt deep dive’ technique to spend lots of time on one prompt to deeply understand it."
"Co-creation between users and technology determines if this AI evolves to be useful or harmful."
"We should feel okay to throw away old assumptions and tooling and invent new techniques for this new world."
Or choose a question:
More Videos
"If a dialog pops up but the keyboard focus isn’t moved into it, a screen reader user might not know it’s there."
Sam ProulxEverything You Ever Wanted to Know About Screen Readers
June 11, 2021
"Sponsor sessions are completely free and offer content just as valuable as the main program sessions."
Bria AlexanderOpening Remarks
November 17, 2022
"If you were blindsided recently, I’ve been there. I just really appreciated a kind voice because I didn’t hear a single word they said."
Corey Nelson Amy SanteeLayoffs
November 15, 2022
"Whenever you make a process model or service blueprint, make a wrong model first, then let the experts correct it to build shared understanding."
Milan GuentherA Shared Language for Co-Creating Ambitious Endeavours
June 6, 2023
"People will talk to customers whether you want them to or not. The question is how to make it a better experience."
Erin May Roberta Dombrowski Laura Oxenfeld Brooke HintonDistributed, Democratized, Decentralized: Finding a Research Model to Support Your Org
March 10, 2022
"When blind users choose between Android and iPhone, they weigh trade-offs between stability and customizability."
Sam ProulxUnderstanding Screen Readers on Mobile: How And Why to Learn from Native Users
June 6, 2023
"We experimented with AI-powered sprints, but had lots of dead ends and false starts before finding what really works for us."
Mujtaba HameedThe new horizon of ethnography: using AI to unlock the full potential of in-person research
March 11, 2026
"Ethical technology governance means anticipating the long-term social impacts of technology today and acting to protect essential public goods."
Ilana LipsettAnticipating Risk, Regulating Tech: A Playbook for Ethical Technology Governance
December 10, 2021
"You own the design system, which gives you the unique ability to integrate that accessibility thinking into all of your components."
Samuel ProulxFrom Standards to Innovation: Why Inclusive Design Wins
September 10, 2025
