Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Hands-on AI #1: Let’s write your first AI eval
This video is featured in the Evals + Claude playlist.
Summary
In this talk, Peter Van Dijck of the helpful intelligence company will walk you through writing your first eval. You will learn the basic concepts and the tools, and write an eval together. This talk is hands on; you can follow along, and there will be plenty of time for questions.
You will go away with an understanding of the basic building blocks of AI evals, and with the confidence that you know how to write one. And more importantly, you’ll build some intuition, some product sense, around how the best AI products today are built, and how that can help you use them more effectively yourself.
Key Insights
-
•
Evals consist of a task, a golden dataset with known correct outputs, and an evaluator that measures correctness.
-
•
Manual AI prompt testing is slow and inconsistent; automated evals accelerate and scale evaluation.
-
•
UX and product teams can and should learn evals as a practical, non-technical skill.
-
•
Creating your own golden dataset is essential and cannot be outsourced or fully automated.
-
•
Models are fixed once trained; improvements happen by refining prompts and context design, not retraining the model.
-
•
Evaluations measure task performance, not the underlying model itself, allowing comparison across models.
-
•
Outputting a confidence score from models is unreliable due to lack of internal memory and inconsistent scale interpretation.
-
•
Biases are baked into models during training via evals used in post-training refinement.
-
•
LLMs can be used to judge other LLM outputs to evaluate tasks with non-binary answers.
-
•
Effective eval work requires collaboration across data analysts, engineers, subject matter experts, and UX/product teams.
Notable Quotes
"Evals are like a way to define what good looks like."
"The model was baked and once it’s baked, it does not learn again until they bake a new one."
"You need to be looking at the data. Nobody wants to, but that’s core work."
"Without a golden dataset, you have to build the golden dataset yourself."
"We’re not teaching the model anything; we’re improving our prompts and context."
"Confidence scores from the model are not a good idea because the model has no memory."
"Biases are baked in through the evals used during model training and post-training."
"LLMs judging other LLMs might sound crazy, but if you do it right, it works."
"Evals are a product and UX skill; learning them lets you make these systems do what you want."
"There is a large and growing capability overhang in these models we haven’t discovered yet."
Or choose a question:
More Videos
"Accessibility isn't about dumbing down interfaces; except for color, most guidelines do not require changing the look or behavior of components."
Sheri Byrne-HaberThe Importance of Accessible Design Systems
January 8, 2024
"Think like a designer: approach how you build your team and scale your company’s product design practice like a design problem."
Abbey Smalley Sylas SouzaScaling UX Past the Size of Your Team
January 8, 2024
"Large language models don’t think like us but have superpowers in pattern matching that help surface relationships in linked notes."
Jorge ArangoExploding the Notebook: How to Unlock the Power of Linked Notes (2nd of 3 seminars)
April 19, 2024
"We realized we didn’t have a well-articulated purpose or guiding principles for the types of customer experiences we aspired to deliver."
Sarah WilliamsA Framework for CX Transformation
June 11, 2021
"People who are struggling are constantly battling an unstable financial situation, which comes with a high cognitive and emotional energy cost."
Yasmine KhanChecking Bias and Listening to Financially Vulnerable Americans
March 30, 2020
"I like to think about individuals as cells, teams as organs, and organizations as organisms in an ecosystem."
Dave MaloufTheme 3: Introduction and Provocation
January 8, 2024
"Managing towards outputs means someone has decided what the solution is going to be and told you to make that thing."
Jeff GothelfWho does what by how much?
November 20, 2025
"Continuous feedback requires collecting responses tied to the same user across multiple channels with behavioral targeting."
Aleksandra Korczynska Caroline Jarrett Justyna ParmeeSurvey Tools
March 12, 2026
"Setting extra places at the table is important because people will join and leave meetings depending on the project phase."
Frank DuranPartnership Playbook: Lessons Learned in Effective Partnership
January 8, 2024
