Measuring What Matters
Summary
Research and analytics are the eyes and ears of DesignOps. Validating whether strategies are effective or ineffective allows course correction and adaptation. Identifying problems that require action is essential for facilitating collaboration and ensuring that team members are engaged. Prioritizing user needs is the heart of user experience and Agile development. None of this can be done well without an effective research program. We’ll share how we scale research efforts to collect feedback from thousands of users per month and track the impact of dozens of projects involving 120+ designers.
Key Insights
-
•
IBM’s CIO design team represents just 1% of the IT org yet supports UX for 385,000 employees across the whole company.
-
•
Over half of IBMers have been with the company less than five years, adding urgency to delivering great onboarding and daily experiences.
-
•
About 30% of IBM employees work remotely or offsite, increasing reliance on intuitive, well-integrated digital tools.
-
•
The team follows a mission to lead with design, engineer with excellence, deepen agile practices, and secure the enterprise.
-
•
Kristen’s team moved from reactive user research to building proactive, predictive capabilities by aggregating decades of experience data.
-
•
Measurement shifted from isolated metrics like NPS to a holistic scorecard including ease of use, capability, and goal completion.
-
•
A voice of the employee continuous feedback tool replaced large, infrequent surveys to capture real-time user sentiment.
-
•
Data-driven decisions led to major redesigns of IBM’s employee directory, help system, and corporate intranet, each achieving significant user appreciation.
-
•
Measurement can be challenging due to resistance and attempts to game survey responses, underscoring the need for data integrity.
-
•
Embracing design ops chaos and continuous iteration is key to maturing UX capability in a large enterprise organization like IBM.
Notable Quotes
"This circle represents the size of IBM, 385,000 IBMers. The CIO team is only 3% of that, and our design team is just 1% of the CIO."
"We have about 5,000 new IBMers arriving every month, with high expectations and low tolerance for poor experiences."
"The state of IT is a daily reflection of what the company thinks and feels about its employees."
"You can't just jump to the top of the pyramid like Maslow’s hierarchy—you have to consider all the underlying layers supporting the experience."
"Fletcher Prevend took us to the next level by reporting design and user research directly into the CIO office—a huge differentiator."
"Every minute spent struggling with an IT system is a minute not spent delivering value for IBM and our clients."
"Before, we never really looked holistically at decades of UX data—now we use it to be proactive and avoid duplicative studies."
"We want to send fewer mailers but make them more impactful, using A/B testing to optimize open and click rates."
"Don’t try to control the data to fit your narrative—the goal is to let insights guide your decisions, not justify them."
"We’re the voice of the employee—we’re arbiters of truth, defenders of experience, and sometimes validators against manipulation."
Or choose a question:
More Videos
"Meisner taught actors to respond truthfully in the moment, prioritizing presence over preconceived ideas."
Daniel GloydDesigning From the Inside Out: How Method Acting Can Inspire Design Research
February 12, 2026
"I just feel like you’re talking to robots in the system a lot. They don’t have any awareness of other people’s cultures or worldviews."
Deirdre Hirschtritt Cesar Paredes Marie PerrotResearch is Only as Good as the Relationships You Build
November 17, 2022
"I believe that maturity models could help define the strategy of the design team."
Benjamin RealMaturity Models: A Core Tool for Creating a DesignOps Strategy
October 1, 2021
"Go beyond climate, look at the personal systems you understand or want to solve to find your career motivation."
Max Gadney Andrea Petrucci Joshua Stehr Hannah WickesAssessing UX jobs for impact in climate
August 14, 2024
"Leading change in a century-old company is a very different challenge."
Lada GorlenkoTheme 1: Intro
January 8, 2024
"Play disrupts hierarchy, which is why it helps teams stuck in defensiveness to move forward."
Feyikemi AkinwolemiwaPlay to innovate: How curiosity and experimentation transform UX
March 11, 2026
"We’re obligated to treat each other with kindness and respect."
Bria AlexanderOpening Remarks
March 29, 2023
"If you can’t see a human in a video, how do you know the entire conversation wasn’t fabricated?"
Llewyn Paine[Demo] Deploying AI doppelgangers to de-identify user research recordings
June 5, 2024
"Junior engineers often resent having to create OKRs that feel disconnected from their day-to-day work."
Bria Alexander Benson Low Natalya Pemberton Stephanie GoldthorpeOKRs—Helpful or Harmful?
January 20, 2022
Latest Books All books
Dig deeper with the Rosenbot
How can barriers like childcare and internet access be addressed to promote equitable participation?
How can organizations ensure they engage people with direct lived experience rather than proxies?
What does the acquisition of User Interviews by UserTesting mean for participants and research ops teams?