Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Making it Count: Developing a custom digital metric framework that works

Friday, October 15, 2021 • QuantQual Interest Group
Share the love for this talk
Making it Count: Developing a custom digital metric framework that works
Speakers: Alberto Ferreira
Link:

Summary

NPS, SUS, HEART, CSAT, CES, CLI—the list of ambiguous acronyms goes on and on. Companies are under more pressure than ever to measure and quantify their results and their interactions with customers, but finding the right metric and the right approach is a challenging process that risks leaving key factors behind as you commit to one sole standard. However, what if there is another way to measure digital transformation and how people see your services? This is what this presentation will focus on. You will learn how to navigate the pitfalls of standardized metrics—with their pros and cons—and learn how to build and implement a custom metric framework that incorporates the best aspects of Net Promoter Score, Customer Effort Score, System Usability Scale, and other, into one cohesive and modern whole aimed at developing the actionability and traceability of your operations and customer services. Attendees takeaways include: how to develop a custom framework without losing benchmarking capability, identifying gaps and needs from this framework that is not focused solely on operational numbers, and how to wade through the murky waters of digital experience quantification. Bring your best questions and leave with actionable insights that you can put in use immediately.

Key Insights

  • NPS often obscures diverse customer experiences by aggregating different user segments into a single score.

  • Custom metric frameworks combining behavioral, attitudinal, and quality-of-experience data yield richer insights than standardized metrics.

  • Separating attitudinal (feelings) and behavioral (actions) metrics avoids the risk of conflating different dimensions of user experience.

  • Quantitative data should be supplemented with qualitative verbatims analyzed via sentiment analysis for fuller context.

  • Business leadership prefers simple headline metrics, so complex multi-dimensional data must be distilled carefully, often accompanied by storytelling.

  • Existing UX metrics frameworks like NASA TLX, SUS, SuperQ, and Google HEART offer valuable components but may need adaptation for specific digital contexts.

  • Designing useful metrics starts by clarifying goals, then defining quantity, signals, timeframe, and scope explicitly.

  • High conversion rates alone can be misleading without considering total volume of conversions and user behaviors in context.

  • Customer sentiment is complex and may not align directly with retention or recommendation behavior, so metrics must capture nuance.

  • Short, targeted surveys on key user journeys help balance depth of insight with user willingness to respond.

Notable Quotes

"A user-centered measurement should not be one dimensional because people are complex."

"NPS scores can rise overall, yet new and existing customers might have completely different views that get obscured."

"A metric is usually the result of a relation between two measures, not interchangeable with a measure itself."

"Customers tend to inflate negative feelings in surveys, but that doesn't always reflect their actual behavior."

"We aim to keep the reporting simple for stakeholders while incorporating multiple data points in the background."

"Combining behavioral and attitudinal data helps us understand not just what users do, but how they feel about it."

"We use sentiment analysis on verbatims to quantify positive, neutral, and negative customer feedback over time."

"Designing a metric starts with the goals, not the metric itself."

"Conversion rate alone isn’t sufficient; total conversions and traffic volume also matter."

"Including human stories alongside numbers helps business leaders grasp the complexity behind the metrics."

Ask the Rosenbot
Jacqui Frey
Flow and Superfluidity for Design Orgs
2018 • DesignOps Summit 2018
Gold
Louis Rosenfeld
Opening Remarks
2023 • Advancing Research 2023
Gold
Adel Du Toit
Get Your CFO To Say: 'Our Strategic Goal is User Obsession'
2022 • Design at Scale 2022
Gold
Kim Lenox
Leading Distributed Global Teams
2019 • Enterprise Community
Sohit Karol
Designing Delightful Listening Experiences: Mixed Methods Research in the Age of Machine Learning
2020 • Advancing Research 2020
Gold
Soma Ghosh
What emerging methods are advancing UX research [Advancing Research Community Workshop Series]
2023 • Advancing Research Community
Phil Gilbert
A Consistent Culture of Design
2015 • Enterprise UX 2015
Gold
Nina Jurcic
The Design System Rollercoaster: From Enabler and Bottleneck to Catalyst for Change
2023 • DesignOps Summit 2023
Gold
Helen Armstrong
Augment the Human. Interrogate the System.
2023 • Enterprise UX 2023
Gold
Kate Koch
Flex Your Super Powers: When a Design Ops Team Scales to Power CX
2021 • DesignOps Summit 2021
Gold
Lada Gorlenko
Theme 1: Discussion
2024 • Enterprise Experience 2020
Gold
Joseph Meersman
Sweating the Pixel: Scaling Quality through Critique
2021 • Design at Scale 2021
Gold
Megan Clegg
Space for Everyone: Reframing Accessibility Through a Wider Lens
2021 • Design at Scale 2021
Gold
Marc Majers
Interrupted UX - Add A Dose of Reality To Usability Testing
2022 • Advancing Research 2022
Gold
Craig Brookes
"Just Make it Look Good" and Other Ways We're Misunderstood
2021 • Design at Scale 2021
Gold
Kristin Sundermeyer
Design Ops Metrics
2021 • DesignOps Summit 2021
Gold

More Videos

Greg Petroff

"AI assistance in user research doesn’t supplant humans; it augments and assists by helping us sift through large volumes of data."

Greg Petroff

Software as Material—A Redux

June 6, 2023

Brendan Jarvis

"Who do you see? Who do you not see? Why do you not see them?"

Brendan Jarvis

Framing Tomorrow by Questioning Today

June 8, 2022

Prerna Makanawala

"Work with majority wins but don’t alienate minority teams; everyone has to sometimes compromise for the greater good."

Prerna Makanawala

Achieving Balanced Design Consistency

June 9, 2021

Tricia Wang

"In Web 2.0, a small group sets the rules and most people just play within them."

Tricia Wang

From Users to Shapers of AI: The Future of Research

March 25, 2024

Cheryl Platz

"Pro-social gaming is about designing for positive social interactions that contribute to thriving communities."

Cheryl Platz

Embrace Your Fun Factor: Game Development Best Practices for Product Design

January 9, 2026

Ariel Kennan

"When I started, there were a few practitioners in the US, a small community, and now I see this growing network and community of practice."

Ariel Kennan

Theme Two Intro

November 17, 2022

Tricia Wang

"It's not about going out to communities but changing your team and leadership to genuinely reflect those communities."

Tricia Wang

The most popular design thinking strategy is BS

January 27, 2022

Kristin Wisnewski

"Before, we never really looked holistically at decades of UX data—now we use it to be proactive and avoid duplicative studies."

Kristin Wisnewski

Measuring What Matters

October 23, 2019

Scott Jensen

"Both of our perspectives changed, and now we’re seeing that come to fruition through the product we’re building."

Scott Jensen Sarah Delaney Carmen Liu

Short Take #2: UX/Product Lessons from Your Industry Peers

December 6, 2022