Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
A Mixed Method Approach to Validity to Help Build Trust
Summary
Key Insights
-
•
Validity involves five evidence sources: content, response processes, internal structure, relations to other variables, and consequences of testing.
-
•
Qualitative methods, especially cognitive interviews, are crucial for understanding how respondents interpret survey items, supporting validity.
-
•
Surveys should be treated as products that need ongoing iteration and testing, not one-off tools.
-
•
Ethical considerations extend beyond data privacy to how survey results affect user experience and product decisions.
-
•
Mixed methods approaches leverage both qualitative insights and quantitative analyses to build a stronger validity argument.
-
•
Breaking down survey validation work across multiple teams and 'bite-sized' efforts makes the process manageable.
-
•
Revising surveys over time to improve validity complicates measuring change longitudinally but increases trustworthiness.
-
•
Stakeholder buy-in improves when validity processes are communicated as phased insights offering tangible results quickly.
-
•
Analyses such as factor analysis and Rasch modeling reveal survey internal structure and help identify item bias across subpopulations.
-
•
It is important to revisit validity considerations continuously, especially after product or user base changes.
Notable Quotes
"Validity is the degree to which evidence and theory support the interpretations of test scores for proposed uses."
"Qualitative research in a mixed methods setting needs to think about validity to support the bigger validity argument."
"What would happen if you do think about validity? Would it change your process or research plans?"
"Surveys are products too, so they need to be iteratively tested."
"You don’t know what you don’t know. Let’s write surveys that cover those blind spots."
"If the survey wording changes, measuring change over time becomes difficult, but improving the survey builds trust."
"Ethical considerations should go beyond typical privacy reviews and think deeply about user impact throughout the process."
"It is better to partner with someone who has quantitative expertise to interpret internal structure analyses."
"Management and stakeholders often care more about getting usable information quickly than understanding the full validity process."
"We can get quick insights in phases to keep teams engaged and slowly build a complete validity argument."
Or choose a question:
More Videos
"Prioritization work is a political process, and politics are hard to do in technology."
Mark Interrante Harry MaxAI for Prioritization (3rd of 3 seminars)
July 11, 2024
"Ethics is fantastic, but for the business, the right thing sometimes is revenue."
Patrizia BertiniThe (r)evolution of designOps: It’s Time to Think (really) BIG
September 11, 2025
"You want to be strict in what you emit but liberal in what you accept, to be clear and precise but also give some grace."
Shan ShenTranslating UX Terms into Business Contexts
November 29, 2023
"60 to 90% of our ideas do not improve the metrics they were intended to improve."
Adam ThomasSurvival Metrics – Making Change in a Fast, Data-Informed, and Politically Safe Way
December 6, 2022
"Turning the research lens on both the ‘in here’ organization and the ‘out there’ customer reality helps reveal misalignments and opportunities."
Robin BeersResearch as a Catalyst for Organizational Transformation
March 12, 2021
"Getting something accessible is a straight line; keeping something accessible requires process change."
Sheri Byrne-HaberAccessibility at Scale
June 9, 2021
"To me, sweating the pixel means paying attention to the details of interactions and keeping our eye on quality as designers."
Joseph MeersmanSweating the Pixel: Scaling Quality through Critique
June 10, 2021
"The next big thing will be knowledge management powered by new no-code and integrative tools."
Deanna Washington Bria Alexander Jon Fukuda Saara Kamppari-Miller Farid SabitovConnecting the Ops: Plenary Panel and Closing Circle
September 9, 2022
"Customer experience cannot truly be good if some are left out or excluded, either intentionally or unintentionally."
Aaron Stienstra Lashanda HodgeLeveraging Civic Design to Advance Equity and Rebuild Trust in the US Federal Government
December 8, 2021
