Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

A Mixed Method Approach to Validity to Help Build Trust

Friday, April 28, 2023 • QuantQual Interest Group
Share the love for this talk
A Mixed Method Approach to Validity to Help Build Trust
Speakers: Chris Engledowl
Link:

Summary

Quantitative instruments are frequently sought because 1) they can be quickly fielded to lots and lots of people, and 2) when carefully sampled, they can be generalizable to the population of users/customers. However, because many times the focus is on speed to launch because decision-makers need results quickly, there is not much depth given to their development, nor an investigation of the validity evidence. In the session, I will share a framework that centers validity and is necessarily a mixed methods approach to research. I will also share ideas on how to scale the research over time so that findings and insights are able to be iteratively delivered to stakeholders, while also iteratively informing one another in a qual-quant research dance that brings more trustworthy, user-centered evidence to decision-makers. Finally, I will share ideas for a course I am developing for supporting qualitative researchers to become more mixed in their approach.

Key Insights

  • Validity involves five evidence sources: content, response processes, internal structure, relations to other variables, and consequences of testing.

  • Qualitative methods, especially cognitive interviews, are crucial for understanding how respondents interpret survey items, supporting validity.

  • Surveys should be treated as products that need ongoing iteration and testing, not one-off tools.

  • Ethical considerations extend beyond data privacy to how survey results affect user experience and product decisions.

  • Mixed methods approaches leverage both qualitative insights and quantitative analyses to build a stronger validity argument.

  • Breaking down survey validation work across multiple teams and 'bite-sized' efforts makes the process manageable.

  • Revising surveys over time to improve validity complicates measuring change longitudinally but increases trustworthiness.

  • Stakeholder buy-in improves when validity processes are communicated as phased insights offering tangible results quickly.

  • Analyses such as factor analysis and Rasch modeling reveal survey internal structure and help identify item bias across subpopulations.

  • It is important to revisit validity considerations continuously, especially after product or user base changes.

Notable Quotes

"Validity is the degree to which evidence and theory support the interpretations of test scores for proposed uses."

"Qualitative research in a mixed methods setting needs to think about validity to support the bigger validity argument."

"What would happen if you do think about validity? Would it change your process or research plans?"

"Surveys are products too, so they need to be iteratively tested."

"You don’t know what you don’t know. Let’s write surveys that cover those blind spots."

"If the survey wording changes, measuring change over time becomes difficult, but improving the survey builds trust."

"Ethical considerations should go beyond typical privacy reviews and think deeply about user impact throughout the process."

"It is better to partner with someone who has quantitative expertise to interpret internal structure analyses."

"Management and stakeholders often care more about getting usable information quickly than understanding the full validity process."

"We can get quick insights in phases to keep teams engaged and slowly build a complete validity argument."

Ask the Rosenbot
Spencer L. A. Stultz
Why Social Justice Frameworks are Necessary for Successful DEI/JEDI Initiatives
2023 • DesignOps Summit 2023
Gold
Aras Bilgen
Research Democratization: A Debate
2023 • Advancing Research 2023
Gold
Megan Blocker
Theme 2 Intro
2025 • Advancing Research 2025
Gold
Elena Naids
The Power of Difficult Conversations: A Case Study on How We Introduced Design Ops in the Federal Government Space
2023 • DesignOps Summit 2023
Gold
Prabhas Pokharel
Order and Chaos: New Ways of Collaborating on Synthesis and Storytelling
2022 • Advancing Research 2022
Gold
Peter Merholz
The 2025 State of UX/Design Organizational Health
2025 • Rosenfeld Community
Brendan Jarvis
Framing Tomorrow by Questioning Today
2022 • Design at Scale 2022
Gold
Johanna Kollmann
Insights-Driven Product Strategy: Get your Research to Count
2022 • Design in Product 2022
Gold
John Donmoyer
Shipping your code generation experiments to production
2025 • Designing with AI 2025
Gold
Cheryl Platz
Merging Improv with Design
2019 • Enterprise Community
Dave Malouf
The Future of DesignOps
2024 • DesignOps Summit 2020
Gold
Florence Okoye
AfroFuturism and UX Research
2023 • Advancing Research 2023
Gold
Tracy McGoldrick
IBM User Experience Program—The What, Why and How
2021 • Advancing Research Community
Sarah Brooks
Theme Three Intro
2022 • Civic Design 2022
Gold
Nicole Bergstrom
AccessibilityOps: Moving beyond “nice to have”
2024 • DesignOps Community
Jack Moffett
UX Metrics That Matter and The Future of our Design at Scale Conference: A Community Conversation
2022 • Enterprise Community

More Videos

Greg Petroff

"We have to be advocates for growth mindset and process improvement because people fall back on old ways even in times of change."

Greg Petroff

Software as Material—A Redux

June 6, 2023

Brendan Jarvis

"That night at 11:59 PM the entire nation of New Zealand would move into what was called a level 4 lockdown."

Brendan Jarvis

Framing Tomorrow by Questioning Today

June 8, 2022

Prerna Makanawala

"Achieving coherency across products leads to less training, fewer support calls, and more users successfully adopting products."

Prerna Makanawala

Achieving Balanced Design Consistency

June 9, 2021

Tricia Wang

"We are stuck in the user trap because these platforms are central to being human today."

Tricia Wang

From Users to Shapers of AI: The Future of Research

March 25, 2024

Cheryl Platz

"Pro-social gaming is about designing for positive social interactions that contribute to thriving communities."

Cheryl Platz

Embrace Your Fun Factor: Game Development Best Practices for Product Design

January 9, 2026

Ariel Kennan

"This field is still really emergent and growing momentum nationally and internationally."

Ariel Kennan

Theme Two Intro

November 17, 2022

Tricia Wang

"Building trust across functions means simple acts like scheduling lunch or coffee meetings with no agenda."

Tricia Wang

The most popular design thinking strategy is BS

January 27, 2022

Kristin Wisnewski

"Before, we never really looked holistically at decades of UX data—now we use it to be proactive and avoid duplicative studies."

Kristin Wisnewski

Measuring What Matters

October 23, 2019

Scott Jensen

"It became less important to keep doing fresh research and more important to grow the collective team knowledge."

Scott Jensen Sarah Delaney Carmen Liu

Short Take #2: UX/Product Lessons from Your Industry Peers

December 6, 2022