Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

A Mixed Method Approach to Validity to Help Build Trust

Friday, April 28, 2023 • QuantQual Interest Group
Share the love for this talk
A Mixed Method Approach to Validity to Help Build Trust
Speakers: Chris Engledowl
Link:

Summary

Quantitative instruments are frequently sought because 1) they can be quickly fielded to lots and lots of people, and 2) when carefully sampled, they can be generalizable to the population of users/customers. However, because many times the focus is on speed to launch because decision-makers need results quickly, there is not much depth given to their development, nor an investigation of the validity evidence. In the session, I will share a framework that centers validity and is necessarily a mixed methods approach to research. I will also share ideas on how to scale the research over time so that findings and insights are able to be iteratively delivered to stakeholders, while also iteratively informing one another in a qual-quant research dance that brings more trustworthy, user-centered evidence to decision-makers. Finally, I will share ideas for a course I am developing for supporting qualitative researchers to become more mixed in their approach.

Key Insights

  • Validity involves five evidence sources: content, response processes, internal structure, relations to other variables, and consequences of testing.

  • Qualitative methods, especially cognitive interviews, are crucial for understanding how respondents interpret survey items, supporting validity.

  • Surveys should be treated as products that need ongoing iteration and testing, not one-off tools.

  • Ethical considerations extend beyond data privacy to how survey results affect user experience and product decisions.

  • Mixed methods approaches leverage both qualitative insights and quantitative analyses to build a stronger validity argument.

  • Breaking down survey validation work across multiple teams and 'bite-sized' efforts makes the process manageable.

  • Revising surveys over time to improve validity complicates measuring change longitudinally but increases trustworthiness.

  • Stakeholder buy-in improves when validity processes are communicated as phased insights offering tangible results quickly.

  • Analyses such as factor analysis and Rasch modeling reveal survey internal structure and help identify item bias across subpopulations.

  • It is important to revisit validity considerations continuously, especially after product or user base changes.

Notable Quotes

"Validity is the degree to which evidence and theory support the interpretations of test scores for proposed uses."

"Qualitative research in a mixed methods setting needs to think about validity to support the bigger validity argument."

"What would happen if you do think about validity? Would it change your process or research plans?"

"Surveys are products too, so they need to be iteratively tested."

"You don’t know what you don’t know. Let’s write surveys that cover those blind spots."

"If the survey wording changes, measuring change over time becomes difficult, but improving the survey builds trust."

"Ethical considerations should go beyond typical privacy reviews and think deeply about user impact throughout the process."

"It is better to partner with someone who has quantitative expertise to interpret internal structure analyses."

"Management and stakeholders often care more about getting usable information quickly than understanding the full validity process."

"We can get quick insights in phases to keep teams engaged and slowly build a complete validity argument."

Ask the Rosenbot
Karen Pascoe
Developing Experience Teams and Talent in the Enterprise
2016 • Enterprise UX 2016
Gold
Bud Caddell
DesignOps Community Sensing Session
2021 • DesignOps Community
Caitlyn Hampton
Compass 101: Growing Your Career In A Startup World
2021 • Design at Scale 2021
Gold
Saara Kamppari-Miller
Theme Three Intro
2022 • DesignOps Summit 2022
Gold
Discussion
2017 • Enterprise Experience 2017
Gold
Sam Proulx
Prototype Reviews, People With Disabilities, and You
2021 • Civic Design 2021
Gold
Anat Fintzi
Delivering at Scale: Making Traction with Resistant Partners
2022 • Design at Scale 2022
Gold
Jose Coronado
From Zero to Hero
2022 • DesignOps Summit 2022
Gold
Louis Rosenfeld
About the Rosenverse
2026 • Rosenfeld Community
Amahra Spence
Designing for Liberation, Rehearsing Freedom
2022 • Civic Design 2022
Gold
Catt Small
What's Next for ICs: Exploring Staff and Principal Designer Roles
2024 • Rosenfeld Community
Tamara Hale
War Stories LIVE! Tamara Hale
2020 • Advancing Research 2020
Gold
Chui Chui Tan
Global insights: Embracing international and intercultural research for innovation
2025 • Advancing Research 2025
Gold
Saara Kamppari-Miller
DesignOps for Inclusive Design and Accessibility
2022 • DesignOps Community
Amelia Cole
Data-Prompted Interviews
2021 • QuantQual Interest Group
Michele Marut
Research Repositories Reconsidered
2019 • DesignOps Community

More Videos

Sheri Byrne-Haber

"Accessibility defects are bugs. If it’s a P1 blocking bug for an accessibility issue, treat it like any other P1 bug."

Sheri Byrne-Haber

Accessibility at Scale

June 9, 2021

Prayag Narula

"If you skip iterative testing, you risk building features nobody really wants."

Prayag Narula Hannah Hudson

Empowering Designers to do Good Research

March 11, 2022

Janelle Estes

"If you build something that's easy to use but not what people want, it doesn't matter."

Janelle Estes

UX Research Trends

January 28, 2021

Craig Brookes

"Style guides and embedding content systems into design systems help scale content without bottlenecks."

Craig Brookes Andreas Huebner Morgan Quinn

"Just Make it Look Good" and Other Ways We're Misunderstood

June 11, 2021

Marc Fonteijn

"Slow and intentional communication allows you to be fully present rather than distracted by constant notifications."

Marc Fonteijn Ru Butler

Increase your confidence, influence, and impact (through a Professional Community)

December 3, 2024

Kate Towsey

"Research ops allows me to indulge my geeky planning spreadsheety side as well as my people-loving supportive side — Saskia, Delivery."

Kate Towsey

The State of ResearchOps: More Than Just Theory

June 20, 2019

Alla Weinberg

"Sometimes the entire leadership team quitting at once is a big sign cultural safety is failing."

Alla Weinberg

Design Teams Need Psychological Safety: Here’s How to Create It

September 9, 2022

Shawna Hein

"We have over 30 contracting teams spread across roughly 10 vendors all working on parts of the veteran experience landscape."

Shawna Hein Kevin Hoffman

Create a Cohesive Civic Design Practice Across Agency, Vendors, and Contracts

November 17, 2022

Tricia Wang

"You have to look two or three steps ahead so you're not chasing what's right in front of you — avoid target fixation."

Tricia Wang

SCALE: Discussion

June 15, 2018