Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing

Thursday, April 4, 2024 • Rosenfeld Community
Share the love for this talk
Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing
Speakers: Cheryl Platz
Link:

Summary

Did you know that you're probably designing multimodal experiences? Most designers today are working in a multimodal environment, but haven't been trained to make the most of the many capabilities the latest generations of devices provide. Your customers have a small universe of devices, and they now come with the ability to handle far more than traditional haptic input like keyboards and mice. Gestural input, from swipes to hand gestures in video calls to stylus input, is becoming more common. Audio input and output are becoming more important in a world where digital assistants are poised to make a second surge powered by large language model AI. Visual output has become much more nuanced, and sometimes spans multiple devices. How do you wrangle all of this, optimize for great experiences, and still keep the human at the center? By becoming more consciously aware of the different inputs and outputs we're working with - and the many ways these inputs and outputs include and exclude our customers - we can build more resilient, more inclusive, and more powerful next-generation experiences.

Key Insights

  • Most designers already practice multimodal design unintentionally by supporting multiple input and output modalities like voice, touch, and haptics.

  • Multimodal design requires orchestrating how different modalities and devices communicate and transition smoothly, rather than designing in isolation.

  • Users switch devices when modality limits push them to—for example, moving from phone to desktop due to input convenience.

  • Cheryl’s CROW framework (Character, Relationship, Objective, Where) provides a comprehensive lens to capture user context relevant for multimodal design.

  • Transitions between input and output modes, devices, and network connectivity represent critical failure points and require intentional design.

  • Proactive, context-aware communication and interruption management improve user experience during multimodal and cross-device interactions.

  • Multimodal design and service design are deeply intertwined, especially across multiple devices and touchpoints.

  • Adaptive experiences that allow users to choose modalities based on context, such as hands-free vs. touch, lead to more inclusive designs.

  • Physical remotes remain relevant because phones are not always accessible, have battery constraints, and because remotes offer lower distraction.

  • Empathy for stakeholders’ goals and framing multimodal design benefits in terms of business priorities helps gain buy-in.

Notable Quotes

"Multimodal design is the design practice you probably don’t realize you’re doing already."

"The future is multimodal because humans are multimodal."

"Customers move from one device to another often because the information density or proximity to the device changes."

"Character, Relationship, Objective, and Where gives us a storytelling shorthand for fully understanding user context."

"Transitions will make or break your cross-device multimodal experiences."

"We don’t want to list every single customer activity; we need to abstract behavior patterns to manage interruptions."

"If you are altering any form of self-expression, you need to be really careful."

"Multimodal design adds a layer of orchestration on top of modality-specific designs like voice or graphical UI."

"Physical remotes have a lower power draw and are often more purpose-driven than phones, which are sources of stress."

"We treat our stakeholders as customers to understand their scenarios and align on a shared north star principle."

Ask the Rosenbot
Himanshu Bharadwaj
If design had a heart
2026 • Rosenfeld Community
Shelby Switzer
Making Space for Community Knowledge-sharing in a Distributed World
2021 • Civic Design 2021
Gold
Louis Rosenfeld
Coffee with Lou
2024 • Rosenfeld Community
Kwabena Opoku
Methodological toolkit for unique research impact
2026 • Advancing Research 2026
Gold
Kat Vellos
Opener: The Other L Word
2024 • DesignOps Summit 2020
Gold
Lavrans Løvlie
Ask me anything – Authors of Service Design: From Insight to Implementation
2025 • Advancing Service Design 2025
Gold
Ryan Rumsey
Business Influence Without Losing Your Soul
2021 • Enterprise Community
Amy Brana Stuart
Rest in Peace Fly-in-fly-out Design
2022 • Design at Scale 2022
Gold
Robin Beers
Beyond Insights: Researchers as Organizational Change Catalysts
2024 • Advancing Research 2024
Gold
Caitlyn Hampton
Compass 101: Growing Your Career In A Startup World
2021 • Design at Scale 2021
Gold
Aras Bilgen
Research Democratization: A Debate
2023 • Advancing Research 2023
Gold
Jemma Ahmed
Theme Three Intro
2023 • Advancing Research 2023
Gold
Jemma Ahmed
Collaboration: learning from other fields beyond our own [Advancing Research Community Workshop Series]
2024 • Advancing Research Community
Raven Veal
Dark Metrics: Illuminating the Negative Impact of Digital Health Design
2021 • Advancing Research 2021
Gold
Cennydd Bowles
Day 1 Panel
2024 • Designing with AI 2024
Gold
Feyikemi Akinwolemiwa
Play to innovate: How curiosity and experimentation transform UX
2026 • Advancing Research 2026
Gold

More Videos

Jim Kalbach

"Having big ears means listening more to others than to yourself during a performance."

Jim Kalbach

Jazz Improvisation as a Model for Team Collaboration

November 6, 2017

Louis Rosenfeld

"We look to work with authors we like and enjoy being around because writing a book is a long, collaborative journey."

Louis Rosenfeld

Coffee with Lou: Should You Write a (UX) Book?

March 7, 2024

Catt Small

"You don’t have to become a manager or director to advance; becoming a principal IC role is equally influential."

Catt Small Micah Bennett Brian Carr Jessica Harllee

What's Next for ICs: Exploring Staff and Principal Designer Roles

February 22, 2024

Marieke McCloskey

"I think we as UX researchers are uniquely positioned to forge bonds with data teams because of our storytelling and cross-functional work."

Marieke McCloskey

User Science: Product Analytics & User Research

March 11, 2021

Llewyn Paine

"Blurring user faces loses context and doesn’t address voice privacy, making it an inadequate solution."

Llewyn Paine

[Demo] Deploying AI doppelgangers to de-identify user research recordings

June 5, 2024

Joshua Noble

"Synthetic controls create counterfactuals using historical data when no suitable control group exists."

Joshua Noble

Casual Inference

October 6, 2023

Sara Logel

"If the scale delivers bad news, we jump on and off to check; if it delivers good news, we accept it quickly."

Sara Logel

Your Colleagues are Your Users Too

March 29, 2023

Bria Alexander

"Learning is a marathon, not a sprint, especially in the context of online conferences."

Bria Alexander Louis Rosenfeld

Welcome

January 8, 2024

Sam Proulx

"Accessibility isn’t a checkbox you run automated tests for; you must involve people with disabilities to benchmark success."

Sam Proulx

Online Shopping: Designing an Accessible Experience

June 7, 2023