Finding the unknown in the known: Harnessing meta-analysis and literature review
Summary
Key Insights
-
•
Revisiting existing research can save weeks or months by building on previously gathered insights.
-
•
Literature reviews provide rich thematic understanding, while meta-analyses quantitatively aggregate findings to reveal robust patterns.
-
•
Thumbtack's Homeowner Playbook synthesized insights from over 85 studies to create a foundational, strategic resource that cultivates empathy and aligns teams.
-
•
Leadership buy-in and internal communications are crucial for ensuring organizational adoption and visibility of research syntheses.
-
•
Meta-analysis can answer new, high-impact questions that original studies were not designed to address.
-
•
The Nielsen Norman Group’s search meta-analysis revealed the 'pinball pattern' where users bounce between multiple search result elements instead of scanning linearly.
-
•
Synthesizing research requires careful handling of selection bias, overgeneralization, variable data formats, and time/resource constraints.
-
•
Creating living documents that are periodically updated keeps synthesized research relevant and impactful over time.
-
•
Collaborative brainstorming and structured databases help organize large volumes of internal research for effective synthesis.
-
•
Even small efforts to revisit and synthesize research can significantly elevate team alignment and strategic decision making.
Notable Quotes
"The answers to our most pressing questions aren’t out there waiting to be discovered, but right here hidden in what we already know."
"Meta-analysis is like looking at the average Rotten Tomatoes score to get an overall sentiment from many individual reviews."
"A literature review pulls together different perspectives to paint a bigger picture, not just averages."
"Every piece of research we’ve conducted represents valuable context and insights that can still guide us today."
"The Homeowner Playbook helped align teams across product, marketing, and operations by giving everyone a shared research fact understanding."
"It doesn’t happen very often, but I’m speechless. I’m truly impressed by the clarity, structure, and impressive synthesis of the Homeowner Playbook."
"Meta-analysis reveals overarching patterns that might otherwise be hidden within isolated studies."
"Users now bounce between different elements on the page influenced by visually compelling features and keywords — what we call the pinball pattern."
"Selection bias, overgeneralization, data variability, and time constraints are common challenges to synthesis."
"Start small, revisit a past study, synthesize related findings, or take a step towards creating something like a playbook for your own organization."
Or choose a question:
More Videos
"Respondents who feel very connected are twice as likely to say they’re making a significant impact."
Marc Fonteijn Ru ButlerIncrease your confidence, influence, and impact (through a Professional Community)
December 3, 2024
"Design’s value is in asking meaningful questions and steering towards the right outcome, not getting bogged down in technical parameters."
Dem Gerolemou Alexis Oh Neef RehmanClimate technology fundamentals
May 15, 2024
"Doing at least two benchmarking rounds lets you see if optimizations have measurable impacts over time."
Daniela Magaña Flores Ariane Rahn Jeff Ephraim BanderAhead of Competition: Learn What UX Benchmarking Can Do for Your Business Today
March 10, 2022
"Think like a designer: approach how you build your team and scale your company’s product design practice like a design problem."
Abbey Smalley Sylas SouzaScaling UX Past the Size of Your Team
January 8, 2024
"Anything below zero on the design staffing score indicates we’re staffing lower priority projects over higher ones."
Patrick CommarfordDesign Staffing for Impact
January 8, 2024
"Engaging communities requires thinking beyond western-centric tools, considering language and cultural relevance for accessibility."
Xenia Adjoubei Sean BruceEmpowering Communities Through the Researcher in Residence Program
March 29, 2023
"Make it easy to do the right thing, and as a side effect, make it harder to do the wrong thing."
Craig VillamorDesign Systems for Ethical Design
January 26, 2023
"User research is really only part of the information schema that we’re talking about."
Jon Fukuda Ellie KryslDesign Planning and Management Support
October 3, 2023
"Our hypothesis was we could create better product faster with more eyes looking at the problem, not just product."
Iain McMaster IHan ChengDesign and Product: from Frenemy to Harmony
November 29, 2023
