[Demo] How to re-categorize content at scale using LLMs
Summary
Large Language Models (LLMs) are to language as spreadsheets are to numbers: tools for modeling, exploration, and development. Among their many capabilities, LLMs can alleviate chores related to the design and implementation of information architectures. But doing so requires venturing beyond chat-based interfaces. In this brief demonstration, we'll see how to use OpenAI's API and a few open source command line tools to re-categorize content in a 1,000+ page website. The techniques demonstrated can be extended to other common content organization tasks.
Key Insights
-
•
Manual retagging of 1,200 blog posts would take about 10 hours, but leveraging GPT-4 reduced active human time to about 2 hours.
-
•
Using GPT-4 via command line and shell scripts enables automated tagging outside typical chat interfaces.
-
•
An organically grown taxonomy over 20 years contained unclear acronyms and inconsistent tag forms that GPT initially struggled with.
-
•
Cleaning and standardizing the taxonomy before prompting GPT is critical for effective AI assistance.
-
•
A review step of AI-suggested tags in CSV format allows human correction to avoid hallucinations entering production.
-
•
GPT-4 can propose new and useful tags outside the original taxonomy, enriching content classification.
-
•
The four-step GRU framework (Gather, Review, Update, Wrap up) balances automation with human oversight.
-
•
Storing blog content as markdown files simplifies integrating AI workflows via scripting and file manipulation.
-
•
The approach is adaptable and scalable to other CMS platforms by replacing scripting with API calls.
-
•
Taxonomies should use clear, unambiguous terms to improve both human and AI understanding.
Notable Quotes
"Some of the older content has discoverability problems, which is typical with blogs."
"Doing this tagging manually would have taken me around 10 hours of mind-numbing work."
"I’m actually using GPT-4, but not via the chat interface—I'm calling it from the Mac’s command line."
"I had to clean the taxonomy up because GPT wouldn’t know what to do with acronyms like TAOI."
"I save the proposed tags to a CSV file so I can preview and edit them before applying the changes."
"A middle review step prevents hallucinations from making it into the production site."
"GPT-4 functioned as an assistant not just in retagging but also in improving the taxonomy itself."
"The entire process took about three hours from start to finish, about a fifth of the manual time."
"Use clear and obvious terms in taxonomies—unusual acronyms won’t make sense to GPT or others."
"You need to review proposed changes before committing them to production, otherwise errors sneak in."
Or choose a question:
More Videos
"Relinquishing control over implementation empowered community-driven solutions and fostered lasting trust."
Alexia Cohen Adriane AckermanIncreasing Health Equity and Improving the Service Experience for Under-Served Latine Communities in Arizona
December 4, 2024
"You want to separate out those distractions from the actual tasks."
Marc Majers Tony TurnerInterrupted UX - Add A Dose of Reality To Usability Testing
March 11, 2022
"Design is a third way of knowing, different from science and humanities, involving making and artifice."
Jorge ArangoDesign as an Antidote to VUCA
May 9, 2019
"It matters what you build, but it matters more if you learn."
Brenna FallonLearning Over Outcomes
October 24, 2019
"Psychological warmth and physical warmth activate the same part of the brain, the insula."
Daniel GloydWarming the User Experience: Lessons from America's first and most radical human-centered designers
May 9, 2024
"Involving people with disabilities from the start reveals impact and usability issues that automated testing simply cannot catch."
Sam ProulxOnline Shopping: Designing an Accessible Experience
October 3, 2023
"Personalization is not just a trend; it’s a necessity."
Kristin SkinnerFive Years of DesignOps
September 29, 2021
"Racism doesn’t just hurt people like me; you’re hurting yourself by perpetuating beliefs that deny a greater humanity."
Denise Jacobs Nancy Douyon Renee Reid Lisa WelchmanInteractive Keynote: Social Change by Design
January 8, 2024
"Sometimes the screening can be done incorrectly by panels, which leads to wasted time and potential data issues."
Roberta Dombrowski Lianna Aduana5 Reasons to Bring your Recruiting in House
September 30, 2021
Latest Books All books
Dig deeper with the Rosenbot
What does a successful healthcare UX career look like in terms of accumulating influence and aligning with clinical/business goals?
How can AI assist in vision work without replacing the need for strategic direction?
What strategies help speed up the procurement and legal review of UX research platforms in organizations like LinkedIn?