[Demo] How to re-categorize content at scale using LLMs
Summary
Large Language Models (LLMs) are to language as spreadsheets are to numbers: tools for modeling, exploration, and development. Among their many capabilities, LLMs can alleviate chores related to the design and implementation of information architectures. But doing so requires venturing beyond chat-based interfaces. In this brief demonstration, we'll see how to use OpenAI's API and a few open source command line tools to re-categorize content in a 1,000+ page website. The techniques demonstrated can be extended to other common content organization tasks.
Key Insights
-
•
Manual retagging of 1,200 blog posts would take about 10 hours, but leveraging GPT-4 reduced active human time to about 2 hours.
-
•
Using GPT-4 via command line and shell scripts enables automated tagging outside typical chat interfaces.
-
•
An organically grown taxonomy over 20 years contained unclear acronyms and inconsistent tag forms that GPT initially struggled with.
-
•
Cleaning and standardizing the taxonomy before prompting GPT is critical for effective AI assistance.
-
•
A review step of AI-suggested tags in CSV format allows human correction to avoid hallucinations entering production.
-
•
GPT-4 can propose new and useful tags outside the original taxonomy, enriching content classification.
-
•
The four-step GRU framework (Gather, Review, Update, Wrap up) balances automation with human oversight.
-
•
Storing blog content as markdown files simplifies integrating AI workflows via scripting and file manipulation.
-
•
The approach is adaptable and scalable to other CMS platforms by replacing scripting with API calls.
-
•
Taxonomies should use clear, unambiguous terms to improve both human and AI understanding.
Notable Quotes
"Some of the older content has discoverability problems, which is typical with blogs."
"Doing this tagging manually would have taken me around 10 hours of mind-numbing work."
"I’m actually using GPT-4, but not via the chat interface—I'm calling it from the Mac’s command line."
"I had to clean the taxonomy up because GPT wouldn’t know what to do with acronyms like TAOI."
"I save the proposed tags to a CSV file so I can preview and edit them before applying the changes."
"A middle review step prevents hallucinations from making it into the production site."
"GPT-4 functioned as an assistant not just in retagging but also in improving the taxonomy itself."
"The entire process took about three hours from start to finish, about a fifth of the manual time."
"Use clear and obvious terms in taxonomies—unusual acronyms won’t make sense to GPT or others."
"You need to review proposed changes before committing them to production, otherwise errors sneak in."
Or choose a question:
More Videos
"You can’t fail 90 times out of 10 without laughing at yourself to keep going."
Erin WeigelUX Lessons from running more than 1,200 A/B Tests
July 10, 2024
"To innovate smarter, you need to get access to the roadmap as early as possible and start research even when not asked for it."
Mike OrenWhy Pharmaceutical's Research Model Should Replace Design Thinking
March 28, 2023
"UX researchers should learn Python or R and how to leverage AI APIs to process unstructured data."
Lija Hogan Milan Mijatovic Sam Proulx Louis RosenfeldThree Years Out: Perspectives on the Near-Term Future of User Research
March 15, 2024
"If Charlie has been tampered with, Alice needs a clear way to be alerted that she shouldn't trust it."
Heidi TrostWhen AI Becomes the User’s Point Person—and Point of Failure
August 7, 2025
"Design thinking at IBM is not just for designers; it’s how we want all 380,000 people to think about problems."
Adam CutlerPeople + Places + Practices = Outcomes
June 8, 2016
"It’s good practice to offer one-on-one interactions with older adults due to their learning preferences."
Rittika BasuAge and Interfaces: Equipping Older Adults with Technological Tools
February 23, 2023
"Accessibility is more of a process than a result; it starts at the very beginning of product development with research including disabled users."
Saara Kamppari-MillerDesignOps for Inclusive Design and Accessibility
May 26, 2022
"It’s about us, the design ops practitioners, and what we need today to be better at our jobs."
Bria AlexanderTheme Two Intro
October 3, 2023
"Regular face time, honest conversations, and working sessions help keep connection and momentum with new practitioners."
Laine Riley Prokay Lisa GordonCarving a Path for Early Career DesignOps Practitioners
September 9, 2022