Customer Survey Analysis Using LLMs — Automating UX Research at Scale
The client runs a platform that turns UXR-designed customer and user surveys into product insight — surfacing what users actually think about specific features. When we came in, that analysis was largely manual; our work was to help them transition to automated analysis powered by LLMs and NLP. The engagement ran from January 2023 to January 2026, and over those three years I led the team through it — starting with 2 engineers and growing to 8+ — managing delivery across a series of customer projects rather than a single build. A number of the features our team built are now live in the platform.
Automated survey summary & analysis
The foundation was automated survey summary and analysis. Because survey questions vary widely, we first classify each question into one of the fixed analysis types set by the client using a fine-tuned model. Then we apply a prompt tailored to each analysis type — one that pulls out the important patterns across all user responses and, critically, tags each pattern back to the specific responses it came from, so the insight stays traceable rather than becoming an unsourced summary. For each question we also generate a high-level summary and map summary phrases back to the underlying responses in the same way — so the final UX survey report is fully backed by the facts it came from.
Live interviews, AI interviewer & mission-level RAG
From there the scope broadened into several complementary capabilities. We built a live interview analysis feature that produces a bullet-point summary of an entire interview session, using a custom prompt adapted from RAG-style techniques. We built an AI Interviewer chatbot that, given an interview goal, asks follow-up questions dynamically based on what the user has just said — making automated interviews feel far more natural and responsive than a fixed script. And we designed and shipped a RAG-based Q&A layer, now live in the platform, that lets users ask questions across an entire mission — multiple related surveys — carrying prior context forward to return quick, grounded answers.