Generative AI case study: role-play training where opticians practise on an AI patient

The challenge: natural conversation and consistent feedback

The platform teaches opticians with courses, assessments and guided practice, and realistic practice is the most nuanced part. A consultation moves through stages: the greeting and first questions, understanding what the patient needs, recommending a product, closing, and sometimes handling a complaint. Useful feedback depends on knowing which stage the learner is in, but there is no script or explicit stage marker. Learners speak freely, skip a question or return to an earlier point, and the scoring still needs to treat each learner consistently across every scenario.

The simulated patient also needed to feel natural and varied. Each scenario called for a patient with a prescription, a budget and day-to-day needs, without sounding scripted or unfolding the same way for every learner. Voice added another consideration because both sides speak. A pause longer than 1 to 2 seconds could make the exchange feel unnatural and make interruptions or repeated phrases more likely.

Then there was the network. On a shaky connection a message could arrive twice, and about 2 to 5% of sessions ended up with duplicate turns in their history, which could move the detected stage and repeat a piece of feedback.

What we built: a generative AI patient, a stage classifier and a feedback model

Learners pick a role-play from a set of scenarios, and a firm with many practices runs all its learners in one place. OpenAI models play the patient, inside the prescription, budget and needs the scenario sets. They also read the dialogue as it happens and label each part with its phase (greeting, discovery, recommendation and the rest), so the system knows which stage the learner has reached. In common scenarios, they name the stage correctly 85 to 92% of the time.

The feedback comes from a separate model, working from a written scoring guide. A live coaching panel guides the learner during the session.

The patient's voice is ElevenLabs, streamed and buffered to keep the back-and-forth smooth in a live session. Every message carries a request ID, so one that's sent twice is stored once.

Around the models: a Next.js front end built with Radix UI and shadcn/ui components, Clerk for sign-in and roles (so access can differ by user and by practice), and Apollo Client talking GraphQL to a NestJS back end. PostgreSQL and Prisma keep the data and Redis caches it. Files, email and background work run on AWS (Lambda, S3 and SES) and on Azure Blob Storage.

Which model does which job, and why

Every call to a large language model costs money, so which LLM handles which step sets the monthly bill. A large, expensive model on every step pushed running costs up by about 30 to 50%. Small models alone would have cost less, but the feedback could have weakened, especially in the hard moments: a complaint, a tight budget, a choice between prescriptions. So the build splits the work. OpenAI models run the conversation and the stage detection, and the feedback goes to a separate model. That split is how it balances cost against quality.

The feedback model doesn't decide what a good consultation looks like. It applies the written guide, so every learner's feedback rests on the same criteria, whichever way their conversation went. And the feedback follows the stage, which is why a duplicated message mattered: it could move the stage label and repeat a piece of feedback the learner had already had.

Results

Sessions are shorter because learners spend less time restarting or wondering what to do next. Learners now take a full conversation from the greeting to the close and receive clear feedback on it. The support enquiries that fell most were the two the coaching and feedback now answer: what to do next, and how the scoring works. Stage detection, at 85 to 92% correct in common scenarios, has no before figure because the role-play itself is new.

Technologies and tools

Timeline

The build took six months.

If your own staff learn a conversation by practising it on each other, our page on LLM development services sets out how we choose a model for each job and work out the monthly bill before the build, and the one on conversational AI development services covers the voice side.