Case Study · AI Systems

Reception System

An AI phone handler prototype for small businesses. Answers calls, extracts intent, books appointments - fine-tuned locally-runnable LLM, no per-call cloud fees. View the code →

Python FastAPI Faster Whisper spaCy NER Phi-3 / Llama 3.2 (QLoRA) Piper TTS Twilio UWE Enterprise Scholarship

99.8%

Action accuracy, fine-tuned (GPU eval)

86.7%

Accuracy, CPU-only, no GPU

111

Automated test checks

The Problem

Small clinics and SMEs miss calls constantly - receptionists are expensive, call-centre services are impersonal, and most AI phone tools run on a cloud subscription that doesn't suit businesses handling sensitive data. A dental clinic can't route patient calls through a third-party cloud LLM without GDPR implications.

The ask: answer calls, extract the reason, and book the appointment, with a model that can run on hardware the business already owns, no data leaving the building at inference time.

Architecture

Inbound call

→

Twilio captures the call and streams audio to the backend

Faster-Whisper

→

CPU-optimised STT transcribes the caller in real time

spaCy NER

→

Extracts DATE, TIME, SERVICE, PERSON entities from the transcript

Fine-tuned LLM

→

Phi-3 mini / Llama 3.2 3B, QLoRA fine-tuned on 600 synthetic call samples, outputs a structured action

Pydantic schema

→

Validates the action as one of 6 typed cases before anything is confirmed

Piper TTS

→

Reads the booked slot back to the caller

Key Challenges

→ Vanilla LLMs can't do this reliably. Phi-3 mini and Llama 3.2 3B scored 0-0.4% action accuracy out of the box. QLoRA fine-tuning on 600 synthetic call samples took that to 98.1% and 99.8%. Prompting alone wasn't close to enough.
→ GPU accuracy doesn't survive quantisation. Compressing the fine-tuned models to GGUF (Q3_K_M) for CPU-only inference drops accuracy to 66.7-86.7%, the real cost of removing the GPU dependency, not hidden in the headline number.
→ Hallucination-safe slot booking. LLMs will confidently invent appointment times. Every action is validated against a Pydantic discriminated union (6 typed cases) and the calendar store before anything is confirmed - the LLM proposes, the schema and calendar confirm.
→ Name capture is messier than it looks. Callers spell names out ("J-A-C-K R-E-A-C-H-E-R"), correct themselves mid-call, or get missed by regex entirely. Layered regex, spaCy PERSON fallback, and an explicit confirmation turn to catch the rest.

Outcome

A working prototype, not a live deployment: no paying client yet. Validated with 111 automated pipeline checks (booking, cancellations, edge cases, profanity handling) and benchmarked on both GPU (fine-tuned, 480 synthetic samples) and CPU-only (quantised, no GPU at inference) conditions. A separate app is currently collecting real caller audio to test against genuine speech before any commercial pilot.

Built under the UWE Enterprise Scholarship (£1,000 award), part of an MSc group project. Plan is to productise the online version as a SaaS for SMEs; the fully offline variant likely won't fit that subscription model given its zero-cloud design. Full code, eval results and test suite on GitHub →

← back to portfolio next: dissertation →