How accurate is the AI triage?
28 synthetic patient descriptions in Roman Urdu, English, Urdu, each with an expected SATS colour and department, run through the real pipeline (AI extraction → deterministic SATS rules → AI routing). Last run 10-Oct-2026, 1:26 am.
Triage colour accuracy
100%
28/28 cases completed
Under-triage rate
0%
Serious case marked less urgent: the error that matters
Over-triage rate
0%
Safe direction, costs queue time
Department: exact
96.4%
Acceptable: 100%
Median latency
6.0 s
Extraction + routing
Confusion matrix · expected × predicted
| R | O | Y | G | |
|---|---|---|---|---|
| RED | 1 | 0 | 0 | 0 |
| ORANGE | 0 | 11 | 0 | 0 |
| YELLOW | 0 | 0 | 5 | 0 |
| GREEN | 0 | 0 | 0 | 11 |
Diagonal = correct. Right of the diagonal = under-triage (dangerous). Left = over-triage (safe).
| Patient said | Expected | AI + SATS | Department |
|---|---|---|---|
“seenay mein dard hai jo baayen baazu tak ja raha hai, paseena bhi aa raha hai” Roman Urdu · age 55 · signs: chest pain | ORANGE · Very urgent | ORANGE · Very urgent | Cardiology |
“I had a fit about ten minutes ago and now I feel very drowsy and confused” English · age 16 · signs: seizure post ictal, reduced consciousness | ORANGE · Very urgent | ORANGE · Very urgent | Medical OPD |
“مجھے اچانک بولنے میں مشکل ہو رہی ہے اور دائیں بازو میں کمزوری ہے” Urdu · age 68 · signs: focal neurology acute | ORANGE · Very urgent | ORANGE · Very urgent | Medical OPD |
“saans bilkul nahi aa rahi, honth neelay ho rahe hain” Roman Urdu · age 60 · signs: not breathing | RED · Emergency | RED · Emergency | Emergency |
“Coughing up blood since yesterday and I have been losing weight for two months” English · age 40 · signs: coughing blood | ORANGE · Very urgent | ORANGE · Very urgent | Medical OPD |
“subah se ulti mein khoon aa raha hai” Roman Urdu · age 45 · signs: vomiting fresh blood | ORANGE · Very urgent | ORANGE · Very urgent | Surgical OPDexpected Medical OPD |
“I am seven months pregnant and I have strong pain in my stomach” English · age 28 · signs: pregnancy abdo, pain severe, abdominal pain | ORANGE · Very urgent | ORANGE · Very urgent | Gynae / Obs |
“do din se pait mein dard hai aur halka bukhar hai” Roman Urdu · age 30 · signs: abdominal pain | YELLOW · Urgent | YELLOW · Urgent | Medical OPD |
“Fell off my motorbike, my wrist is swollen and looks bent” English · age 22 · signs: fracture closed | YELLOW · Urgent | YELLOW · Urgent | Orthopaedics |
“تین دن سے بخار ہے اور جسم میں درد ہے” Urdu · age 30 | GREEN · Routine | GREEN · Routine | Medical OPD |
“Itchy rash on both arms for two weeks” English · age 22 | GREEN · Routine | GREEN · Routine | Dermatology |
“teen din se daant mein dard hai aur gaal thora sooj gaya hai” Roman Urdu · age 35 | GREEN · Routine | GREEN · Routine | Dental |
“Pain and discharge from my left ear for five days” English · age 19 | GREEN · Routine | GREEN · Routine | ENT |
“do din se aankh laal hai aur paani aa raha hai” Roman Urdu · age 27 | GREEN · Routine | GREEN · Routine | Eye |
“I have been feeling very low for a month, I can't sleep and have no interest in anything” English · age 33 | GREEN · Routine | GREEN · Routine | Psychiatry |
“sugar ki mareez hoon, sugar 450 aa rahi hai, ultiyan ho rahi hain aur saans tez chal rahi hai” Roman Urdu · age 50 · signs: diabetic ketosis, vomiting persistent | ORANGE · Very urgent | ORANGE · Very urgent | Medical OPD |
“He swallowed rat poison about half an hour ago” English · age 4 · signs: poisoning overdose | ORANGE · Very urgent | ORANGE · Very urgent | Paediatrics |
“haath par garam paani gir gaya, chhaala ban gaya hai” Roman Urdu · age 26 · signs: burn other | YELLOW · Urgent | YELLOW · Urgent | Surgical OPD |
“Knee pain for six months, worse when climbing stairs” English · age 58 | GREEN · Routine | GREEN · Routine | Orthopaedics |
“do hafton se periods bohat zyada aa rahe hain aur kamzori hai” Roman Urdu · age 38 | GREEN · Routine | GREEN · Routine | Gynae / Obs |
“There is a lump in my neck that has been growing for three months” English · age 45 | GREEN · Routine | GREEN · Routine | Surgical OPD |
“achanak sar mein shadeed dard shuru hua, zindagi ka sab se bura dard hai” Roman Urdu · age 42 · signs: pain severe | ORANGE · Very urgent | ORANGE · Very urgent | Medical OPD |
“Got an electric shock while fixing a wire, there is a burn on my hand” English · age 30 · signs: burn major | ORANGE · Very urgent | ORANGE · Very urgent | Surgical OPD |
“مجھے ایک ہفتے سے کھانسی ہے” Urdu · age 25 | GREEN · Routine | GREEN · Routine | Medical OPD |
“do din se dast aur ultiyan ho rahi hain, kuch khaa pee nahi raha” Roman Urdu · age 3 · signs: vomiting persistent | YELLOW · Urgent | YELLOW · Urgent | Paediatrics |
“I cut my hand with a kitchen knife, the bleeding stopped after I pressed on it” English · age 29 · signs: haemorrhage controlled | YELLOW · Urgent | YELLOW · Urgent | Surgical OPD |
“kabhi kabhi ghabrahat hoti hai aur dil tez dharakne lagta hai” Roman Urdu · age 31 | GREEN · Routine | GREEN · Routine | Cardiology |
“Severe chest tightness for the last hour and I can't breathe properly” English · age 62 · signs: chest pain, sob acute, pain severe | ORANGE · Very urgent | ORANGE · Very urgent | Cardiology |
Syndrome tagging: the AI feeding the early-warning network
44 hand-written complaints (Urdu, Roman Urdu, English), including look-alikes that must not be tagged: a fever alone, a cough without fever, a hepatitis follow-up. The measured recall is the default in the outbreak lead-time study.
Precision
93.1%
Recall
87.1%
Look-alikes left untagged
100%
Dengue-like recall (n=5)
60%
Stress test: 110 hard cases, text and real speech
A second, harder set written independently of the training data: red flags hidden in mild wording, negations and resolved symptoms, one-word complaints, typos and code-switching, relatives speaking, children, pregnancy and long stories. 30 of them were turned into real Urdu/English speech with Gemini TTS and sent through the voice path.
| System | Cases | Colour accuracy | Under-triage | Over-triage |
|---|---|---|---|---|
| Gemini pipeline (single reading) | 110 | 93.6% | 0.9% | 5.5% |
| Gemini + self-consistency (two readings, most urgent kept; shipped) | 110 | 92.7% | 0% | 7.3% |
| Priora Lite (offline model) | 110 | 68.2% | 12.7% | 19.1% |
| Voice: real synthesised speech | 30 | 100% | 0% | 0% |
hidden red flag
93% Gemini · 60% Lite · n=15
negation resolved
93% Gemini · 47% Lite · n=15
vague short
80% Gemini · 87% Lite · n=15
code switch typos
100% Gemini · 73% Lite · n=15
relative speaking
100% Gemini · 73% Lite · n=15
age edge
90% Gemini · 80% Lite · n=10
pregnancy
100% Gemini · 80% Lite · n=5
multi symptom
95% Gemini · 60% Lite · n=20
The two-reading check is live in the kiosk: in this test it caught the single under-triage (an electrical burn described as “he seems fine now”), at the cost of a little extra over-triage, which is the safe direction. Raw results: eval/hard-results.json.
Priora Lite: our own offline model
When the internet drops, Priora falls back to a model we trained ourselves: Gemini generated 5,176 labelled complaints in three languages, and we distilled them into a 18,000-feature character + word n-gram classifier that predicts SATS signs and the department in ~8 ms on a CPU, with no network. A red-flag phrase lexicon can only add urgency; the SATS engine still sets the colour and a nurse confirms. Details: ml/README.md.
| Test set | Model | Colour accuracy | Under-triage | Department |
|---|---|---|---|---|
| 28 hand-written cases | Gemini 2.5 Flash (online) | 100% | 0% | 100% acceptable |
| 28 hand-written cases | Priora Lite (offline) | 85.7% | 3.6% | 96.4% acceptable |
| 636 held-out synthetic | Priora Lite (offline) | 81.6% | 5.2% | 76.9% exact |
Offline is a safety net, not a replacement: results are provisional and nurse-confirmed. The lexicon was refined after an early miss on one hand-written case, so its score there is optimistic; the held-out set is the fairer measure.
Limitations: a small synthetic set written by the team, text input only (voice is tested manually), provisional triage without vitals. Reproduce with npm run eval.