What It Takes to Build Compact AI for ED Triage: R&D Project by ScienceSoft

Editor’s note: Vadim Belski and Kate Lukina, MD, present ScienceSoft’s R&D project on building a custom AI system for emergency department triage and explain three architecture patterns that proved promising. The article also links to the published model files and browser demos. If you are exploring a similar clinical AI system, you can consult the authors or ScienceSoft’s healthcare AI development experts.

Why Emergency Triage May Favor Compact Models Over LLMs

In emergency departments (EDs), triage nurses must assign each patient an Emergency Severity Index (ESI) level within minutes to determine urgency. Level 1 signals immediate lifesaving intervention, level 2 covers high-risk presentations, and levels 3–5 reflect stable patients who can wait (depending on the expected number of tests, procedures, and consultations). To support this time-critical decision, an AI system can provide a preliminary ESI assessment based on the information collected at patient intake.

A common assumption is that the simplest way to generate this preliminary recommendation is using an LLM. However, despite performing well on broad medical reasoning benchmarks, generic LLMs were not trained specifically for ESI triage. In our zero-shot research evaluation, none of the GPT-4-class systems (such as GPT-4o or Claude 3.5 Sonnet) achieved more than 55% accuracy in matching expert-assigned ESI levels.

An even bigger problem with LLMs is that they are mostly not suitable for local deployment. Processing PHI with a cloud-based commercial LLM requires the hospital to sign a HIPAA-compliant BAA with the cloud provider and maintain HIPAA security and risk management controls. Moreover, LLMs we tested returned responses in 1.5 to 4 seconds, which is too slow for real-time software operations.

Seeing this problem, our team launched an internal R&D project to test whether a compact custom AI model could provide accurate ESI recommendations from real ED notes, while running locally.

Training Data and Evaluation Sets

We based the project on MIMIC-IV-ED, a deidentified dataset of 425,087 ED visits from Beth Israel Deaconess Medical Center in Boston from 2011 to 2019. It included fields for the patient chief complaint and mode of arrival, along with vital signs, pain score, and the nurse-assigned ESI level.

We also used MIETIC, an LLM-generated corpus of 9,629 narrative cases derived from MIMIC-IV-ED, which added 9,629 narrative cases derived from the same MIMIC records, as well as the MC-MED dataset from Stanford and ER-REASON from the University of California as auxiliary resources to expand the project data across institutions and documentation styles. However, the main evidence behind this project's results came from the single hospital, Beth Israel Deaconess Medical Center.

For model evaluation, we used two datasets intended to examine different model failure modes. Kate Lukina, MD, clinically reviewed and refined a 50-case expert evaluation set derived from synthetic case drafts. It emphasized incomplete note data, changing symptoms, and ambiguous ESI boundaries. We also selected a separate evaluation subset of 36 narrative cases from MIETIC to test performance on prose-style triage notes. These small sets were sufficient for our R&D, as our goal was to assess the system’s technical feasibility, not its clinical performance validation.

Why Fine-Tuning a Generalist Model Was Not Enough

We started with the most direct route and fine-tuned Qwen3.5-9B, a strong general-purpose small language model (SLM) that could run on a hospital's own infrastructure. In a similar pediatric ED study, Qwen2.5-7B reached 74% exact ESI agreement after fine-tuning on a specific hospital's data.

Yet our experiments quickly hit an accuracy ceiling at about 68% on the 36-case MIETIC narrative evaluation set. Further training adjustments shifted individual errors, yet they did not produce a stable improvement beyond that range. The results showed promise but fell short of the reliability required for ESI support, prompting us to test other approaches.

Three Feasible Design Patterns for ESI Decision Support

Compact Medical Encoder With Deterministic ESI Rules

The ESI methodology gave us a clue for the next experiment. Much of the final ESI assignment follows a published sequence of rules. This meant one model did not need to interpret the note and calculate the final ESI level in one step. We therefore separated clinical feature extraction from ESI calculation.

For note interpretation, we chose BiomedBERT, a 110-million-parameter text encoder pretrained on biomedical literature. Encoder models read text and fill predefined data fields instead of writing a free-form answer like generative models. In our pipeline, BiomedBERT converted ED notes into structured clinical features, such as symptoms, risk flags, and expected care resources. We then developed a Python rules engine based on the ESI Version 5 implementation handbook, which maps the extracted features to an ESI recommendation through a traceable rule path. For this formalized scoring stage, fixed deterministic logic was more consistent, testable, and traceable than an AI-generated ESI decision.

On the 50-case expert evaluation set, the pipeline reached 88.9% exact accuracy in predicting the reference expert-assigned ESI level, 97.2% accuracy within one ESI level, and 93% accuracy in identifying ESI 1, which represents the most urgent patients.

The design also offers practical advantages beyond accuracy. It can run locally, even on a mobile phone. In our tests, CPU inference took under 50 milliseconds. The visible rule path also makes the system easier to review, validate, and document for future regulatory assessments.

However, due to limited contextual reasoning, the encoder workflow lost 5–15 accuracy points when note details appeared in unfamiliar formats (e.g., abbreviations vs. full prose) and struggled with cases absent from training.

Generative Decoders With Structured Supervision and Clinically Weighted Rewards

Alongside the encoder work, we gave generative models another try, as they handled varied narrative notes better. Here, we refer to generative models as decoders because they produce a complete text response from an ED note. This time we used supervised fine-tuning (SFT) with Qwen3.5-9B, the 9-billion-parameter generalist, and MedGemma-4B, the 4-billion-parameter model that Google pre-trained on medical text and medical question-answer pairs. With SFT, they were trained to follow the expected ESI decision path before producing a final ESI recommendation.

For each training record, we paired a narrative note with a target output containing extracted clinical facts, ESI algorithm steps, and the final ESI prediction. This way, the models first reasoned through the ESI logic and only then selected a level, which improved both accuracy and decision traceability.

Sample Training Record for Supervised Fine-Tuning (SFT)

Structured supervision improved the model’s consistency, but error analysis revealed a more important issue. The SFT objective did not distinguish among ESI errors according to their clinical consequences. Under-triaging an ESI 1 case could delay immediate lifesaving care, while confusing ESI 4 and ESI 5 primarily changed the expected resource count. This became a turning point in the experiment. We added a reinforcement learning stage with clinically weighted rewards that penalized dangerous under-triage more heavily while also controlling unnecessary over-triage.

We applied the same two-stage training approach to both Qwen3.5-9B and MedGemma-4B. The smaller MedGemma-4B model produced the strongest decoder result on the 36-case MIETIC narrative evaluation set after both training stages. It achieved 83.3% exact accuracy, compared with 77.8% for Qwen3.5-9B and roughly 55% for the MedGemma-27B, the 27-billion-parameter LLM without triage-specific training. (Since the earlier BiomedBERT result came from a separate 50-case expert evaluation set, the percentages do not form a direct model ranking.)

The results suggest that task-specific training matters more than model size, while medical pre-training offers an additional advantage. However, the MedGemma model version used in this project remains limited to R&D use, so we plan to test MediPhi-PubMed as an alternative with more permissive licensing.

Confidence-Gated Hybrid Cascade

By then, our experiments had exposed a practical trade-off. The BiomedBERT pipeline offered low latency and a traceable rule path, but it remained sensitive to ED note format. MedGemma could analyze varied narratives that challenged the encoder, although it required more compute and offered weaker traceability on its own. We therefore combined both approaches in a confidence-gated cascade.

We kept BiomedBERT as the primary path because it delivered high exact accuracy with the lowest latency on familiar ED note formats. Meanwhile, MedGemma handled cases where the encoder showed low confidence.

When encoder confidence fell below the threshold, MedGemma analyzed the ED note and extracted its own set of clinical features. The rules engine then combined the features from both models, applied the ESI algorithm, and produced the final ESI recommendation. MedGemma’s ESI proposal did not set the final triage level, so each recommendation remained traceable to the ESI rules applied by the engine.

Because the earlier encoder and decoder results came from different evaluation sets, we assessed the cascade on both of them. These tests showed that selective routing preserved the strongest component result on each set without increasing overall exact accuracy. On the 50-case expert evaluation set, the cascade routed 15–25% of cases to MedGemma and retained the encoder pipeline’s 88.9% exact accuracy. On the 36-case MIETIC narrative set, MedGemma use rose to about 60%, while exact accuracy matched the decoder’s 83.3% result.

The prototype processed encoder-only cases in under 50 milliseconds and decoder-assisted cases in about two seconds. Thus, the hybrid architecture extended high-accuracy coverage to familiar, unfamiliar, and rare narratives while keeping each case on the lowest-latency path appropriate for its complexity. The complete system ran locally on one NVIDIA GB10 workstation, which kept triage notes away from cloud services. The modular design also allows BiomedBERT, MedGemma, and the rules engine to be updated separately.

Public Models, Demos, and Research Materials

• BiomedBERT v42 adapted for the hybrid cascade, with model documentation and evaluation results.
• BiomedBERT v85, the latest standalone safety-first encoder version that prioritizes ESI-1 recall and harmful under-triage rates over overall exact accuracy.
• Latest MedGemma-4B version.
• Browser demos for BiomedBERT v62 and for Qwen3.5-9B vs. MedGemma-4B, where readers can enter a triage note, review the proposed ESI level, and inspect the supporting clinical factors.
• Latest Qwen3.5-9B version.
• Full research preprint that presents the project methods, experimental results, and limitations.

Usage note: These artifacts are intended for technical inspection and reproduction of the reported R&D results, not for clinical use.

What Clinical Deployment Would Require

Our R&D models demonstrate that a compact, auditable ED triage architecture is feasible, but they cannot support clinical deployment. The core training datasets carry research-use restrictions. Besides, most of the project evidence comes from data associated with one hospital, so the models reflect its patient mix, specialty coverage, documentation patterns, and care protocols.

Therefore, the project's main practical result is the pipeline for dataset preparation, model fine-tuning, and evaluation, rather than demo model weights. Turning this research pipeline into a deployable hospital system would involve four broad steps.

1. Hospital-specific model adaptation. We would need historical ED records from the target hospital for model retraining and calibration. They will help align the system with local documentation, patient populations, specialty coverage, and care practices.
2. Local clinical performance validation. The adapted system would first undergo retrospective testing on local ED cases outside the training set. A subsequent silent-mode assessment would record AI recommendations in live workflows without displaying them to clinical staff. These tests would measure system reliability and clinically significant under-triage among the most urgent patients.
3. Integration into the triage workflow. A limited pilot would place AI recommendations at a defined point within the hospital’s ED triage process, with clinical staff retaining final responsibility. We would monitor overrides, response time, and signs of automation bias.
4. Regulatory assessment and authorization. Emergency triage is time-critical, so medical staff may lack time to review the basis for an AI recommendation before they act. This intended use is likely to bring the software under medical device oversight and require premarket authorization before clinical deployment.

Together, these milestones establish the local evidence, clinical safeguards, and regulatory basis needed to move beyond an R&D prototype.