Clinical Triage via Graph-Constrained LLMs
Integrating structured medical knowledge graphs with LLMs allows for more accurate identification of clinically relevant ED revisit patterns compared to standalone models.
Emergency department quality assurance efforts are often limited to a 1-to-14-day window because of the manual effort required to review patient records for missed diagnoses. A retrospective analysis of 99 diagnosis pairs demonstrates that while a raw LLM struggles to distinguish between clinically relevant and irrelevant revisit patterns, an algorithm using a Knowledge Graph Augmented (KGA) approach achieves a positive predictive value of 83% to 100% relative to clinician rater assessment. This approach moves beyond simple semantic probability to ground model decisions in established clinical causal pathways.
| Approach | Sensitivity / Precision Profile | Clinical Utility |
|---|---|---|
| Standard Manual Review | High specificity | Gold standard |
| Raw GPT-4 | High false-positive rate | Low |
| KGA (LLM + Knowledge Graph) | 83-100% PPV (vs clinicians) | Potentially high |
The Mechanism of Graph-Constrained Inference
Prompting an LLM like GPT-4 to identify whether an ED revisit warrants further assessment leads to excessive flagging. In the study, GPT-4 identified 94% of pairs as concerning, reflecting extreme sensitivity. Raw LLMs rely on semantic proximity, interpreting any two diagnoses as linked because most medical conditions share some linguistic or loose biological connection. The KGA pipeline addresses this by constraining the model's reasoning through a graph.
In this system, the model acts as an extraction engine that identifies diagnostic entities from unstructured clinical text. The system then forces these entities to map onto a predefined set of nodes in a clinical knowledge graph. The mapping process uses a strict filter; if the entity extracted by the LLM does not map to a recognized canonical node within the ontology, the system rejects the link. Once entities are mapped, the system traverses the graph to check for a documented edge—a defined clinical complication or causal relationship—between the two diagnoses. If no such path exists, the revisit is discarded as clinically irrelevant. By treating the clinical history as a deterministic graph-traversal problem, the KGA pipeline prevents the model from hallucinating associations based on statistical word co-occurrence.
Building for Clinical Reality
Reliability in this system hinges on the normalization of raw medical text. Because LLMs are prone to variation when extracting entities, the KGA architecture requires a high-fidelity mapping step to ensure that clinical concepts like "abdominal pain" consistently link to a specific node rather than an approximation. Engineers must maintain the graph structure rather than relying on prompt engineering. The current study shows that when the LLM is subordinate to the graph topology, its output remains anchored in medical reality, effectively curbing the over-sensitivity seen in standard LLM zero-shot classification.
We do not yet know how this architecture performs when scaling to the entire spectrum of primary care and specialty diagnoses, or how it handles the high levels of noise inherent in noisy, real-time clinical documentation. While the current results on the 99-pair set are promising, the long-term feasibility depends on automating the entity-to-node alignment process without sacrificing precision. Scaling the graph to cover millions of potential patient histories without introducing drift or gaps in clinical coverage remains an open technical challenge.