Development of a CDH-Specific Conversational Assistant Using RAG
In Pediatric and Lifespan Data Science, First International Conference (IPLDSC 2024), Anaheim, California. Revised Selected Papers, Springer CCIS vol. 2386.
Abstract
We develop and evaluate a medical conversational assistant that combines retrieval-augmented generation (RAG) with GPT-4 to answer questions about Congenital Diaphragmatic Hernia (CDH). With an initial knowledge base of 230 papers, the assistant correctly answered 19 of 100 expected questions internally and 67 of 258 questions in external evaluation by healthcare professionals, while establishing baseline behavior for declining out-of-scope questions. After expanding the knowledge base with 770 additional papers, accuracy improved substantially, to 91 of 100 questions internally and 224 of 258 in external re-evaluation after deployment.
Cite
@inproceedings{KlotzmanIPLDSC2025,
title = {Development of a CDH-Specific Conversational Assistant Using RAG},
author = {Klotzman, Vanessa and Lopes, Cristina V. and Schomberg, John and Armanyous, Yostina S. and Linden, Danielle and Ma, Iris},
booktitle = {Pediatric and Lifespan Data Science: First International Conference, IPLDSC 2024, Revised Selected Papers},
series = {Communications in Computer and Information Science},
volume = {2386},
publisher = {Springer},
year = {2025},
doi = {10.1007/978-3-031-88346-0_3},
url = {https://doi.org/10.1007/978-3-031-88346-0_3}
}