Generalizable Clinical Note Section Identification with Large Language Models

Publication date: Feb 19, 2024

Abstract Objective Clinical note section identification helps locate relevant information and could be beneficial for downstream tasks such as named entity recognition. But the traditional supervised methods suffer from transferability issues. This study proposes a new framework for using large language models for section identification to overcome the limitations. Materials and methods We framed section identification as question-answering and provided the section definitions in free-text. We evaluated multiple LLMs off-the-shelf without any training. We also fine-tune our LLMs to investigate how the size and the specificity of the fine-tuning dataset impacts model performance. Results GPT4 achieved the highest F1 score of 0.77. The best open-source model (Tulu2-70b) achieved 0.64 and is on par with GPT3.5 (ChatGPT). GPT4 is also found to obtain F1 scores greater than 0.9 for 9 out of the 27 (33%) section types and greater than 0.8 for 15 out of 27 (56%) section types. For our fine-tuned models, we found they plateaued with an increasing size of the general domain dataset. We also found that adding a reasonable amount of section identification examples is beneficial. Discussion These results indicate that GPT4 is nearly production-ready for section identification, and seemingly contains both knowledge of note structure and the ability to follow complex instructions, and the best current open-source LLM is catching up. Conclusion Our study shows that LLMs are promising for generalizable clinical note section identification. They have the potential to be further improved by adding section identification examples to the fine-tuning dataset.

Concepts Keywords
Beeferman Clinical
Impressing Doi
Radiology Gpt4
Https
International
Llm
Match
Medrxiv
Models
Note
Open
Org
Preprint
Section
Sections

Semantics

Type Source Name
gene GENE LARGE1
gene GENE BEST1
gene GENE JTB
gene GENE NR1I2
gene GENE PDC
gene GENE NINL
gene GENE AGRP
gene GENE ARTN
gene GENE SGSM3
gene GENE SET
gene GENE REST
gene GENE LRWD1
gene GENE LINC00273
gene GENE THOP1
gene GENE BPIFA4P
gene GENE SH2D3C
gene GENE CHAT
gene GENE GBP1P1
drug DRUGBANK Alpha-1-proteinase inhibitor
gene GENE SERPINF2
drug DRUGBANK Chlorhexadol
gene GENE TNFRSF11A
disease MESH allergies
drug DRUGBANK Gold
drug DRUGBANK Aspartame
gene GENE GPT
gene GENE DPAGT1
gene GENE SULT2A1
gene GENE ZAP70
gene GENE SMIM10L2A
gene GENE EHD1
gene GENE LAT2
gene GENE RASA1
gene GENE RGS6
drug DRUGBANK Medrysone
gene GENE CTSC
gene GENE LINC02381
drug DRUGBANK Coenzyme M
gene GENE KCNK3
gene GENE LINC01194
gene GENE TMED7
gene GENE SERPINB1
gene GENE COL9A2
gene GENE COMP
gene GENE COL9A3
gene GENE SCN8A
gene GENE COL9A1
gene GENE PROC
gene GENE INTU
gene GENE ELOVL6
gene GENE FANCE
gene GENE SLC29A1
gene GENE PLXNA1
gene GENE CCN3
gene GENE RPL10
gene GENE PTEN
gene GENE PLXNA2
gene GENE SHOX2
gene GENE NR1H2
gene GENE CEP55

(Visited 1 times, 1 visits today)

Leave a Comment

Your email address will not be published. Required fields are marked *