Foundation models are machine learning models trained using predominantly self-supervised strategies on large datasets, capable of transforming the descriptive variables of a domain into a latent representation (embedding) that allows for subsequent specialization for a wide range of downstream tasks, particularly in the predictive field. These models are finding increasing use in the clinical setting, and it is particularly interesting to investigate the potential and limitations of their use in the analysis of longitudinal data contained in Electronic Health Records (EHR), characterized by heterogeneity, noise, and temporal complexity. This is the context for CLMBR-T-Base (Clinical Language-Model-Based Representations using Transformers), an autoregressive model developed at Stanford University, based on transformer architecture, pre-trained on EHRs harmonized according to the Observational Medical Outcomes Partnership (OMOP) data model, and designed to generate clinical embeddings capable of capturing temporal and semantic relationships between events. This study evaluates the applicability of CLMBR-T-Base in the Italian context, and in particular at the ICS Istituti Clinici Scientifici Maugeri in Pavia. The thesis explores the use of the model to represent data relating to 99,884 hospital admissions between 2012 and 2025. The embeddings generated are then used to estimate the length of hospital stay (LoS). Specifically, the representations generated by CLMBR-T-Base were integrated into supervised predictive models (Multilayer Perceptron (MLP) and Gradient Boosting (GB)) and compared with a traditional variable representation approach based on counts of occurrences of events of interest, such as the presence of procedures during hospitalization. Analysis of the latent structure using HDBSCAN clustering showed that the embeddings naturally organize hospitalizations into coherent groups, suggesting that the model learns interpretable clinical semantics that can potentially be reused in other tasks. In order to perform a detailed analysis, five predictive models were developed: an overall model trained on data from all departments, and three specific models dedicated to the surgery, oncology, and rehabilitation departments, respectively, and a complementary model including all other departments. The results demonstrate the feasibility and effectiveness of the embedding-based approach in capturing temporal and semantic information useful for predicting LoS, highlighting how performance varies depending on the department and the distribution of clinical data. The embeddings generated by CLMBR-T-Base have proven capable of encoding complex clinical relationships between events, promoting good generalization capabilities. Especially with regard to the overall model, the embedding representation allows for excellent predictions with an R² value of 0.705. However, the work also highlights the limitations related to the partial coverage of the CLMBR-T-Base vocabulary compared to the codes used in the Italian healthcare system and the lack of outpatient or follow-up information in the ICS Maugeri data. Overall, the work demonstrates the feasibility and effectiveness of using pre-trained foundation models to represent longitudinal clinical data and support complex predictive tasks, highlighting how their use can, with some adjustments, constitute a new paradigm for the creation of tools capable of supporting predictive and personalized medicine in hospitals.
I foundation models sono modelli di apprendimento automatico addestrati mediante strategie prevalentemente auto-supervisionate su dataset di grandi dimensioni, in grado di trasformare le variabili descrittive di un dominio in una rappresentazione latente (embedding) che ne consente la successiva specializzazione per un’ampia gamma di task downstream, in particolare in campo predittivo. Tali modelli stanno trovando un crescente impiego in ambito clinico, ed è particolarmente interessante investigare potenzialità e limiti del loro utilizzo nell’analisi dei dati longitudinali contenuti nelle Electronic Health Records (EHR), caratterizzate da eterogeneità, rumore e complessità temporale. In questo contesto si colloca CLMBR-T-Base (Clinical Language-Model-Based Representations using Transformers), un modello autoregressivo sviluppato presso la Stanford University, basato su architettura transformer, pre-addestrato su EHR armonizzate secondo il modello dati Observational Medical Outcomes Partnership (OMOP) e progettato per generare embeddings clinici capaci di catturare relazioni temporali e semantiche tra eventi. Il presente lavoro valuta l’applicabilità di CLMBR-T-Base nel contesto italiano, e in particolare presso l’ICS Istituti Clinici Scientifici Maugeri di Pavia. La tesi esplora l’utilizzo del modello per la rappresentazione dei dati relativi a 99.884 ricoveri avvenuti tra il 2012 e il 2025. Gli embeddings generati vengono poi impiegati per la stima della durata della degenza ospedaliera (Length of Stay, (LoS)). Nello specifico, le rappresentazioni generate da CLMBR-T-Base sono state integrate in modelli predittivi supervisionati (Multilayer Perceptron (MLP) e Gradient Boosting (GB)) e confrontate con un approccio di rappresentazione tradizionale delle variabili basato sui conteggi delle occorrenze degli eventi di interesse, come la presenza di procedure durante il ricovero. L’analisi della struttura latente mediante clustering HDBSCAN ha mostrato che gli embeddings organizzano naturalmente i ricoveri in gruppi coerenti, suggerendo che il modello apprende una semantica clinica interpretabile e potenzialmente riutilizzabile in altri task. Al fine di svolgere un’analisi dettagliata, sono stati sviluppati cinque modelli predittivi: un modello complessivo allenato sui dati provenienti da tutti i reparti, e tre modelli specifici dedicati rispettivamente ai reparti di chirurgia, oncologia, riabilitazione e un modello complementare comprendente tutti gli altri reparti. I risultati dimostrano la fattibilità e l’efficacia dell’approccio basato su embedding nel catturare informazione temporale e semantica utile alla predizione della LoS, evidenziando come le prestazioni varino in funzione del reparto e della distribuzione dei dati clinici. Gli embeddings generati da CLMBR-T-Base si sono dimostrati capaci di codificare relazioni cliniche complesse tra eventi, favorendo una buona capacità di generalizzazione. Soprattutto per quanto riguarda il modello complessivo, la rappresentazione embeddings consente di ottenere delle ottime predizioni con un valore di R² pari a 0.705. Il lavoro mette in luce, però, anche i limiti legati alla copertura parziale del vocabolario CLMBR-T-Base rispetto ai codici utilizzati nel sistema sanitario italiano e alla mancanza, nei dati di ICS Maugeri, di informazioni ambulatoriali o di follow-up. Nel complesso, il lavoro dimostra la fattibilità e l’efficacia dell’utilizzo di foundation models pre-addestrati per rappresentare dati clinici longitudinali e supportare compiti predittivi complessi, evidenziando come il loro impiego possa costituire, con alcuni accorgimenti, un nuovo paradigma per la realizzazione di strumenti capaci di sostenere la medicina predittiva e personalizzata nelle strutture ospedaliere.
Uso dei foundation models per la rappresentazione della storia clinica ospedaliera: applicazione per la predizione della lunghezza del ricovero
NEGRI, FRANCESCA
2024/2025
Abstract
Foundation models are machine learning models trained using predominantly self-supervised strategies on large datasets, capable of transforming the descriptive variables of a domain into a latent representation (embedding) that allows for subsequent specialization for a wide range of downstream tasks, particularly in the predictive field. These models are finding increasing use in the clinical setting, and it is particularly interesting to investigate the potential and limitations of their use in the analysis of longitudinal data contained in Electronic Health Records (EHR), characterized by heterogeneity, noise, and temporal complexity. This is the context for CLMBR-T-Base (Clinical Language-Model-Based Representations using Transformers), an autoregressive model developed at Stanford University, based on transformer architecture, pre-trained on EHRs harmonized according to the Observational Medical Outcomes Partnership (OMOP) data model, and designed to generate clinical embeddings capable of capturing temporal and semantic relationships between events. This study evaluates the applicability of CLMBR-T-Base in the Italian context, and in particular at the ICS Istituti Clinici Scientifici Maugeri in Pavia. The thesis explores the use of the model to represent data relating to 99,884 hospital admissions between 2012 and 2025. The embeddings generated are then used to estimate the length of hospital stay (LoS). Specifically, the representations generated by CLMBR-T-Base were integrated into supervised predictive models (Multilayer Perceptron (MLP) and Gradient Boosting (GB)) and compared with a traditional variable representation approach based on counts of occurrences of events of interest, such as the presence of procedures during hospitalization. Analysis of the latent structure using HDBSCAN clustering showed that the embeddings naturally organize hospitalizations into coherent groups, suggesting that the model learns interpretable clinical semantics that can potentially be reused in other tasks. In order to perform a detailed analysis, five predictive models were developed: an overall model trained on data from all departments, and three specific models dedicated to the surgery, oncology, and rehabilitation departments, respectively, and a complementary model including all other departments. The results demonstrate the feasibility and effectiveness of the embedding-based approach in capturing temporal and semantic information useful for predicting LoS, highlighting how performance varies depending on the department and the distribution of clinical data. The embeddings generated by CLMBR-T-Base have proven capable of encoding complex clinical relationships between events, promoting good generalization capabilities. Especially with regard to the overall model, the embedding representation allows for excellent predictions with an R² value of 0.705. However, the work also highlights the limitations related to the partial coverage of the CLMBR-T-Base vocabulary compared to the codes used in the Italian healthcare system and the lack of outpatient or follow-up information in the ICS Maugeri data. Overall, the work demonstrates the feasibility and effectiveness of using pre-trained foundation models to represent longitudinal clinical data and support complex predictive tasks, highlighting how their use can, with some adjustments, constitute a new paradigm for the creation of tools capable of supporting predictive and personalized medicine in hospitals.| File | Dimensione | Formato | |
|---|---|---|---|
|
Tesi_NegriFrancesca_527784.pdf
accesso aperto
Descrizione: File .pdf contenente la tesi "Uso dei foundation models per la rappresentazione della storia clinica ospedaliera: applicazione per la predizione della lunghezza del ricovero"
Dimensione
4.79 MB
Formato
Adobe PDF
|
4.79 MB | Adobe PDF | Visualizza/Apri |
È consentito all'utente scaricare e condividere i documenti disponibili a testo pieno in UNITESI UNIPV nel rispetto della licenza Creative Commons del tipo CC BY NC ND.
Per maggiori informazioni e per verifiche sull'eventuale disponibilità del file scrivere a: unitesi@unipv.it.
https://hdl.handle.net/20.500.14239/33643