Revolutionizing Medicine with Healthcare Foundation Models

The rapid advancement of artificial intelligence (AI) is driving a transformative shift in healthcare. At the forefront of this transformation are Healthcare Foundation Models (HFMs)—a class of pre-trained AI models built to address the diverse needs of medical practice. These versatile models promise to bridge the gap between the specialized nature of traditional AI systems and the complex, multifaceted challenges of modern healthcare.

In this blog, we delve into the essence of HFMs, their current applications across healthcare domains, the challenges they face, and their future directions in reshaping medicine.

What Are Healthcare Foundation Models?

Healthcare Foundation Models (HFMs) are pre-trained on massive and diverse datasets, enabling them to recognize patterns and relationships across various types of data. These models can be fine-tuned for specific healthcare tasks, making them adaptable and capable of addressing real-world clinical needs.

Unlike traditional AI models, which are often designed for single, narrow applications, HFMs are versatile and scalable, enabling them to handle a broader spectrum of tasks, from diagnosis and treatment planning to research and education.

Timeline of Main Events & Model Development

Early 2010s:

• 2011: The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI) create a reference database of lung nodules on CT scans (LIDC-IDRI).
• 2011: The Parkinson Progression Marker Initiative (PPMI) database is launched to study Parkinson’s disease.
• 2012: NCBI GenBank database provides a vast collection of DNA sequences (3.7B).
• 2014: The Multimodal Brain Tumor Image Segmentation Benchmark (BraTS) is created.

Mid-2010s:

• 2017: The Human Cell Atlas project is started.
• 2017: The Cancer Genome Atlas Program (TCGA) is established as a large-scale cancer genomics initiative.

Late 2010s – Early 2020s:

• 2018: DeepLesion dataset is created for large-scale lesion annotation and detection.
• 2018: The GENCODE reference annotation project for human and mouse genomes is published.
• 2019: BioBERT and ClinicalBERT are developed as early language models for healthcare using Hybrid Learning (HL). Med3D is developed for 3D medical image segmentation via Supervised Learning (SL). Models Genesis are also developed via Generative Learning (GL).
• 2019: The Chinese Medical Knowledge Graph CMeKG-8K is created.
• 2019: MedDialog dataset of medical dialogue is made available.
• 2020: The HyperKvasir, a dataset of gastrointestinal endoscopy, is created.
• 2020: The Momentum Contrast method for unsupervised visual representation learning is published.
• 2020: The AlphaBERT and BEHRT language models for healthcare are developed based on Generative Learning (GL).
• 2020: C2L, a Contrastive Learning (CL) model for X-Ray image classification, is introduced.
• 2021: PubMedBERT is developed using Hybrid Learning (HL).
• 2021: The AlphaFold Protein Structure Database is created.
• 2021: DNABERT and other early biological foundation models are introduced.
• 2021: The Multi-modal BERT (MMBERT) architecture is developed.
• 2021: TransVW, a Hybrid Learning (HL) model for CT and X-Ray image analysis, is introduced.
• 2021: The CTSpine1K and CTPelvic1K datasets are created for spinal vertebrae and pelvic bone segmentation in CT images.
• 2021: The scBERT model for single-cell RNA-seq data is introduced.
• 2021: A large-scale dataset for detecting abnormalities in musculoskeletal radiographs is created (MURA).
• 2021: A computer-aided detection system for colonoscopy and a public video database (SUN) is developed.
• 2021: The Endoslam dataset is introduced for endoscopic visual odometry and depth estimation.
• 2021: GLoRIA, a CL-based model for X-ray images, is introduced.

2022:

• The “High-resolution image synthesis with latent diffusion models” paper is published.
• ConVIRT, a contrastive learning method for X-ray images, is introduced.
• GVSL, a Hybrid Learning model for 3D segmentation, is introduced.
• The Kvasir-Capsule video endoscopy dataset is created.
• The scRNA-seq expression data is organized into the CellxGene corpus.
• The Deeplesion Dataset for Lesion Annotation is made available.
• A large dataset for stroke neuroimaging is created (ATLAS v2.0).
• The MedCLIP method is developed.
• The Clinical-BERT model is developed.
• The MoCo-CXR model is developed using Contrastive Learning and Adapter Tuning (CL FT, AT).
• The MedViLL model is introduced using Hybrid Learning (HL).
• AROGS retinal fundus photograph dataset is introduced.
• The LdPolypVideo dataset is released.
• The ChestX-ray14 Dataset is released.
• The MedRoBERTa model for Dutch Electronic Health Records is created.
• The M-FLAG model is developed using Contrastive Learning.
• The Med-Flamingo model is introduced.
• The Roentgen visual language model for chest X-Ray generation is introduced.

2023:

• The Segment Anything Model (SAM) is released.
• The TotalSegmentator dataset is made available.
• MedCPT, PMC-LLaMA, GatorTronGPT, and other Language Foundation Models (LFM) are developed.
• The Geneformer model is introduced.
• Many Vision Foundation Models (VFM) are created with various pre-training and adaptation methods, such as STU-Net, UniverSeg, RETFound, VisionFM, SegVol, and many adaptations of the SAM Model.
• Many Bio Foundation Models (BFM) are developed, using both Generative and Contrastive learning techniques, such as ProGen, ProGen2, scBERT, DNABERT, and HyenaDNA.
• Many Multi-modal Foundation Models (MFM) are created, using Generative and Contrastive learning methods, including MMBERT, MRM, BiomedGPT, and RadFM. Many models are adapted from CLIP or Stable Diffusion pipelines.
• Many multimodal datasets are made available such as MIMIC-CXR, PadChest, and CheXpert.
• The Med-PaLMM model is created.
• The Medagents framework is developed.
• The UniBrain model is developed.
• The Molecular Structure-text model (MoleculeSTM) is developed.
• The LLaVA-Med model is created.
• The MedKLIP Model is introduced.
• The xTrimoPGLM model is introduced.
• The CXR-CLIP is introduced.
• The Med-UniC model is introduced.
• Many variations of the Segment Anything Model for medical image segmentation are developed, such as MA-SAM, 3DSAM-adapter, and MedSAM.
• The scGPT model for single-cell multi-omics data is introduced.
• The Uni model is introduced.
• The Virchow model is introduced.
• The Brow model is developed.
• The CtransPath model is developed.
• The PubmedCLIP model is introduced.

2024:

• BioMistral, Me LLaMA, Zhongjing, and OncoGPT are released as Language Foundation Models.
• The BiMediX model for QA is introduced.
• The LVM-Med model for multi-modal image analysis is introduced.
• The Adaptivesam model is developed for surgical image analysis.
• The SegmentAnyBone model is developed.
• The PathAsst and PathChat models are developed.
• The RudolfV model is developed for digital pathology analysis.
• The EviPrompt method is introduced for medical image analysis.
• The PUNETR Model is introduced.

Key Components of HFMs

The development of HFMs revolves around three core components:

1. Methods
   HFMs leverage a range of machine learning paradigms, such as:
   • Generative Learning (GL): Enables the creation of new data based on learned patterns.
   • Contrastive Learning (CL): Helps models distinguish between subtle differences in data.
   • Hybrid and Supervised Learning (HL/SL): Combines labeled and unlabeled data for robust learning.

   These methods are complemented by adaptation techniques like fine-tuning, adapter tuning, and prompt engineering, which tailor models to specific clinical scenarios.

2. Data
   HFMs thrive on diverse datasets, including:
   • Textual data from medical literature and health records.
   • Imaging data from radiology and pathology.
   • Biological data, such as omics (genomics, proteomics).
   • Other modalities like audio (e.g., heart sounds) and bioelectric signals.

   High-quality, large-scale datasets are indispensable for the effective training of HFMs.

3. Applications
   HFMs offer vast applications, including:
   • Diagnosis and Prognosis: Enhancing clinical decision-making.
   • Medical Imaging: Improving accuracy in image analysis.
   • Drug Discovery: Accelerating the identification of potential therapeutics.
   • Education and Chatbots: Supporting medical professionals and patient engagement.

Progress in HFM Sub-Fields

HFMs are making strides in several specialized domains:

1. Language Foundation Models (LFMs):
   LFMs excel in processing and understanding medical text, aiding in tasks like diagnosis, consultation, and report generation. Notable examples include MedCPT, BioBERT, and ClinicalBERT.
2. Vision Foundation Models (VFMs):
   VFMs process medical images for tasks like segmentation and analysis, relying on advanced pre-training methods such as self-supervised and contrastive learning.
3. Bioinformatics Foundation Models (BFMs):
   BFMs analyze biological sequences (DNA, RNA, proteins) using specialized methods like masked omics modeling and next-token prediction. These models play a pivotal role in understanding disease risk and drug sensitivity.
4. Multimodal Foundation Models (MFMs):
   MFMs integrate diverse data types (e.g., text and images) for comprehensive analysis. These models are critical for complex tasks requiring multi-faceted inputs, such as virtual consultations.

Challenges Facing HFMs

Despite their promise, HFMs face significant hurdles:

1. Data Limitations:
   • Lack of large-scale, high-quality, and diverse datasets.
   • Ethical concerns and costs of data collection and annotation.
2. Algorithmic Issues:
   • Limited model adaptability and scalability.
   • Challenges in ensuring model reliability and explainability.
3. Computing Infrastructure:
   • High computational demands make training HFMs costly.
   • Dependence on specialized hardware (e.g., GPUs).
4. Real-World Complexity:
   • HFMs must generalize across different populations and healthcare systems.
   • Integration of data from multiple modalities and handling missing data remain challenging.
5. Trust and Explainability:
   • The “black box” nature of HFMs impedes trust.
   • Models must become more transparent and secure to gain widespread acceptance.

Future Directions for HFMs

To fully realize their potential, HFMs must evolve along several key dimensions:

1. From Single-Task to Multi-Task:
   HFMs should handle multiple healthcare tasks simultaneously and adapt to open-ended scenarios.
2. From Single-Modality to Multi-Modality:
   Future models must integrate text, images, and biosignals seamlessly, leveraging complementarities between data types.
3. From Exploration to Trust:
   Emphasis on explainability, security, and reliability will build user confidence.
4. From Single-Domain to Multi-Domain:
   HFMs must generalize across diverse populations, regions, and medical centers, ensuring equitable healthcare solutions.

Conclusion

Healthcare Foundation Models represent a groundbreaking advancement in AI, offering unparalleled potential to transform medicine. While challenges remain, continued research into data quality, algorithmic improvements, and ethical practices will ensure these models become indispensable tools in healthcare.

By fostering collaboration between AI and human expertise, HFMs promise a future where medicine is not only more efficient but also more personalized and accessible.

The rise of HFMs heralds a new era in medicine—one where AI empowers humanity to achieve greater health outcomes than ever before.

FAQ’s – Healthcare Foundation Models

What are foundation models in healthcare, and why are they needed?

Foundation models in healthcare are AI models trained on massive, diverse datasets of healthcare information (text, images, biological data, etc.) with the goal of adapting to a wide range of healthcare applications. They are needed because current specialist AI models are often limited to specific tasks and don’t generalize well to the diverse scenarios in real-world healthcare settings. These models aim to bridge the gap and bring general AI capabilities to the medical field.

What are the main pre-training paradigms used for healthcare foundation models, and how do they work?

The main pre-training paradigms include:

• Generative Learning (GL): Models learn to generate meaningful data, like predicting the next word in a sentence or reconstructing a masked image, by learning representations that capture patterns in the data. Examples include Next Token Prediction (NTP) and Masked Language/Image Modeling (MLM/MIM).
• Contrastive Learning (CL): Models learn representations where similar data instances are close together and dissimilar instances are far apart in a representation space. This helps to distinguish between different patterns and features.
• Hybrid Learning (HL): Combines different learning methods, such as generative and contrastive, to achieve a more comprehensive understanding of the data.
• Supervised Learning (SL): Uses labeled data to directly train models to predict outcomes and recognize patterns specific to a task. This is often used when annotated data is available and effective for specific applications.

What are Language Foundation Models (LFMs) for healthcare, and what tasks do they perform?

LFMs in healthcare are AI models pre-trained on large text corpora of medical and general language data. They learn to understand and generate human language, which helps them in various NLP tasks like:

• Information Retrieval (IR): Finding relevant medical documents or information.
• Named Entity Recognition (NER): Identifying medical entities like diseases, drugs, and symptoms in text.
• Relation Extraction (RE): Determining relationships between entities in text, like drug-disease interactions.
• Question Answering (QA): Answering medical questions from text data.
• Dialogue (DIAL): Engaging in conversational interactions for medical purposes.
• Text Classification (TC): Categorizing medical texts into predefined groups.
• Summarization (SUM): Creating concise summaries of medical documents or information.

How do Vision Foundation Models (VFMs) for healthcare leverage pre-training, and what are some common tasks they perform?

VFMs in healthcare are pre-trained on large datasets of medical images. Pre-training helps these models learn generic image features and then adapt to specific medical tasks like:

• Image Classification (CLS): Classifying images into different categories (e.g. identifying disease or organ).
• Image Segmentation (SEG): Identifying and delineating regions of interest in an image (e.g. organ or tumor).
• Image Detection (DET): Identifying the location of objects in an image (e.g. polyps or surgical tools).
• Report Generation (RG): Generating text reports from medical images.
• Prognosis (PR): Predicting the likely course of a disease based on medical images.

VFMs leverage both supervised pre-training on annotated data, where semantics in medical images is learned, and self-supervised methods, such as masked image modeling, to learn universal representations from non-annotated data.

What are Biological Foundation Models (BFMs), and how are they pre-trained?

BFMs are foundation models pre-trained on biological data, such as DNA, RNA, proteins, and single-cell expression data. They are trained using generative (e.g., masked omics modeling, next token prediction) and contrastive learning techniques to learn contextual dependencies and extract meaningful biological features, enabling the models to:

• Sequence Analysis (SA): Analyze biological sequences, such as DNA or protein sequences.
• Interaction Analysis (IA): Understanding interaction between biological entities.
• Structure and Function Analysis (SFA): Predict and understand the structure and function of molecules or cells.
• Disease Research and Drug Response (DR): Aid in the investigation of diseases and how they respond to treatments.

What are Multimodal Foundation Models (MFMs) for healthcare, and how do they handle multiple data types?

MFMs for healthcare are trained using multiple data modalities (text, images, biological data) simultaneously. This enables them to understand the complex interrelationships between different types of medical information. Pre-training paradigms are used that:

• Generate cross-modal data: (e.g., generate text from an image)
• Contrast data across modalities: (e.g., group similar text and image pairs).
• Use hybrid methods to leverage each of these different tasks.

They can perform a variety of tasks, such as:

• Visual Question Answering (VQA): Answering questions based on both images and text.
• Cross-Modal Retrieval (CMR): Retrieving data from one modality based on a query in another.
• Cross-Modal Generation (CMG): Generating data in one modality from another.
• Report Generation (RG): Creating textual reports from medical images.
• Phrase-Grounding (PG): Identify specific regions in an image based on text inputs.

How are these foundation models adapted to specific healthcare tasks after pre-training?

After pre-training, foundation models are adapted to specific tasks using techniques like:

• Fine-Tuning (FT): Adjusting the parameters of the pre-trained model using a task-specific dataset. This involves further training the whole model for a task.
• Adapter Tuning (AT): Adding new, task-specific parameters (adapters) into the pre-trained model and only training these added parameters, while keeping the original parameters fixed. This is parameter efficient and keeps the model’s general knowledge.
• Prompt Engineering (PE): Designing or learning specific prompts (inputs) that guide the model to perform the desired task. The pre-trained model is leveraged through prompt tuning without parameter updating.

These methods allow models to generalize effectively to new and specific tasks in healthcare.

What are some of the challenges and considerations when deploying healthcare foundation models?

Some important challenges include:

• Data Bias: Ensuring that training data is representative of diverse patient populations to prevent biased outcomes.
• Explainability: Developing models that can explain their decisions to build trust and aid in clinical decision-making. This is crucial for clinicians to understand how the model arrived at a diagnosis or treatment recommendation.
• Data Privacy and Security: Protecting sensitive patient data during training and deployment.
• Computational Resources: Foundation models often require significant computational resources, which can limit their accessibility and deployment.
• Open-set recognition: Medical data can contain unseen classes and require models to be robust in generalizing to unknown situations.
• Missing modality robustness: Ensuring that models can maintain high performance, even when some data modalities are unavailable.
• Evaluation: Measuring the performance of foundation models can be difficult, with the need for rigorous methods for fair and reliable benchmarks.

Addressing these challenges is critical for the responsible and effective use of foundation models in healthcare.

Glossary of Key Terms

Adapter Tuning (AT): A method for adapting pre-trained models by adding new parameters (adapters) and only training these, leaving the original parameters unchanged.

Biological Foundation Model (BFM): AI model pre-trained on large biological datasets to understand and model biological systems.

Contrastive Learning (CL): A pre-training technique that learns representations where similar instances are close together and dissimilar instances are far apart.

Fine-tuning (FT): The process of adjusting the parameters of a pre-trained model on a new task-specific dataset.

Generative Learning (GL): A pre-training technique that learns to generate meaningful information from represented data.

Hybrid Learning (HL): A pre-training approach that combines different learning methods.

Language Foundation Model (LFM): AI model pre-trained on large text datasets to understand and generate natural language.

Masked Image Modeling (MIM): A pre-training task where parts of an image are masked, and the model is trained to reconstruct the original image.

Masked Language Modeling (MLM): A pre-training task where parts of a sentence are masked, and the model is trained to predict the masked words.

Masked Omics Modeling (MOM): A pre-training task where parts of biological data sequences or values are masked and the model is trained to reconstruct.

Multimodal Foundation Model (MFM): AI model pre-trained on multiple data modalities (e.g., text, images, biological data) to understand and integrate different types of information.

Next Token Prediction (NTP): A generative learning method where the model learns to predict the next token in a sequence.

Prompt Engineering (PE): The process of designing or learning prompts to guide pre-trained models to perform desired tasks.

Self-Supervised Learning (SSL): A pre-training technique where a model learns from unlabeled data by creating a pretext task without human annotation.

Supervised Learning (SL): A learning paradigm that uses labeled data to train models to predict outcomes and recognize patterns.

Vision Foundation Model (VFM): AI model pre-trained on large image datasets to understand and process visual information.

Healthcare Foundation Models Study Guide

Quiz

1. What is the primary challenge that healthcare foundation models (HFMs) aim to address?
2. Briefly describe the four pre-training paradigms used in HFMs, including examples of each.
3. Explain the difference between fine-tuning and adapter tuning in the context of HFM adaptation.
4. What is masked language modeling (MLM), and how is it used in pre-training language foundation models?
5. How does the continuity of vision information pose challenges for vision foundation models, and what are some approaches to address this?
6. Describe the masked image modeling (MIM) approach used in vision foundation model pre-training.
7. What is masked omics modeling (MOM) and what type of data is it typically used on?
8. What is the difference between generative learning and contrastive learning in multimodal foundation model (MFM) pre-training?
9. Give an example of a public multimodal dataset and its potential use.
10. Explain how the Segment Anything Model (SAM) has been adapted for medical image analysis.

Quiz Answer Key

1. The primary challenge HFMs address is the contradiction between specialized AI models for specific healthcare tasks and the diverse, real-world healthcare scenarios and requirements, thus hindering widespread application.
2. The four pre-training paradigms are: generative learning (GL), which includes methods like next token prediction (NTP); contrastive learning (CL), which learns representations where similar instances are close and dissimilar instances are far apart; hybrid learning (HL), which combines methods; and supervised learning (SL), which uses labeled data to train models for predictions.
3. Fine-tuning adjusts the parameters within pre-trained models, while adapter tuning adds new parameters (adapters) to pre-trained models and only trains these additional parameters, leaving the original model’s parameters unchanged.
4. Masked language modeling is a pre-training technique where a portion of input tokens in a sentence are randomly masked, and the model is trained to predict those masked tokens using the surrounding context. It helps the model understand the relationship between words in a sentence.
5. The continuity of vision information makes it difficult to separate the semantics of content; self-supervised learning (SSL) and supervised learning (SL) using annotations for specific tasks address this issue.
6. Masked image modeling is a pre-training method used in VFMs that corrupts images (e.g., masking regions) and trains a model to reconstruct the original images from the corrupted versions.
7. Masked omics modeling is a pre-training technique used in biological foundation models where values or sequences in biological data are randomly masked and the model is trained to reconstruct them; it’s typically used on data such as scRNA-seq, DNA, and RNA.
8. Generative learning (GL) in MFM pre-training focuses on generating data across modalities and reconstructs information from different modalities; contrastive learning (CL) encodes data from different modalities into the same space to improve cross-modal understanding.
9. An example is the MIMIC-CXR dataset, which pairs x-ray images with medical reports and is used for vision-language learning, training models to understand and generate text based on images, or vice-versa.
10. SAM has been adapted for medical image analysis by training new adapters to the model that keep its segmentation capabilities but allow for specialized training in the medical domain via few parameters, allowing it to transfer its large-scale training for natural images to medical images.

Essay Questions

1. Discuss the impact of foundation models on healthcare, and analyze the potential benefits and challenges of their widespread implementation.
2. Compare and contrast the pre-training paradigms used in language, vision, biological, and multimodal foundation models for healthcare, illustrating the unique considerations for each.
3. Evaluate the various adaptation strategies employed in healthcare foundation models, including fine-tuning, adapter tuning, and prompt engineering, and assess their effectiveness in diverse clinical tasks.
4. Explore the types of datasets used for training healthcare foundation models, including language, vision, biological, and multimodal datasets, and discuss the challenges of data curation, privacy, and ethics.
5. Critically assess the current state of multimodal foundation models in healthcare, analyzing the effectiveness of different pre-training paradigms and their potential for integrated diagnostic and therapeutic applications.

Reference

He, Y., Huang, F., Jiang, X., Nie, Y., Wang, M., Wang, J., & Chen, H. (2024). Foundation model for advancing healthcare: Challenges, opportunities, and future directions. arXiv preprint arXiv:2404.03264.