An NLP framework for extracting family medical history and inferring hereditary disease risk from unstructured clinical text using Transformers (BioBERT), Deep Learning, and ML
📄 Project Overview Understanding family medical history is crucial for early disease prevention, yet this information is often locked in unstructured clinical notes. This project develops an NLP pipeline to extract family history, identify relationships, and infer multi-generational inheritance patterns (e.g., Autosomal Dominant risks) directly from text.
🚀 Key Features
Synthetic Data Generation: A robust script to generate realistic clinical notes with ground-truth genetic risk labels.
Multi-Model Approach: Implemented and compared 5 distinct architectures:
Machine Learning: Random Forest (Structured) vs. SVM (TF-IDF).
Deep Learning: Bi-LSTM vs. 1D-CNN (Keyword scanning).
Transformers: Fine-tuned BioBERT (SOTA performance).
Inference Engine: Detects risk for complex diseases (Breast Cancer, Diabetes, Colon Cancer) from natural language narratives.
🛠️ Tech Stack
Languages: Python
Libraries: TensorFlow/Keras, PyTorch, Hugging Face Transformers, Scikit-Learn.
Models: BioBERT v1.1, Bi-LSTM, 1D-CNN, LinearSVC.
📊 Results Highlight | Model | Accuracy | Key Finding | | :--- | :--- | :--- | | Random Forest | 97.0% | Strong baseline on structured data. | | Bi-LSTM | ~42.0% | Struggled with class imbalance ("Accuracy Trap"). | | BioBERT | 97.6% | Best Performer. Successfully handled multi-disease context. |