Detecting HIV-Related Stigma in Clinical Narratives Using Large Language Models
Researchers built the first NLP tool to automatically detect HIV-related stigma in clinical notes, but accuracy remains modest and real-world deployment faces steep hurdles.
The Thesis
HIV-related stigma — the social shame and discrimination experienced by people living with HIV — quietly shapes whether patients stay in care, take their medications, and disclose their status to providers. This stigma often surfaces in clinical notes as offhand phrases or patient self-reports, but no automated tool existed to extract it at scale. This paper trains and compares several large language models (LLMs) on a manually annotated dataset of 1,332 sentences from a Florida health system, finding that a specialized clinical encoder model called GatorTron-large performs best, though its Micro F1 score of 0.62 (a measure of combined precision and recall, where 1.0 is perfect) leaves meaningful room for error. The catch is significant: this is a single-institution dataset, the rarest stigma category resists detection, and zero-shot generative models — those given no examples before being asked to classify — failed to produce usable output nearly a third of the time.
Catalyst
Clinical NLP has reached a point where domain-specific large language models trained on millions of medical records (like GatorTron, developed at the University of Florida) are publicly available and can be fine-tuned on specialized tasks without building from scratch. Simultaneously, the emergence of capable open-weight generative models like LLaMA and MedGemma has made it feasible to test few-shot prompting — giving the model a handful of labeled examples at inference time — as an alternative to full fine-tuning, lowering the bar for rapid experimentation. Regulatory and clinical interest in social determinants of health (the non-medical factors that influence health outcomes) has also sharpened demand for tools that can surface psychosocial signals buried in unstructured text.
What's New
Prior work on stigma detection in clinical text relied almost entirely on manual chart review or simple keyword searches, which are expensive, slow, and miss context-dependent expressions. A handful of early NLP studies used traditional machine learning on small, narrow corpora, but none systematically compared modern LLMs across multiple stigma subtypes using a clinically grounded annotation framework. This paper introduces the first annotated benchmark dataset for HIV stigma in clinical notes and is the first to benchmark both encoder-based fine-tuned models and generative LLMs — including open-weight models that could be deployed on-premises without sending patient data to a third-party API.
The Counter
The dataset of 1,332 annotated sentences from a single academic health system in Florida is far too small and geographically narrow to claim a generalizable clinical tool. A Micro F1 of 0.62 means the model is wrong on roughly four out of every ten stigma-relevant sentences — an error rate that could introduce systematic bias into any downstream clinical decision support or research pipeline. The four stigma subscales are expert-defined constructs with known inter-rater reliability challenges; the paper does not report inter-annotator agreement in detail, so it is unclear how much of the model's struggle reflects genuine linguistic ambiguity versus label noise. Zero-shot generative models failed to produce parseable output up to 32% of the time, which is an alarming reliability floor for any clinical deployment scenario. Finally, HIV stigma in clinical notes is notoriously underreported — providers often avoid documenting sensitive psychosocial content — so even a perfect classifier would only capture a fraction of true stigma burden, limiting the tool's utility for population-level research.
Longs
- ORCL — Oracle Health (Cerner EHR) is a natural integration point for clinical NLP modules detecting social determinants
- VEEV — Veeva Systems, which serves life sciences with clinical data infrastructure, could embed stigma-detection tooling in real-world evidence pipelines
- RXRX — Recursion Pharmaceuticals (speculative), if clinical NLP expands into patient stratification for HIV treatment trials
- HIMS — Hims & Hers Health, which operates telehealth for HIV PrEP, has direct patient-engagement incentive to reduce stigma barriers
Shorts
- Manual chart review vendors and research CROs that currently charge for labor-intensive social determinants of health abstraction — automated NLP directly undercuts their billing model
- Generic (non-clinical) LLM API providers: the paper shows zero-shot GPT-class models underperform domain-specific fine-tuned models significantly, weakening the 'just use the frontier API' pitch for clinical NLP tasks
Enablers (Picks & Shovels)
- GatorTron — the University of Florida's clinical LLM trained on 90 billion words of de-identified medical text, which underpins the best-performing model in this study
- i2b2 / n2c2 NLP Research Consortia — provide the annotation infrastructure and shared task frameworks that make clinical NLP benchmarking credible
- Meta LLaMA open-weight model family — enables on-premises inference without exposing protected health information to external APIs
- FHIR (Fast Healthcare Interoperability Resources) — the data standard that makes clinical note extraction pipelines tractable across hospital systems
Private Watchlist
- Abridge — clinical note AI company that could extend stigma-detection as a module
- Nuance (now Microsoft) — already embedded in EHR workflows for ambient clinical documentation
- Regard — AI clinical reasoning platform that surfaces social and psychiatric signals from notes
- DeepScribe — ambient AI for clinical documentation with a growing NLP classification layer
Resources
The Paper
Human immunodeficiency virus (HIV)-related stigma is a critical psychosocial determinant of health for people living with HIV (PLWH), influencing mental health, engagement in care, and treatment outcomes. Although stigma-related experiences are documented in clinical narratives, there is a lack of off-the-shelf tools to extract and categorize them. This study aims to develop a large language model (LLM)-based tool for identifying HIV stigma from clinical notes. We identified clinical notes from PLWH receiving care at the University of Florida (UF) Health between 2012 and 2022. Candidate sentences were identified using expert-curated stigma-related keywords and iteratively expanded via clinical word embeddings. A total of 1,332 sentences were manually annotated across four stigma subscales: Concern with Public Attitudes, Disclosure Concerns, Negative Self-Image, and Personalized Stigma. We compared GatorTron-large and BERT as encoder-based baselines, and GPT-OSS-20B, LLaMA-8B, and MedGemma-27B as generative LLMs, under zero-shot and few-shot prompting. GatorTron-large achieved the best overall performance (Micro F1 = 0.62). Few-shot prompting substantially improved generative model performance, with 5-shot GPT-OSS-20B and LLaMA-8B achieving Micro-F1 scores of 0.57 and 0.59, respectively. Performance varied by stigma subscale, with Negative Self-Image showing the highest predictability and Personalized Stigma remaining the most challenging. Zero-shot generative inference exhibited non-trivial failure rates (up to 32%). This study develops the first practical NLP tool for identifying HIV stigma in clinical notes.