Grounding Clinical AI Competency in Human Cognition Through the Clinical World Model and Skill-Mix Framework
A new conceptual framework argues clinical AI must be evaluated by precise context coordinates, not broad benchmarks.
The Thesis
Clinical AI systems are routinely validated on broad benchmarks — but those benchmarks may tell us almost nothing about how a tool performs in a specific hospital ward, for a specific patient type, on a specific task. This paper introduces the Clinical World Model, a formal structure that defines clinical care as a three-way interaction among patient, provider, and healthcare ecosystem. It then defines eight dimensions — things like care setting, provider role, and how much authority the AI is given — whose combinations produce billions of distinct 'competency coordinates.' The authors argue that validating an AI in one coordinate provides minimal evidence it works in another, making the evaluation space effectively irreducible. The catch: this is a theoretical framework with no empirical validation of its own, so its practical impact depends entirely on adoption by regulators, developers, and health systems.
Catalyst
The FDA and international regulators have accelerated approval of AI-based clinical decision tools, creating pressure to define what 'validated' actually means in a clinical context. Meanwhile, high-profile failures of AI diagnostic tools when deployed in settings different from their training environments have made the generalization problem impossible to ignore. This framework arrives as the field is actively searching for a shared vocabulary to connect developers, regulators, and clinicians.
What's New
Prior frameworks in clinical AI focused on one of three things in isolation: performance evaluation (benchmark scores), regulatory compliance (FDA guidance documents), or system design principles. None provided a shared model of the clinical world that could connect all three. This paper proposes a unified grammar — the Clinical World Model and its Skill-Mix dimensions — that explicitly maps how any agent, human or AI, transforms information into clinical action, then uses that map to argue that validation must be coordinate-specific rather than general.
The Counter
This paper is pure conceptual architecture — there is no dataset, no model, no experiment, and no empirical test of whether the framework actually improves clinical AI safety or evaluation quality. The claim that the competency space is 'irreducible' is presented as a structural implication of the framework's own design, which risks being circular: the authors built a space with billions of coordinates and then concluded that each coordinate is unique. The practical question — whether distinguishing, say, coordinate A from coordinate B actually predicts meaningfully different AI performance in the real world — is left entirely unaddressed. Regulators and health systems already operate under severe resource constraints; demanding coordinate-specific validation for every deployment context could make clinical AI development economically nonviable without clear evidence that the granularity buys safety. Finally, the field already has frameworks from the FDA, WHO, and academic groups like the DECIDE-AI initiative; whether this one offers enough additional structure to displace or integrate with existing standards is an open question the paper does not engage.
Longs
- VEEV (Veeva Systems) — clinical data infrastructure that would need to tag AI outputs by competency context
- IQVIA (IQV) — real-world evidence generation aligned with coordinate-specific validation demands
- HIMS — digital health platforms facing new evaluation scrutiny
- RXRX (Recursion Pharmaceuticals) — AI drug discovery firms that could face analogous validation frameworks
Shorts
- Clinical AI vendors who have built regulatory strategies around single large benchmark studies — their validation evidence would be reframed as narrow and insufficient
- General-purpose LLM providers (OpenAI, Google) positioning foundation models as broadly clinical — the framework explicitly challenges the idea that one model can be validated across the competency space
- CROs (contract research organizations) selling generic AI clinical validation study designs — their standard protocols may not map to coordinate-specific requirements
Enablers (Picks & Shovels)
- HL7 FHIR (Fast Healthcare Interoperability Resources) standard — the data plumbing needed to tag clinical AI interactions by context
- FDA's Digital Health Center of Excellence — the regulatory body most likely to adopt or reject this kind of framework
- SNOMED CT and ICD-11 — medical ontologies that could operationalize the 'condition' dimension of the Skill-Mix
- Open-source clinical NLP projects like scispaCy — tooling that would need to implement coordinate-aware evaluation
Private Watchlist
- Abridge — AI clinical documentation, directly affected by care-setting and role-specific validation requirements
- Nabla — ambient clinical AI whose deployment context varies widely across provider roles
- Hippocratic AI — clinical AI agent startup navigating exactly the authority and role dimensions this framework defines
- Aidoc — radiology AI whose regulatory strategy depends on condition- and setting-specific claims
Resources
The Paper
The competency of any intelligent agent is bounded by its formal account of the world in which it operates. Clinical AI lacks such an account. Existing frameworks address evaluation, regulation, or system design in isolation, without a shared model of the clinical world to connect them. We introduce the Clinical World Model, a framework that formalizes care as a tripartite interaction among Patient, Provider, and Ecosystem. To formalize how any agent, whether human or artificial, transforms information into clinical action, we develop parallel decision-making architectures for providers, patients, and AI agents, grounded in validated principles of clinical cognition. The Clinical AI Skill-Mix operationalizes competency through eight dimensions. Five define the clinical competency space (condition, phase, care setting, provider role, and task) and three specify how AI engages human reasoning (assigned authority, agent facing, and anchoring layer). The combinatorial product of these dimensions yields a space of billions of distinct competency coordinates. A central structural implication is that validation within one coordinate provides minimal evidence for performance in another, rendering the competency space irreducible. The framework supplies a common grammar through which clinical AI can be specified, evaluated, and bounded across stakeholders. By making this structure explicit, the Clinical World Model reframes the field's central question from whether AI works to in which competency coordinates reliability has been demonstrated, and for whom.