A visual foundation model trained directly on images produced in the course of routine care — unsorted, unannotated, without even the text of the reports — outperforms models built on public corpora. But the most instructive result is not that one: several of the models it beat drew on the same hospital archive. What the study isolates is the effect of the learning objective.

Source: nature.com

Discovery

Technical explanation

1. Why neuroimaging is missing from the public domain. The starting point is not technical but regulatory: a cranial CT or MRI volume contains the patient's face, reconstructible in 3D. This intrinsic identifiability has kept neuroimaging out of the large open corpora that fed the foundation models. The data exist — by the million, in every hospital's PACS archives — but they are private, heterogeneous and unannotated. The authors' bet is to go and fetch them where they are, rather than wait for a public corpus that will not come.

2. Vol-JEPA — predicting in latent space, not in pixel space. This is the mechanical core of the work, and it differs sharply from a classical masked autoencoder. The 3D volume is first cut into patches (volumetric tokens). After background removal, the remaining patches are split into a small visible context and a larger masked target — it is this asymmetry that makes the task hard. A student encoder — an "online" 3D Vision Transformer — computes a representation of the context alone; a predictor must then produce, from that representation, what the representation of the masked regions would be. In parallel, an "offline" teacher encoder sees the complete volume and supplies the target, with gradients blocked through it. Training minimises a Huber loss (smooth L1) between prediction and target:

$\ell(x)=\begin{cases}0{,}5\,x^{2} & \text{si } |x|<1\\ |x|-0{,}5 & \text{sinon}\end{cases}$ℓ(x)={0,5x2∣x∣−0,5​si ∣x∣<1sinon​

The decisive difference: the target is not the missing pixel but its encoding. Reconstructing pixels would force the model to model acquisition noise, motion artefacts, detector grain — everything that varies from one machine to another without carrying clinical information. By predicting in latent space, the model is free to ignore the unpredictable and retains only what is structurally predictable: the anatomy and its alterations. That is precisely what makes the approach tolerant of uncurated data.

3. Why the teacher does not collapse. An architecture in which the student learns to imitate a target produced by a twin network risks the trivial solution: both networks emit a constant, the loss falls to zero, nothing is learned. The safeguard is that the teacher is not trained by gradient descent — its weights are an exponential moving average of the student's:

$\theta_{\text{enseignant}} \leftarrow \tau\,\theta_{\text{enseignant}} + (1-\tau)\,\theta_{\text{étudiant}}$θenseignant​←τθenseignant​+(1−τ)θeˊtudiant​

The teacher thus remains a smoothed, delayed version of the student: a target that evolves too slowly to be caught by a shortcut, but fast enough to progress with it. No label, no radiology report intervenes at this stage. (The explicit form of these two equations is a standard gloss of the method, not a quotation from the article.)

4. What the evaluation measures — and what it demonstrates. AUROC quantifies the ability to separate presence from absence of a condition, independently of the chosen threshold: $\mathrm{AUROC}=0{,}5$AUROC=0,5 is equivalent to chance, $1{,}0$1,0 to perfect separation. It is macro-averaged here: each diagnosis weighs the same, including the rarest. That choice is what makes the figure informative — a frequency-weighted average would have been inflated by common, easy cases. Obtaining $92{,}68\%$92,68% across 82 CT tasks and $92{,}49\%$92,49% across 74 MRI tasks therefore measures a competence spread over 156 diagnoses, not a peak on a few dominant pathologies. One clarification that matters for interpreting the figure: the encoder is frozen after pre-training, and it is supervised classification probes, trained on top of it, that produce these scores. The pre-training is label-free; the evaluation is not.

5. The comparison that counts: same archive, different objectives. This is the article's strongest methodological point, and the one the raw figures mask. PRIMA, HLIP, NeuroMAE and NeuroVFM were all pre-trained on UM-NeuroImages. The comparisons therefore control for the data source, and the gaps are attributable to the pre-training objective and the architecture, not to the provenance of the images. NeuroMAE is not, incidentally, a competing model published by a third-party team: it is a compute-budget-matched baseline, built by the authors themselves (voxel reconstruction, random masking at 85%) for the sole purpose of isolating the effect of the objective. On the primary endpoint, NeuroVFM leads HLIP by $+0{,}98$+0,98 point (95% CI: 0.76–1.20), NeuroMAE by $+1{,}55$+1,55, DINOv3 by $+2{,}24$+2,24, BiomedCLIP by $+2{,}88$+2,88 and PRIMA by $+3{,}87$+3,87 — this last margin standing apart, PRIMA having been trained and evaluated only on the MRI subset. Nor is the overlap ever total: HLIP was trained on 444,188 study–report pairs, a fraction of the 566,915 studies.

6. From diagnosis to triage: the prospective test. Coupled with the Qwen3-14B language model in a LLaVA-1.5-type architecture, NeuroVFM powers a system able to draft a preliminary report and prioritise examinations. The protocol is a prospective, silent feasibility study — the system runs without influencing care — conducted at health-system scale over one consecutive week, from 18 to 25 January 2026, covering 1,155 examinations (601 MRI, 544 CT). The generated reports are filtered by a screening language model, then a blinded panel of expert clinicians renders the triage decision from the generated reports — that is the one being scored — and the reference decision from the real reports. The 92.6% balanced accuracy of the NeuroVFM arm (95% CI: 89.8–95.2), against 71.2% for the GPT-5 arm (95% CI: 67.2–75.2), that is $+21{,}4$+21,4 points, therefore measures a complete chain — model, automatic screening and human reading — and not the model alone.

Why it worked

The gain comes less from the provenance of the data than from the learning objective. The raw archive is a necessary condition — without it, no 5.24 million volumes — but it does not suffice to explain the gap: three of the five comparators drew on the same archive. What the article establishes is that, at identical source, latent-space prediction produces representations that generalise better as task diversity increases. The nominally highest margin ($+3{,}87$+3,87) is PRIMA's, but the article does not present it as such: it reserves mention of the widest margins for the models trained at internet scale — DINOv3 ($+2{,}24$+2,24) and BiomedCLIP ($+2{,}88$+2,88) — where the objective and the training data both differ, their effects compounding.

The limits are explicit, and they seriously temper the narrative. Against HLIP — the strongest baseline, trained on the same archive — the advantage is only $0{,}98$0,98 point: NeuroVFM wins on 10 of the 17 CT categories without a single defeat, but on only 5 of the 18 MRI categories, HLIP taking the inflammatory category and the remainder being non-significant. On CQ500, a small test set whose labels track the wording of the reports closely, HLIP even outperforms NeuroVFM on the majority of tasks. The authors draw the right conclusion: models supervised by reports remain competitive on specific diagnoses, NeuroVFM's advantage appearing as task diversity increases. At triage, sensitivity tops out at 86.5% (95% CI: 81.0–91.6) — 21 of 155 patients carrying a critical abnormality were missed, roughly one in seven, which rules out any autonomous use. The data come from a single health system; generalisation to other institutions and populations is not demonstrated. Finally, "surpassing GPT-5" concerns a precise neuroradiological triage task, against a generalist model never designed for it, and within a pipeline where clinicians render the final decision — it is not a comparison with a radiologist under real conditions. The public repository states it bluntly: research use only, this is not a medical device.

Causal chain

Convolutional networks applied to medical imaging (2010s) → rise of self-supervised foundation models on massive public corpora (2020-2023) → a blockage specific to neuroimaging: the skull is identifiable, the data stay out of the public domain → recognition that hospital archives already contain the raw material, uncurated → first wave of models trained on health-system data, supervised by the text of the reports (HLIP, PRIMA) → adaptation of the JEPA objective to 3D volumes (Vol-JEPA), which predicts in latent space and dispenses with labels entirely → training on 5.24 M volumes drawn from 566,915 studies → macro-averaged AUROC of 92.68% (CT) and 92.49% (MRI) across 156 tasks, above HLIP, NeuroMAE, DINOv3, BiomedCLIP and PRIMA → coupling to Qwen3-14B and a one-week silent prospective triage study (January 2026) → publication in Nature Medicine and release of the code (MIT) and the weights (July 2026) → remaining obstacles: external multi-site validation, and a sensitivity of 86.5% insufficient for autonomous use

Anecdote

The paradox of medical data holds in a single image: what barred neuroimaging from public corpora is not an abstract medical secret, it is the patient's face — a cranial scan reconstructs in 3D and restores recognisable features. The discipline richest in 3D volumes was therefore the poorest in shareable data. NeuroVFM does not solve this problem, it circumvents it: rather than exporting the images to a common corpus, it goes and learns where they already reside, in the 566,915 studies accumulated over twenty years by a single health system.

Legacy and current data

The field was already populated before this work: HLIP and PRIMA, both trained on health-system data and supervised by the text of the reports, constituted the main approaches of this family — and appear here as points of comparison. The question is therefore no longer whether foundation models are needed in imaging, but with which training signal to build them: the radiologist's report, or the image alone. NeuroVFM's own contribution is not a clinical product — no routine deployment is reported, and the repository explicitly excludes use in medical decision-making — but a proof of principle backed by an effective release: the code is under MIT licence and the weights are published on Hugging Face under CC-BY-NC-SA 4.0, a portion remaining conditional on an institutional request. The authors themselves place the sequel on the side of integrating temporal and multimodal data — pathology, genomics, longitudinal clinical outcome.

Sources

References verified during the fact-checking audit of 5 August 2026: these are the pages against which the claims in this bulletin were checked.

1. Health system learning enables generalist neuroimaging models — Nature Medicine (2026) — peer-reviewed, open access (CC BY 4.0), published 10 July 2026. DOI: 10.1038/s41591-026-04497-1
2. Health system learning achieves generalist neuroimaging models — arXiv preprint 2511.18640 (2025) — preprint, 23 November 2025. Earlier version of the same work, including neither the PRIMA and NeuroMAE baselines nor the prospective triage study.
3. Learning from routine health system data builds better neuroimaging AI models — Nature Medicine (2026) — Research Briefing of 31 July 2026, written by the authors of the study themselves: this is not an independent editorial commentary.
4. Official MLNeurosurg/neurovfm repository — GitHub — code (MIT), weights and the notice "research use only, not a medical device".

Transparency: the numerical values in this bulletin come from the peer-reviewed, open-access article and from the authors' official repository. The detailed composition of the cohorts and the per-diagnosis curves have not been re-verified line by line.