A generalizable deep learning system for cardiac MRI

Ethics assertion

Data assortment and analysis started following approval and waiver of consent by the Institutional Review Board of Stanford University (Protocol #60342, March 2021). Additional knowledge from the University of Pennsylvania had been sourced after retrospective assortment was deemed IRB exempt by the University of Pennsylvania Health System (Protocol #852332, November 2022).

Computational {hardware} and software program

MRI DICOM knowledge had been pre-processed on siloed HIPAA-certified n2 situations on the Stanford Nero–Google Cloud platform. Specifically, we used an 8-core digital machine with 52 GB of reminiscence and 6 TB of connected stable state storage. Data from the UK BioBank had been pre-processed on the Stanford Sherlock High Performance Computing Cluster, utilizing 24 CPU cores (Intel Xeon Gold 5118, 2.30 GHz). Anonymized reviews had been tokenized on a neighborhood encrypted desktop utilizing 48 CPU cores (AMD Threadripper, Lambda Computers). All fashions had been educated on the Stanford Sherlock High Performance Computing Cluster utilizing servers with 4× Nvidia A100 GPUs, every with both 40 GB or 80 GB VRAM, and 64 CPU cores (AMD Epyc). External validation on knowledge from the University of Pennsylvania was carried out on the Penn CUBIC High Performance Computing Cluster on a single Nvidia A40 GPU with 10 CPU cores. Additional exterior assessments befell on the Penn Advanced Research Computing Center (PARCC) Betty cluster, on a single Nvidia Blackwell B200 GPU with 10 CPU cores. Hyperparameter optimization experiments had been run on servers with quite a lot of GPU assets (Nvidia V100, 32 GB VRAM; Nvidia H100, 80 GB VRAM; Nvidia A100, 40 GB/80 GB VRAM; Nvidia P100, 32 GB VRAM; Nvidia Blackwell RTX 6000 MaxQ, 96 GB VRAM). We used the PyTorch deep-learning library (v.1.11.0) and the pytorch lightning framework (v.1.8.6)⁷⁰. Major Python packages used on this work embody numpy (v.1.21.2), pydicom (v.2.0.0), transformers (v.4.4.2) and stanza (v.1.5.0).

Datasets

Specifics of the pre-processing pipelines for each the MRI scans and the free-text reviews are detailed within the ‘Dataset pre-processing’ part of Supplementary Information. Briefly, from every distinctive MRI examine, related scans had been extracted (4CH, 3CH, 2CH and SAX cine sequences) as 4D arrays and saved inside a single hdf5 file. Free texts from the reviews had been segmented into particular person sentences utilizing the stanza pure language processing pipeline, tokenized utilizing the usual BERT auto-tokenizer, and the ensuing anonymized numeric arrays had been saved in a single listed json file^27,71. Across the pre-training datasets, fine-tuning datasets, exterior take a look at datasets and the UK BioBank, we included 65,492 people with ~550,156 distinctive movies throughout completely different view planes and cross-sections.

Clinical CMR dataset

The whole medical CMR dataset comprised 19,122 distinctive people. Cardiac MRI scans had been sourced from 17,088 particular person sufferers from a consortium of educational hospital techniques based mostly within the United States (Stanford Healthcare, UCSF, Medstar). Cine MRI scans had been procured through Bunkerhill Health (San Francisco, CA) as de-identified DICOM information, and related radiology reviews had been sourced as a single csv file (IRB Protocol #60342, March 2021). The whole pre-training dataset consisted of 293,110 distinctive 4CH, 3CH, 2CH and SAX movies. The scans had been carried out as a part of routine medical apply and reviews had been generated by board-certified physicians with particular experience in cardiac MRI. Sequences had been acquired on a variety of scanners together with these manufactured by Siemens (Siemens Healthcare), General Electric (GE) and Philips (Philips Healthcare), leading to substantial variance within the variety of frames per slice, imaging decision and reconstruction strategies (Supplementary Table 2). Demographics wherever possible are detailed in Supplementary Table 1. The knowledge had been first separated into pre-training and downstream datasets in an approximate 75:25 break up on the affected person stage. For the pre-training break up, we additional divided the info right into a coaching and validation set with an approximate 66:33 break up. Similarly, for downstream break up (meant for use as a labelled fine-tuning dataset for medical duties of curiosity) we additional divided the info into coaching, validation and testing datasets with an approximate 50:25:25 break up. We didn’t selectively exclude sufferers from this dataset; nevertheless, a fraction of the dicom information had been obtained as duplicates or had been corrupted and had been subsequently discarded. Supplementary Fig. 1 particulars the info splits and enumerates the excluded research at every stage. Cardiac MRI scans from an extra 2,033 particular person sufferers had been secured from the University of Pennsylvania Health System (IRB exempt, Protocol #852332, November 2022). These scans had been carried out as a part of routine medical apply and purchased on scanners manufactured by Siemens and GE. Data from the University of Pennsylvania had been used solely for exterior testing. While rule-based automated knowledge labelling strategies have been used up to now, these have been outdated by giant language fashions⁷². Building on our earlier work in exploring the zero-shot capabilities of enormous language fashions for medical textual content, we utilized a publicly obtainable giant language mannequin (medgemma3, 27-billion parameter variant) to parse free-text reviews generated as a part of routine medical apply into pre-defined ‘disease labels’ for the illness prognosis duties^73,74. Specific prompts, parameters, efficiency comparisons vs human annotators, and a collection of random non-curated reviews with critique of the deep-learning-predicted labels are detailed in Supplementary Fig. 5.

UK BioBank cardiac MRI cohort

Cine bSSFP-cardiac MRI sequences from 45,623 members had been sourced from the UK BioBank (Project ID: 71226). SAX sequences had been obtainable for 11,005 members and comprise stacks of 8–10 particular person slices. One slice was obtainable for every of the 4CH, 3CH and 2CH scans. This amounted to a complete of 257,046 distinctive movies obtainable for evaluation. Sequences within the UK BioBank had been acquired on a medical 1.5 Tesla scanner utilizing a standardized protocol (MAGNETOM Aera, Syngo Platform VD13A, Siemens Healthcare)⁵⁶. As a part of this protocol, the overwhelming majority of ventricular volumes and useful metrics had been calculated through automated contouring of the ventricular endocardium and epicardium with out guide professional qc^56,75. For fine-tuning and transfer-learning experiments to estimate LVEF%, we break up the UK BioBank dataset into an approximate 80:10:10 break up on the participant stage into coaching (n = 31,693), validation (n = 3,938) and hold-out take a look at datasets (n = 3,938).

ACDC dataset

The ACDC dataset is a publicly obtainable cardiac MRI dataset of 100 sufferers from the University Hospital of Dijon, France²⁸. Each SAX sequence was paired with patient-level non-overlapping labels (n = 20 every) for hypertrophic cardiomyopathy, earlier myocardial infarction, dilated cardiomyopathy, irregular proper ventricles and regular controls. The scans had been acquired on both a 1.5 Tesla (Siemens Area, Siemens Healthcare) or 3.0 Tesla (Siemens Trio Tim, Siemens Healthcare) scanner with a traditional SSFP sequence in breath maintain and gating.

Kaggle Data Challenge dataset

The 2015 Kaggle Data Science Bowl launched knowledge from 700 sufferers compiled by the National Institutes of Health and the Children’s National Medical Center, and was on the time, an order of magnitude bigger than any cardiac MRI dataset beforehand described. Patients had been recruited from the United States and scans had been carried out within the Washington DC space. While demographic splits from the dataset will not be obtainable, the unique knowledge had been sourced from a number of hospital techniques throughout a spread of age teams containing each regular and diseased hearts. The competitors closed on 14 March 2016, however knowledge from 697 circumstances stay publicly obtainable in DICOM format³⁹. 2CH, 4CH and SAX cine sequences had been obtainable for use, together with professional annotations for left ventricular end-systolic and end-diastolic volumes. The entirety of the obtainable dataset was used for exterior validation as is, with none high quality management.

Neural community architectures

We examined imaginative and prescient encoder architectures together with 3D residual convolutional networks and video imaginative and prescient transformers. We settled on utilizing an implementation of a multiscale imaginative and prescient transformer (mViT) with 36.3 million trainable parameters as our video encoder after experiments displaying superior generalization and embedding high quality²⁶. Vision transformers have just lately emerged as a performant various to convolutional neural networks, particularly within the setting of large-scale self-supervised pre-training^76,77. Vision transformers retain the skip connections seen in conventional convolutional networks, however are additionally in a position to attend to native and world options of a picture in earlier phases⁷⁸. The mViT structure is a imaginative and prescient transformer designed particularly for video knowledge, which foregoes the successive layers of convolutional operations seen in typical convolutional neural networks, for a single convolutional layer to divide the enter video right into a linear collection of overlapping cubes. These linear components are processed by 16 layers of stacked transformer modules, permitting the community to successfully attend to distant enter options. Specific to the mViT structure is a sequential collection of pooling and scaling operations that successfully allow the community to take care of easy visible options at excessive decision in early layers, adopted by advanced high-dimensional relationships at a coarser decision in deeper layers. As a consequence, in contrast with different extensions of 2D-image transformers to the video area, mViT by design has a stronger temporal inductive bias. While extra computationally costly than comparable convolutional networks, mViT is extra environment friendly than comparable imaginative and prescient transformers, requiring remarkably much less pre-training knowledge to realize state-of-the-art outcomes on typical motion recognition datasets. Finally, in comparison with conventional convolutional neural networks, mViT has proven superior efficiency on giant video motion recognition datasets regardless of fewer trainable parameters²⁶.

We elected to make use of a pre-trained BERT mannequin for our textual content encoder²⁷. Unlike different language fashions which have come earlier than it, BERT is educated utilizing a ‘bidirectional’ method, the place the mannequin is educated to be taught the construction and context of human language by attending to sentences in each the left-to-right and right-to-left route. Specific particulars of the pre-training strategies for BERT are detailed within the unique paper²⁷. We used a 12-layer variant of BERT base, with 12 consideration heads and a hidden dimension of measurement 786, with a complete of 110 million trainable parameters. We examined a mix of various pre-trained weights together with these from the unique publication, weights tremendous tuned on the MIMIC dataset, and weights from a mannequin educated on biomedical abstracts from PubMed with a customized vocabulary of 30,522 tokens^79,80 (Supplementary Fig. 3).

Pre-training framework

We constructed on earlier makes an attempt at learning visible representations utilizing naturally occurring pairing of 2D medical imaging and textual knowledge, extending these ideas to the spatiotemporal video-like nature of cardiac MRI scans^{14,15,16,17,19,20}. Two parallel encoders had been educated: one for processing the MRI cine sequences and the opposite for processing the subsampled textual content from paired radiology reviews. Self-supervised transformer networks specifically have proven superior efficiency on downstream duties in comparison with conventional supervised strategies^76,81,82. We used an implementation of mViT with Kinetics-400-initialized weights for the imaginative and prescient encoder, and a pre-trained BERT mannequin for the textual content report encoder. Specifically, we utilized weights from BERT pre-trained on abstracts of biomedical publications on PubMed with a customized vocabulary⁷⁹. Data from 8,513 sufferers (9,427 scans and paired reviews) had been used for coaching, and a separate set from 4,194 sufferers (4,646 scans and paired reviews) had been used for validation.

We employed randomized sequential knowledge augmentation schemes (AugMix) to stochastically pattern and layer a collection of chained transformations together with however not restricted to resizing, solarization, shear, translate and random rotation of movies within the spatial dimensions, all whereas preserving the identical augmentations alongside the temporal dimension for temporal consistency⁸³. Uniform temporal subsampling tremendously improved downstream efficiency and generalizability. We augmented the radiology reviews by randomly sampling 5 sentences from your complete report for every scan per coaching step. The output of every encoder was handed by a one-layer linear projection head to yield a pair of 512-dimensional embeddings. These low-dimensional, 512-dimensional embeddings are a compressed numeric illustration of the data contained throughout the enter MRI scan and paired textual content report.

Previous work has additionally proven the significance of enormous batch sizes for efficient contrastive illustration learning⁸¹. To examine this, we pre-trained fashions with a batch measurement of 16, 32 and 128 video–textual content pairs. For the UK BioBank LVEF prediction job, we discovered that tremendous tuning from the larger-batch-size pre-trained fashions led to improved downstream outcomes (Supplementary Fig. 2). While computational budgets didn’t permit for an intensive hyperparameter search with the bigger batch sizes, we notice that the downstream advantages didn’t seem like clinically vital for this particular job. Nonetheless, this stays an space for further future exploration.

Vision-only self-supervised strategies could be difficult to include the place scans from a number of visually distinct view planes exist for the identical affected person. We centered our efforts on text-to-video approaches given the success with textual content supervised visible illustration learning throughout radiology and motion recognition^14,16,84,85. We thought of approaches equivalent to Contrastive Language-Image Pre-Training (CLIP); nevertheless, these are restricted by a brief context size appropriate for captions somewhat than the bigger, largely unstructured paragraphs which might be typical of cardiac MRI reviews⁸⁵. Similar to the work of ref. ¹⁶, we elected to make use of an uneven bidirectional implementation of the InfoNCE loss to maximise mutual data between every MRI video–textual content report pair^16,22. The contrastive losses used are basically log-loss of an n-way classifier to foretell the proper pair of MRI scan and report (the place n = batch measurement). The first loss operate is a video-to-text contrastive loss for the ith pair, the place v_i represents a video embedding and u_i represents a textual content embedding of the ith video–textual content pair. N right here represents the variety of video–textual content pairs in a complete batch being evaluated.

The second loss operate is a equally structured text-to-video contrastive loss. The tunable temperature parameter (left({tau }proper)) controls the power of penalties on onerous detrimental pairs sampled throughout coaching⁸⁶.

The remaining loss was outlined as a weighted mixture of the 2 losses averaged over all constructive video–textual content pairs in every batch of knowledge. The scalar weight is given by λ.

We moreover carried out a ‘flooding’ regularization approach to stop the coaching loss (({mathscr{L}})) to method zero⁸⁷. We set the flood stage (scalar worth given by b) to a coaching lack of 0.05 to permit for higher generalization. The remaining loss ((widetilde{{mathscr{L}}})) is thus given by:

The particular pre-trained weights and vocabulary used for initializing the textual content encoder, batch measurement, augmentation scheme, InfoNCE temperature parameter and flood regularization had been essential for mannequin convergence⁸⁸. The remaining mannequin was pre-trained with a batch measurement of 32 per GPU, for 600 epochs. The first 6 layers of the BERT textual content encoder was frozen, and your complete community was educated with a learning fee of 4.8 × 10⁻⁵ utilizing the AdamW optimizer with weight decay set to 1 × 10⁻⁶ and eps set to 1 × 10⁻⁸. We decayed the learning fee by an element of 0.1 at 300 epochs. Checkpoints had been saved each 10 epochs throughout the pre-training course of and the final checkpoint was used for tremendous tuning on downstream medical duties. The whole time taken for pre-training was 13 days and 14 h (4 × 80 GB Nvidia A100 GPUs). The skill of the imaginative and prescient transformer encoder to cluster completely different illness situations with none further specific supervised coaching was visualized utilizing the uniform manifold approximation and projection (UMAP) algorithm initialized utilizing default values⁸⁹.

Multi-instance self-attention and downstream analysis

A gated multiview self-attention community was educated to assign an consideration worth (a_ok) to every MRI view embedding produced by the primary imaginative and prescient encoder^13,31. For every embedding inside a bag of ok embeddings, a excessive rating after softmax activation (close to 1) signifies {that a} specific MRI view aircraft is extremely informative for the downstream diagnostic job. Conversely, a low rating (close to 0) signifies that the MRI view aircraft has little to no diagnostic worth. For classification duties, every enter embedding was moreover handed by a LayerNorm operate earlier than a ahead move into the self-attention blocks (Supplementary Fig. 6)⁹⁰ (w^T, consideration scoring vector; V, view stage weight parameters; U, view stage weight parameters; h_j, low-dimensional embeddings; ⊙, element-wise product; tanh, tanH activation operate; sigm, sigmoid activation operate; N, whole variety of MRI view embeddings for a specific examine).

We made use of an consideration pooling mechanism to common the embeddings from all MRI views weighted by their predicted consideration scores, to return a single 512-dimensional embedding. This embedding may be handled as a ‘feature representation’ of your complete MRI examine for a selected downstream job of curiosity. For every downstream classification job of curiosity, we used a binary classification head with a sigmoid activation operate, as illness labels are normally not mutually unique within the setting of cardiovascular issues. For downstream duties that contain regression of a numeric variable, we changed the binary classification head with a single output neuron with a linear activation operate.

LVEF regression job

We examined two modes of coaching for LVEF% prediction: (1) ‘fine tuning’ the place the final linear layer of the imaginative and prescient encoder and the classifier head are trainable and (2) ‘transfer learning’ the place the imaginative and prescient encoder, linear layer and classifier heads are all trainable. ‘Fine tuning’ permits for a point of flexibility in the way in which embeddings are generated however retains the imaginative and prescient encoder frozen to utilize the discovered representations. With the system set to ‘transfer learning’, the community begins from the discovered representations; nevertheless, for the reason that total community is unfrozen, it’s doable to ‘overwrite’ these parameters with every new replace of the coaching course of. For these experiments, we initialized the imaginative and prescient encoder with the contrastive pre-trained weights (ours) or Kinetics-400 weights (baseline), onto which we connected the regression head as described above.

We tremendous tuned our pre-trained checkpoints with 32-bit precision utilizing the AdamW optimizer, with a learning fee set to 1 × 10⁻⁴ and default worth of 0.01 for weight decay. We explored completely different augmentation schemes and achieved superior validation efficiency with AugMix on restricted hyperparameter sweeps with 10% of the coaching knowledge⁸³. For all experiments involving tremendous tuning with subsets of accessible knowledge, we used a guide seed worth for random subsampling to make sure reproducibility of outcomes. We made use of all obtainable 4CH, 2CH, 3CH and a random subsample of fifty% of SAX views per examine, with no guide screening for high quality management. We elected to coach our regression fashions with a Huber loss operate, and we used imply squared errors and imply absolute errors as efficiency metrics⁹¹. We moreover calculated the AUROC for diagnosing coronary heart failure on the premise of an LVEF cut-off of 40%. We educated fashions for a most of 100,000 steps on GPUs with no less than 16 GB of VRAM every. For experiments described in Fig. 3a,d, configuration information had been generated for every experimental setup and had been educated in parallel throughout quite a few GPUs on Stanford Sherlock.

Disease classification job

We outline each ‘fine tuning’ and ‘transfer learning’ as above, and used the identical community structure initialized with Kinetics-400 weights as our baseline. We tremendous tuned our pre-trained checkpoints with the identical total settings as described above for the regression duties, besides for using a weight decay worth of 5 × 10⁻⁴ and the addition of a LayerNorm operate for the embeddings earlier than a ahead move by the multi-instance self-attention modules to help with convergence. We empirically used AugMix for our knowledge augmentation technique, given the successes famous above. We made use of all obtainable 4CH, 2CH, 3CH and SAX views per examine with no high quality management or screening. We utilized a binary cross-entropy loss operate with a sigmoid activation weighted by a scalar multiplier equal to the proportion of constructive vs detrimental lessons for every illness (calculated utilizing the interior coaching set prevalences). We used the AUROC as a efficiency metric, given the appreciable class imbalance of constructive and detrimental lessons⁹². For every illness label of curiosity, we educated fashions for 24 epochs on GPUs with no less than 24 GB of VRAM. For experiments described in Fig. 4a,b, configuration information had been generated for every experimental setup and had been educated in parallel throughout quite a few GPUs on Stanford Sherlock. External take a look at knowledge had been evaluated on the Penn CBICA cluster on a single Nvidia A40 GPU with 40 GB VRAM, and on the PARCC Betty cluster on a single Nvidia Blackwell B200 GPU with 180 GB VRAM. In addition to the losses and metrics, we saved predicted possibilities and relative self-attention scores for every view for downstream processing and statistical analyses.

Statistical analyses

We used the torchmetrics (v.1.0.1) bundle to calculate MSE and MAE for regression duties, and AUROC values for classification duties throughout the coaching and validation loops. We moreover manually calculated AUROCs as empirical curves within the sensitivity and specificity area, computed from predicted possibilities generated by our fashions⁹³. To examine the efficiency of fine-tuned classifier fashions (that’s, contrastive pre-trained vs baseline), we calculated non-parametric confidence intervals on the AUROC utilizing DeLong’s technique (paired)⁹⁴, following which P values had been computed for the imply distinction between AUROC curves. Additional analyses had been carried out to calculate the accuracy for every diagnostic label at completely different thresholds (optimizing for Youden’s statistic, a sensitivity of 0.90 or a specificity of 0.90). Differences between predicted LVEF% values and floor reality had been assessed utilizing Bland–Altman plots. Statistical analyses had been carried out and graphs had been plotted utilizing R (v.4.1.0); main packages used included pROC (v.1.17.0), ggplot2 (3.3.5) and blandr (0.5.1). The on-line test-set leaderboard webapp was created utilizing shiny (1.8.1).

Attention visualizations

For each enter scan, we output the uncooked self-attention tensors from every head of every layer of the MRI imaginative and prescient encoder throughout analysis and processed them to yield 65 separate consideration warmth maps. As described earlier, the spatiotemporal decision was diminished with every successive stage within the mViT structure; the self-attention tensors had been diminished from an preliminary spatiotemporal decision of 8 × 56 × 56 on the first layer, to eight × 7 × 7 at the previous couple of layers. We saved solely the eye values from the output patches for the needs of visualization, and spatiotemporally interpolated these tensors again to a measurement of 16 × 224 × 224 through nearest-neighbour resampling. These arrays had been exported to mp4 information utilizing imageio and the ffmpeg library (Supplementary Figs. 17 and 18). Aside from the self-attention warmth maps for every enter video, we additionally computed the uncooked self-attention values from the multi-instance classifier head for related downstream duties. After every scan was handed by the imaginative and prescient encoder, the resultant embedding was assigned a leaned uncooked self-attention rating throughout the multi-instance self-attention modules. We calculated the relative variations in self-attention scores throughout completely different view planes for every illness label. These relative self-attention values had been visualized as 2D warmth maps as proven in Fig. 4c. The multi-instance classifier head self-attention scores confirmed that the community learns to differentially prioritize view planes for completely different medical duties.

Reporting abstract

Further data on analysis design is offered within the Nature Portfolio Reporting Summary linked to this text.