Medical Imaging Models Disagree Across Scanner Types: Why?

In the realm of medical https://seo.edu.rs/blog/counterfactual-augmentation-for-disputed-inputs-how-does-it-work-11189 imaging, deploying machine learning classifiers to assist diagnosis is nothing short of revolutionary. However, a pervasive challenge lurks beneath the surface: models often disagree across different scanner types. This phenomenon, driven by scanner domain shift, complicates clinical deployment and undermines trust. In this post, we'll unpack why these disagreements arise, explore how disagreement rate and predictive entropy serve as crucial risk indicators, and examine key factors like distribution shifts, artifact memorization, and loss function tradeoffs that shape model behavior.

Understanding Scanner Domain Shift in Medical Image Classification

Medical images are captured using various scanners from different manufacturers, with distinct hardware settings and imaging protocols. This diversity creates scanner domain shifts — variations in image characteristics that do not necessarily reflect underlying pathology but do impact model predictions.

Differences in image resolution, contrast, or noise patterns
Variations in reconstruction algorithms and image artifacts
Calibration and scanner aging effects

Such domain shifts cause even state-of-the-art medical image classifiers to produce inconsistent outputs when interpreting images from different scanners, revealing cracks in their generalization capabilities.

Disagreement as a High-Signal Risk Indicator

One effective way to detect risk and uncertainty in model predictions across scanner domains is by examining disagreement rates among ensemble models or multiple prediction heads. Disagreement rate measures how often different models or checkpoints produce conflicting labels on the same input.

Why is disagreement so valuable?

Flags uncertain or ambiguous cases: When models disagree, it usually signals edge cases or inputs lying near decision boundaries.
Signals distribution shifts: A surge in disagreement on images from a new scanner type often points to domain shift that the model struggles to handle.
Predictive entropy enhancement: Disagreement correlates with increased predictive entropy— a measure of uncertainty in the predicted probability distribution — further quantifying risk.

Leveraging disagreement rate alongside predictive entropy provides a complementary duo to monitor model uncertainty beyond simple accuracy metrics that obscure the true risk.

What Happens on the Worst Day in Prod?

My go-to question here: What happens on the worst day in production? Imagine a batch of images from an unfamiliar scanner floods the system. If the disagreement rate spikes drastically with elevated predictive entropy, that's a red flag— potentially indicating many misclassifications or unexpected artifacts.

Edge Cases and Distribution Shift in Scanner Domains

Edge cases often coexist with distribution shifts, compounding the model's challenge. In medical imaging, edge cases may manifest as:

Rare pathologies or atypical presentations
Images with subtle abnormalities masked by noise or artifacts
Unusual patient demographics underrepresented in training data

Domain shifts induced by different scanners distort image textures, brightness, or noise patterns. Models trained predominantly on a limited set of scanners internalize these domain-specific features, inadvertently leveraging spurious correlations rather than pathology-relevant signals.

Importantly, artifact memorization — where models learn to latch onto scanner-specific artifacts or noise as predictive cues — is a stealthy risk contributing to disagreement. When encountering images from a scanner with new artifact patterns, the model’s outputs can become unreliable, triggering disagreement spikes.

Data Gaps and Subgroup Coverage

Distribution shift and edge case challenges often stem from data gaps in subgroup coverage. Inadequate representation of certain scanner types or patient subgroups during training impairs model robustness. For example:

Few samples from scanner B in training data, resulting in poor performance on its images
Underrepresented ethnic or age groups whose anatomical structures differ subtly
Missing examples of rare diseases or specific image acquisition protocols

Closing these gaps requires curated datasets encompassing diverse scanner types and patient demographics to ensure the model experiences the full spectrum of variation during training.

Things Accuracy Hides

One of my persistent annoyances is teams that solely rely on test-set accuracy. Such aggregate metrics flatten critical nuances like:

Variance in per-scanner performance
Uncertainty in ambiguous edge cases
Performance degradation under scanner domain shift

counterfactual augmentation

High average accuracy can mask catastrophic failures on specific scanner types, misleading stakeholders about real-world reliability.

Objective Mismatch and Loss Function Tradeoffs

Many medical imaging classifiers optimize loss functions like cross-entropy that prioritize overall accuracy or likelihood but do not explicitly handle domain shifts or uncertainty calibration. This leads to several failures:

Overconfident probability scores: Poorly calibrated outputs erode trust and hinder risk-aware decision-making.
Objective mismatch: The loss function may steer the model to exploit spurious correlations (e.g., scanner artifacts), boosting accuracy but hurting generalization.
Lack of uncertainty modeling: Focusing purely on point predictions without incorporating predictive entropy or disagreement as learning signals misses opportunities to improve robustness.

Effective loss design should balance between accurate classification, uncertainty calibration, and domain invariance. Strategies include:

Incorporating auxiliary losses promoting domain-agnostic feature learning
Using ensemble methods to capture prediction variance
Augmenting training with data from diverse scanner types
Adopting Bayesian or evidential frameworks modeling predictive uncertainty explicitly

Mitigation Strategies and Future Directions

So, what can we do to tackle scanner domain shift and reduce disagreement? Here are some pragmatic approaches:

Data collection and augmentation: Curate balanced datasets spanning all relevant scanner types and use augmentation techniques mimicking scanner-specific variations.
Model calibration: Post-hoc calibration methods like temperature scaling reduce overconfident predictions, improving uncertainty quantification.
Ensembling and disagreement monitoring: Implement multi-model ensembles and track disagreement rates in production as a continuous risk indicator.
Domain adaptation: Use adversarial training or feature alignment to learn scanner-invariant representations.
Robust loss design: Incorporate uncertainty-aware losses and domain generalization objectives to prioritize reliable prediction over raw accuracy.
Human-in-the-loop systems: Flag high-disagreement or high-entropy cases for expert review, combining AI strengths with clinical oversight.

Conclusion

Medical imaging models disagreeing across scanner types is a direct manifestation of scanner domain shift, edge cases, and artifact memorization amplified by data gaps and objective mismatches. Rather than ignoring these discrepancies as mere noise, disagreement rates and predictive entropy offer actionable, high-signal windows into model risk — lighting the path toward safer AI in healthcare.

By embracing rigorous data curation, calibrated uncertainty estimates, and robust loss engineering, we can mitigate scanner domain shifts and unlock the transformative promise of medical image classifiers across diverse clinical settings.

References

Guo et al., "On calibration of modern neural networks," ICML 2017
Gulshan et al., "Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy," JAMA 2016
Zhao et al., "Domain Adaptation for Medical Imaging," IEEE TMI 2022
Jiang et al., "Predictive Entropy as an Uncertainty Measure for Medical Image Classification," MICCAI Workshop 2021