Pathologists—the people with medical licenses, specialist training, and responsibility for the final report—are now being asked to assess not only an algorithm’s output, but also the algorithm’s account of how it reached that output. That second-order task is where deployment friction actually lives. It is also why explainable AI has moved from a research curiosity to a clinical bottleneck.
The scale of the underlying problem is not in dispute. WHO/IARC tallied roughly 19.29 million new cancer cases and 9.96 million cancer-related deaths globally in 2023. Vendors are not wrong that AI-assisted screening is one of the few realistic ways to increase diagnostic capacity without simply asking pathologists to read more slides, faster. The difficulty begins when a model emits a probability score and a human has to defend that score at a tumor board, in a multidisciplinary review, or during a laboratory validation process.
Black-box deep networks have spent a decade producing impressive AUC values and roughly the same amount of time failing to convince many pathology departments that their internal logic is sufficiently auditable. Interpretability is the proposed bridge. Whether the bridge can carry a clinical workflow, rather than merely support a convincing product demonstration, is the question.
The Black Box Dilemma in Computational Pathology
Opacity is not just a marketing problem. It is a workflow problem. When a deep convolutional network flags a region of interest on a whole-slide image, the operator at the diagnostic workstation has no native way to interrogate the chain of reasoning that produced the flag. The model has learned, during training, a representation of malignancy distributed across millions of parameters and dozens of layers. Nobody—including the vendor’s engineering team—can read those weights in the same direct way a pathologist reads an H&E slide.
This matters because clinical decisions are not made on probabilities alone. They are made on justifications. A pathologist who assigns a Gleason pattern 4 needs to identify cribriform architecture, poorly formed glands, or another relevant morphological feature and explain the call to a colleague. An AI system that returns an 85% probability of adenocarcinoma may be useful as a prioritization signal, but the score does not by itself explain what the pathologist should inspect or why the prediction deserves confidence.
That distinction is central to AI model interpretability in digital pathology diagnostics. A model can be statistically well calibrated and still be clinically opaque. It can perform well on a held-out test set while relying on staining artifacts, tissue folds, scanner-specific characteristics, or preparation patterns that happen to correlate with the target diagnosis in the development data. A visual explanation may reveal some of those shortcuts. It may also create a false sense of reassurance if the highlighted region looks plausible but is not causally important to the prediction.
The gigapixel scale of whole-slide imaging makes the problem harder. A prostate biopsy slide scanned at 40x can exceed 100,000 pixels per side, while most deep-learning pipelines operate on smaller image patches tiled across that canvas. The output reaching the pathologist is therefore a stitched-together mosaic of patch-level predictions, attention weights, or region scores. Tracing a slide-level diagnosis back to the morphology that influenced it requires an additional visualization layer.
That layer is what the field usually calls explainable AI in histopathology. The label covers several different approaches, and they should not be treated as interchangeable. Some methods highlight image regions after a model has made its prediction. Others build relevance weighting into the model architecture. Still others create modified versions of the tissue and test whether the prediction changes. Each answers a different question, with a different degree of reliability.
Visualizing Diagnostic Logic: Grad-CAM and Attention Maps
The workhorse of post-hoc interpretability in computational pathology is Grad-CAM, or Gradient-weighted Class Activation Mapping. The mechanism is conceptually simple: Grad-CAM calculates the gradient of a target class score with respect to the activation maps in the last convolutional layer, weights those activations according to the gradients, and produces a coarse localization heatmap. The result indicates which areas were associated with the model’s output. It does not reproduce the model’s full reasoning process.
That distinction matters. A heatmap is an attribution, not a transcript of the classifier’s internal logic. It is usually generated after the model has been trained, without changing the underlying weights. This makes Grad-CAM attractive to laboratories that want an interpretability layer without rebuilding an established classifier from scratch. It also makes the method vulnerable to the limitations of post-hoc explanation: the map is an approximation of relevance, not a direct readout of a human-understandable decision path.
The variants have proliferated accordingly. Grad-CAM++ refines the weighting scheme and can be more useful when several instances of the target class appear in one image. SHAP, or SHapley Additive exPlanations, takes a game-theoretic approach and assigns contribution values to features or input components. In pathology, the practical implementation of SHAP depends heavily on the model and the input representation. A patch classifier, a multiple-instance learning model, and a tabular molecular model do not produce explanations in the same way, even when they are all described as explainable AI.
A parallel track exists inside the model architecture itself. Transformer-based multiple-instance learning frameworks such as TransMIL incorporate attention into the slide-level classifier. Individual tiles receive attention weights, which can be displayed as a relevance map across the whole slide. This is often described as intrinsic interpretability because the weighting mechanism is part of the model rather than an external explanation added afterward.
Intrinsic attention is not automatically a faithful explanation, however. An attention score can show that a tile contributed to the aggregation process without proving that the score reflects the feature a pathologist considers diagnostically meaningful. The map still requires validation. The advantage is architectural coherence; the cost is that the laboratory is committing to a particular model family and its assumptions.
| Method | Type | Requires retraining | Typical output | Honest caveat |
|---|---|---|---|---|
| Grad-CAM | Post-hoc saliency | No | Coarse class heatmap | Spatially approximate and generally tied to late network features |
| Grad-CAM++ | Post-hoc saliency | No | More refined class-activation map | Can improve handling of multiple instances but remains an approximation |
| SHAP | Post-hoc attribution | No | Feature contribution scores | Computationally demanding, especially for large image inputs |
| TransMIL attention | Intrinsic attention | Yes, as part of the architecture | Tile-level relevance weights | Attention is not automatically a faithful causal explanation |
| MoPaDi counterfactuals | Generative and multiple-instance learning approach | Yes, as part of the framework | Morphed tissue regions and prediction changes | More computationally intensive and still limited in routine clinical validation |
That last column is the one vendors tend to leave out of product demonstrations. Grad-CAM heatmaps can be noisy. SHAP at whole-slide resolution may be slow enough to affect turnaround time. Saliency maps can highlight visually plausible areas that were not decisive, or miss features that a pathologist would consider central. A map that looks convincing at one magnification may become ambiguous when the user zooms in.
The proper use of these methods is therefore narrower than the marketing language suggests. They can help a pathologist locate suspicious tissue, identify a potential shortcut in the model, compare model behavior across cases, or design a targeted error analysis. They should not be treated as independent evidence that the prediction is correct. The explanation must be tested against known annotations, representative cases, scanner variation, staining variation, and the actual decisions the tool is intended to support.
A heatmap can show where a model placed its attention. It cannot, by itself, show that the highlighted tissue caused the diagnosis.
A useful validation exercise is to ask several related questions rather than relying on a single visual impression:
- Does the highlighted region overlap with a clinically relevant annotation?
- Does the explanation remain stable when the slide is rescanned or the image is presented at a different resolution?
- Does a small, clinically irrelevant change cause a large shift in the map?
- Does masking or perturbing the highlighted tissue materially change the prediction?
- Can pathologists interpret the display quickly enough for it to fit the intended workflow?
These are not merely interface questions. They determine whether a visualization is a useful control on black box algorithms in pathology or just an attractive overlay.
Counterfactual Reasoning: Testing AI Predictions with MoPaDi
Heatmaps answer a version of the question “where did the model look?” They do not answer the question a pathology department often needs answered: what would the model see if the tissue were different?
That second question belongs to counterfactual reasoning. Instead of only highlighting an existing region, a counterfactual method modifies the input and observes whether the prediction changes. In principle, this moves interpretability closer to a testable hypothesis. If the model’s classification changes after a meaningful alteration in glandular architecture, lymphocytic infiltration, or another relevant feature, the pathologist has more information than a static saliency map provides.
MoPaDi—Morphing histoPathology Diffusion—is an example of this direction. The framework pairs a diffusion-based generative component with a multiple-instance learning classifier and uses a learned latent space to manipulate histopathology images along selected morphological dimensions. A generated counterfactual version of a slide can then be compared with the original. The important object is not the altered image by itself, but the relationship between the morphological change and the classifier’s response.
In microsatellite instability prediction, counterfactual approaches have been used to examine how predictions relate to altered lymphocytic infiltration and modified gland morphology. This produces a more clinically legible question than a simple relevance map. Instead of asking only which region received a high score, the reviewer can ask whether a change in a recognizable tissue pattern is associated with a corresponding change in the model’s output.
There is a catch, and it is a substantial one. A generated image is not automatically a faithful version of a plausible biological specimen. Diffusion models can produce visually convincing tissue while introducing artifacts or changing several properties at once. If the generated counterfactual alters gland shape, texture, staining appearance, and local context simultaneously, the prediction shift cannot be attributed confidently to one morphological factor. The method may be useful for probing model behavior without offering a clean causal experiment.
The operational cost is also non-trivial. Diffusion-based counterfactual generation requires substantial computation and a framework that is more complex than adding a saliency layer to an existing classifier. Few clinical laboratories are likely to run such a system as part of routine production today. The more realistic near-term use is model development, error analysis, and prospective evaluation of whether the generated changes correspond to features pathologists recognize as diagnostically meaningful.
The conceptual move still matters. Interpretability research is moving away from the assumption that pointing at a region is sufficient. The stronger question is whether the model’s decision responds to tissue changes in a direction that makes sense clinically. That is closer to testing a model’s reasoning, although it should not be confused with proving that the model has human-like understanding of morphology.
Counterfactual reasoning also exposes a weakness in many current interpretability claims: explanations are often evaluated for visual appeal rather than clinical utility. A useful explanation should help a qualified user detect an error, decide what to review, or understand the limits of the output. If it cannot change a review decision, expose a shortcut, or clarify a disagreement, its value may be mostly cosmetic.
Regulatory Requirements for End-to-End Clinical Validation
Heatmaps and counterfactuals belong largely to the research and development conversation. AI-enabled diagnostic tools that enter clinical use operate within a different frame. The FDA evaluates software functions as part of regulated medical devices, and an AI diagnostic tool is considered in the context of its intended use, inputs, outputs, and complete clinical workflow. That means the algorithm cannot be separated conceptually from the image acquisition and processing steps on which its performance depends.
The narrower, more defensible point is this: AI diagnostic tools—and explanation features included within their intended use—should be validated as part of the complete end-to-end system. The exact regulatory pathway and evidence expectations depend on the device, its intended use, its risk profile, and the claims made about its performance. It is not accurate to suggest that the FDA ignores software or evaluates only a physical laboratory pipeline. Software functions are themselves part of regulated medical devices. The practical issue is that their performance is not established in a vacuum.
An explanation feature is part of the clinical product when the product asks users to rely on it. Its validation belongs in the same end-to-end conversation as the prediction.
For a computational pathology system, end-to-end evaluation may involve tissue preparation, staining, slide scanning, image quality checks, preprocessing, tiling, inference, display, and the pathologist’s interaction with the result. If a heatmap is presented as a way to locate tumor or support a diagnostic decision, the laboratory needs evidence that the display behaves acceptably across the conditions covered by the intended use.
That does not mean every visualization must meet an identical performance threshold to the underlying classifier. A heatmap and a classification output serve different functions and may require different measures. It does mean that the explanation cannot be treated as automatically trustworthy simply because the classifier has been evaluated. The laboratory should define what the explanation is supposed to do, how that function will be assessed, and what happens when the visualization is unstable or misleading.
Scanner variation makes the point concrete. A heatmap that appears to localize tumor accurately on one scanner and magnification may be spatially shifted, blurred, or less informative on another. Staining protocols, section thickness, tissue folds, focus quality, compression, and preprocessing can all influence the image presented to the model. If the intended-use environment includes those variations, they belong in the validation strategy.
The same principle applies to the human factors of interpretability. A visualization may be technically reproducible but clinically unusable if it obscures the original tissue, takes too long to load, uses an unintuitive color scale, or encourages users to over-trust a highlighted area. A pathologist should be able to distinguish the raw slide from the model overlay and understand whether the display is showing confidence, attention, anomaly detection, or a different quantity altogether. “Red means important” is not a sufficient explanation of what the system has calculated.
For laboratories operating laboratory-developed tests under CLIA, the regulatory setting is different from the FDA device pathway, but the validation problem does not disappear. Internal validation committees still have to decide whether the tool performs as intended, whether users can interpret its output, and whether changes in scanners, staining, software versions, or model thresholds require re-evaluation. The absence of a federal reviewer in a particular workflow does not make a weak explanation layer clinically sound.
This is where clinical validation of AI diagnostic models and interpretability validation meet. The model’s discrimination, calibration, robustness, and failure modes need to be considered alongside the behavior of the explanation feature. A useful record should make clear:
- what the model predicts and for which specimen types;
- what the explanation is intended to communicate;
- which image and workflow conditions were tested;
- how explanation quality or stability was assessed;
- how users are instructed to respond when the prediction and the display disagree;
- and which changes trigger additional validation.
There is no single universal metric that turns a saliency map into a validated clinical explanation. Overlap with annotations can be informative, but annotations are not always complete and may not capture the distributed features used by a model. Perturbation tests can reveal sensitivity, but an artificial perturbation may not represent a plausible tissue change. Reader studies can assess usability, but a user’s confidence is not proof that the explanation is faithful. The evidence has to match the function claimed for the feature.
Balancing Algorithmic Transparency with Pathologist Oversight
So does explainable AI actually bridge the trust gap, or does it merely dress the black box in a readable costume?
The honest answer—the one worth giving a lab manager in a Tuesday procurement meeting—is that interpretability techniques bridge part of the gap, and only the part that lies between the model and the human reviewing the slide. They can make a prediction easier to inspect. They can expose obvious shortcuts, focus review on a suspicious region, and support a structured conversation about why the algorithm failed. They do not eliminate the need to validate the scanner, the staining process, the image pipeline, the intended-use population, or the human-computer interaction.
Nor do they transfer responsibility for the diagnosis to the model. Pathologists still own the clinical interpretation. AI still proposes, prioritizes, or flags. An explanation display can support oversight, but it cannot turn a probabilistic system into an autonomous expert.
The distinction between interpretability and transparency is useful here. Interpretability concerns how a particular prediction is explained: which tiles, features, or changes were associated with the output. Transparency is broader. It includes information about training data, intended use, exclusions, performance across relevant populations, known failure modes, version changes, and the conditions under which the system should not be trusted. A beautiful heatmap with no account of domain shift is not a transparent product.
This is also why trusting AI predictions in tissue analysis should not be framed as a psychological conversion. The goal is not to persuade pathologists to feel comfortable with an opaque model. The goal is to give them enough reliable information to decide when the output is useful, when it requires closer review, and when it should be disregarded. Trust, in a clinical setting, should be conditional and earned through evidence.
A practical deployment decision can be organized around the relationship between the prediction, the explanation, and the workflow:
1. The prediction must have a defined clinical role. Screening, triage, quantification, classification, and decision support create different expectations. A tool that prioritizes slides for review does not need to make the same claim as a tool that generates a diagnostic classification.
2. The explanation must have a defined function. If it is intended to point users toward suspicious tissue, evaluate localization. If it is intended to support a classification decision, assess whether it improves review without increasing false confidence. If it is only a development aid, do not present it as a validated clinical feature.
3. The complete image pathway must be represented. Validation should reflect the scanners, staining variation, magnifications, tissue types, and preprocessing steps covered by the intended use—not just the cleanest development set.
4. Failure should remain visible. A low-confidence or unstable explanation should not be rendered with the same visual authority as a consistent one. Users need a way to recognize that the tool is outside its reliable operating conditions.
5. The human review step must be tested. The relevant question is not whether a researcher finds a map interesting. It is whether the intended users can use it accurately, efficiently, and without being pushed toward inappropriate certainty.
This is the point at which transparency in computational pathology becomes more than a model-design preference. It becomes a property of the entire service: the specimen, the scan, the software, the interface, the explanation, and the professional judgment applied at the end.
The clinical reality is therefore less dramatic than the product language. Interpretability will not make black-box algorithms transparent in the ordinary human sense. It can make their behavior more inspectable. That is already valuable, provided the limits are visible and the claims remain proportionate to the evidence.
A Grad-CAM heatmap can localize a region. An attention map can show how a multiple-instance model weighted tiles. A counterfactual can test whether a prediction shifts after a synthetic morphological change. None of these, alone, establishes that the model recognized the correct disease process. The evidence comes from the combination of model performance, robustness testing, explanation assessment, user evaluation, and end-to-end clinical validation.
The deployment question is not whether the interface contains a heatmap. It is whether the laboratory has demonstrated what that heatmap means, when it remains reliable, and how a pathologist should use it when the tissue and the algorithm disagree. If those answers are clear, interpretability can become a meaningful safeguard. If they are not, the explanation is decoration—no matter how compelling the demonstration looks.