Reading the PathoROB robustness index with fixed measurements: a no-signal level, a colour reference, and what 48 named numbers explain in a foundation model's hospital signal

Parallax Pathology · Technical note

Short technical note and public artifact

Russell Parrish · analysis carried out with AI assistants · Version 3, 11 November 2026

Read the matched metric comparison · GitHub release v3.0.0 · Download the public control example

1. Question

PathoROB asks whether a foundation model's representation is organised by biology or by medical centre. Its Robustness Index (RI) compares, among each tile's nearest neighbours, those with the same tissue class from another centre against those with another tissue class from the same centre [1]. The published leaderboard lists 26 models.

The study began with one question, written down before any tile was measured: where does a fixed, training-free instrument sit on this index? The answer led to three further questions, each taken in turn:

• What does the index score for features with no signal, and for plain colour?
• Can fixed, named measurements explain, and remove, a foundation model's hospital signal?
• Does that removal hold on hospitals the fit never saw, and how does it compare with a standard batch correction?

2. Setup

The instrument returned a valid status on every tile. Eight of its values were undefined (seven in Tolkach, one in Camelyon) and were set to the dataset mean, as planned.

3. Reference rows

3.1 The index for features with no signal

The no-signal level is not the same number on each dataset. The index compares counts of two kinds of neighbour, and how often each kind occurs for unrelated tiles depends on the dataset's mix of classes and centres. A score of 0.60 therefore sits 0.10 above no signal on TCGA and 0.30 above it on Tolkach, and an average over the three datasets mixes those scales.

On Camelyon, the 11 models below the random-feature level are MUSK, Prov-GigaPath, RETCCL, RudolfV, Kaiko ViT-B/8, UNI, Ciga, CTransPath, Kang-DINO, Phikon-v2 and Phikon (0.467 down to 0.011). For these, neighbours follow the hospital more than they would for unrelated tiles.

3.2 Colour as a reference

Six colour numbers score below every published model on Camelyon (0.009 against a lowest published value of 0.011), and below the random-feature level on all three datasets. They are not uninformative: their tissue balanced accuracy is well above the random-feature values in section 3.4. On Camelyon, colour separates tissue and separates hospitals, and the hospitals dominate the neighbourhoods.

3.3 A view relative to no signal

Two quantities read against the random-feature run on the same dataset:

• Tissue signal beyond chance = (balanced accuracy − random-feature value) / (1 − random-feature value).
• Hospital pull beyond chance = random-feature RI − RI. Positive means neighbours follow the hospital more than they would for unrelated tiles.

On Camelyon, Phikon-v2 and RGB had similar differences from the Gaussian RI reference (about 0.48 and 0.49), while their chance-adjusted tissue balanced accuracies differed (0.91 and 0.60). Comparable RI can coexist with different tissue discrimination.

3.4 The fixed instrument

The plan predicted that the instrument would sit low on average, plausibly below most foundation models, and that Camelyon was the dataset where it could land among them. Both held: rank 23 of 27 on average, above 11 of 26 models on Camelyon. The instrument is built to be sensitive to how a slide was handled, so a low index is its expected operating window on this benchmark. Its tissue signal is modest beside the foundation models'.

3.5 Predicting the hospital from fixed numbers

Logistic regression predicting the medical centre, five folds grouped by slide (by case for TCGA), all tiles. Accuracy / balanced accuracy / one-vs-rest AUC:

Colour alone matches or exceeds the instrument alone on accuracy and AUC on every dataset. Adding the 42 measurements to colour raises balanced accuracy on all three (0.15 to 0.27, 0.64 to 0.83, 0.81 to 0.87). The benchmark paper reports that the centre can be predicted from foundation-model features with 88% to 98% mean accuracy [1]; its protocol may differ from this one.

3.6 What the lower tail adds

Can changing the metric reverse the colour result? We scored the same six RGB statistics and three Gaussian seeds under RI, margin-aware RI (MaRI) and the cross-confounder robustness margin (CRoMa) [10]. RGB scored below every seed under every metric in each cohort. The margins reveal how far apart the two kinds of neighbours were.

The two neighbour types are same tissue, other centre and other tissue, same centre. RI counts them within a fixed neighbourhood. MaRI gives more weight to closer neighbours, with a temperature set separately for each representation and cohort. CRoMa finds the five closest eligible neighbours of each type, compares their mean cosine distances and divides the difference by their sum. A negative margin means the same-centre group was closer. The share at or below zero measures frequency. The mean among margins at or below the 10th percentile measures severity at the adverse end of the distribution.

Matched public controls. Gaussian entries average three separately scored seeds. Fixed-k support is the share with at least one informative neighbour under RI and MaRI. CRoMa had full support in every run.

Cohort	Control	RI	MaRI	CRoMa	CRoMa tail mean	Margin ≤ 0	Fixed-k support
TCGA 2×2	Gaussian	0.500	0.500	≈0	−0.059	50.0%	100.00%
TCGA 2×2	RGB	0.463	0.457	−0.168	−0.868	55.3%	98.58%
Camelyon	Gaussian	0.499	0.499	≈0	−0.051	50.1%	99.96%
Camelyon	RGB	0.009	0.006	−0.888	−0.979	97.1%	63.41%
Tolkach ESCA	Gaussian	0.300	0.300	−0.031	−0.086	84.6%	100.00%
Tolkach ESCA	RGB	0.237	0.281	−0.442	−0.922	73.6%	98.33%

TCGA illustrates what a pooled score can hide. Colour RI was 0.463 against a Gaussian reference of 0.500. The colour headline margin was −0.168 and its lower-tail mean was −0.868, versus −0.059 for Gaussian. The colour tail contained 11,280 occurrences, representing 8,878 distinct tiles from 942 slides and all 94 pairings. All 11,280 already had fixed-k support. Here the deeper tail comes from measuring distance among covered observations. Repeated tiles count once per pairing in this calculation.

Camelyon distinguishes missing evidence from adverse evidence. For RGB, 12,935 of 20,400 tiles had fixed-k support. Fully 12,794 of those supported tiles had local RI equal to zero, so its tie-inclusive lower tail has mean zero and covers almost the entire supported set. CRoMa scored all 20,400. Among the 7,465 tiles without fixed-k evidence, 6,971 (93.4%) had nonpositive margins. Of the 2,040 tiles in the CRoMa lower tail, 2,033 already had fixed-k support. Extending coverage and resolving severity within the covered tiles are distinct contributions.

Tolkach separates frequency from severity. Gaussian produced more nonpositive margins than RGB (84.6% versus 73.6%). Its lower-tail mean was only −0.086, while RGB reached −0.922. Thus a count of which neighbour type wins and a measure of how far it wins can order the same controls differently.

Scope. This comparison scores 88,650 distinct tiles from the 99,392-tile feature inventory. It preserves TCGA's 94 paired 2×2 comparisons (59,250 distinct tiles), the two-centre Camelyon scoring manifest (20,400 tiles), and reduced three-centre Tolkach ESCA (9,000 tiles). All metrics exclude the query slide. Centre is an observational label that can capture tissue mix, staining, scanning and other differences. The three seeds describe simulation variation. They supply no patient-level uncertainty estimate. The margins in this section concern the two public controls only.

Medical interpretation. “Same tissue” means the same supplied benchmark label. Camelyon has tumour/normal labels, TCGA uses project-level cancer classes, and reduced Tolkach uses six tissue annotations. A nonpositive margin includes exact ties. It describes distances between labelled representations and gives neither a misclassification count nor a probability of patient harm. No patient outcomes or diagnostic decisions entered the control analysis. Three simulation seeds and repeated tile occurrences do not provide independent patient- or hospital-level replication.

The comparison plan fixed the three metrics before scoring. The tail-membership counts and coverage cross-tabulations above are descriptive second-pass checks. Across 12 runs the maximum RI difference between packages was 0.000009, attributable to near-tied neighbours at the cutoff. In an exhaustive check of 576 sampled occurrences the largest CRoMa margin discrepancy was 0.000023. The example includes the full-precision scores and the scripts to recover the local values. Download the reproducible example.

4. What fixed measurements explain in a foundation model

4.1 Design

The plan was written and hashed before any Phikon-v2 feature was extracted. Per dataset, every one of Phikon-v2's feature dimensions is regressed on a set Z of fixed measurements (ordinary least squares with intercept, Z standardised, all tiles of that dataset, no labels of any kind). The residual is kept and scored by the benchmark, unchanged. The sets:

• Instrument (42 columns), colour (6), both (48).
• Two small named subsets: five axes that had failed to carry between laboratories in separate work (section 6), and the axes that were most hospital-driven in the other two datasets (3 to 20 axes).
• Null: 42 random Gaussian columns, three seeds, to show what removing 42 directions of anything does.

Criterion, fixed in advance. A removal works on a dataset if its RI gain exceeds the best random-column gain by at least 0.05 and tissue balanced accuracy stays within 0.02 of the unmodified model. Prediction. Removing colour raises the Camelyon index. No direction was claimed for the instrument beyond colour.

4.2 Explained variance

Forty-eight named numbers predict 12.7%, 30.9% and 27.5% of the model's feature variance on TCGA, Camelyon and Tolkach. Random columns predict 0.1% to 0.3%. The instrument alone predicts more than colour alone on every dataset (25.1% against 20.5% on Camelyon).

4.3 Removal

Rank places each average against the 26 published rows, which include Phikon-v2 itself. Centre AUC is the benchmark's own linear hospital-prediction score; it reflects folds below 0.5, so noise scores above 0.5.

Best random-column gain: −0.005, +0.006 and +0.003.

• The prespecified tissue-accuracy tolerance was met. In the in-sample Camelyon fit, removing the component associated with RGB increased RI from 0.019 to 0.411. Benchmark tissue balanced accuracy decreased from 0.954 to 0.952, within the 0.02 tolerance.
• Colour carries most of the gain. It met the criterion on all three datasets by itself.
• The instrument beyond colour (gain of both minus gain of colour): +0.083 on Camelyon, +0.004 on TCGA, +0.003 on Tolkach. Clear on Camelyon only.
• Few named axes are not enough. The small subsets carried little of the effect; the full 42-column block did.
• Two readouts disagree in an informative way. On Camelyon, removing the instrument lowered the model's linear hospital-prediction AUC more than removing colour did (1.000 to 0.646, against 0.865) while raising the index less (0.361 against 0.411). Linear separability of hospitals and the purity of neighbourhoods respond differently to the same removal.

5. How does the result change when the fit uses other centres or datasets?

Section 4 fitted the removal on the same tiles it was scored on, as ComBat is fitted. Two checks were planned and hashed before they ran.

5.1 A design that failed its own null

The first held-out design fitted a separate removal for each hospital from the other hospitals. Removing 42 random columns this way lowered Phikon-v2's index (Tolkach 0.768 to 0.589, Camelyon 0.019 to 0.000). The design itself injects hospital structure: each hospital is shifted by a fit that excludes it, so hospitals are shifted by different amounts. Those numbers are kept and not interpreted. The redesign below was written and hashed after this fault was found and before it ran, and is declared as post hoc.

5.2 One fit, applied to every tile

• Other datasets. Fit on tiles from the two other named collections and apply one transformation to the scored collection. Source hospitals and patients were not reconciled across collections. The Tolkach inventory includes a TCGA-derived source.
• Half the source centres. Fit on alternate source centres in sorted order (13/26, 3/5 and 2/4) and apply the transformation to all benchmark evaluation rows. Camelyon’s RI combines UMCU, which contributed to fitting, with RUMC, which did not. On Tolkach, one of three scored centres contributed to the fit. These pooled scores do not estimate unseen-centre performance alone.

• Cross-dataset fit. No variant met the declared gain criterion on these three collections. The observed RI gains were within 0.022 of zero.
• Fitted on half the hospitals, a gain survives on Camelyon only. The instrument reached 0.269 and colour 0.163, both meeting the criterion (random columns: 0.066). On TCGA no variant met it; on Tolkach the index fell.
• The instrument held up better than colour on Camelyon. It kept 73% of its in-sample gain against 37% for colour. The plan had predicted that colour would keep most of its gain; that did not hold.
• Limits of the half-hospital fit. On Tolkach it uses 2 of 4 centres, one of them derived from TCGA.

5.3 Matched ComBat

We ran pycombat (inmoose 0.8.1) with medical centre as the batch, no biological covariates and all source tiles for fitting. For Phikon-v2 the publisher reports ComBat RI of 0.762 on TCGA paired 2×2, 0.446 on Camelyon and 0.923 on Tolkach. Our estimates were 0.755, 0.424 and 0.922, respectively. These are close numerical comparisons across implementations. Published Supplementary Tables 6–8, pages 13–15.

On Camelyon the label-free removal is above ComBat (0.494 against 0.424); on TCGA and Tolkach ComBat is higher. Applying the removal before ComBat adds 0.098 on Camelyon and 0.009 on TCGA, and costs 0.039 on Tolkach. Both methods here are fitted on the scored tiles. The benchmark paper reports that ComBat did not consistently help downstream models [1]; downstream training is outside this note.

6. Which named axes follow the hospital

For each of the 42 axes, the share of between-group variance that follows the medical centre and not the tissue class was computed per dataset and averaged. This is a one-way variance share per axis, not the neighbour index, and centre and class are not fully balanced in every dataset.

• Most hospital-driven: delta_r, gap_ds, blob_size_mean, g3_infl, gamma, k_rho, k_ds, k_rv.
• Least hospital-driven: g4_slope, g4_s32, g4_s8, omega, g1_curv, rc_s, rc_f, b_mu.
• Agreement with separate work. A per-field map between the Kather colorectal collection and SurGen slides [8], on different data and a different contrast, had marked five of these axes as breaking and six as holding. The five breaking axes rank 5, 6, 8, 10 and 11 here. The probability that five randomly chosen axes all fall in the top 11 of 42 is 0.0005. Of the six holding axes, five rank 27 or lower; one ranks 13. The 0.0005 figure is a descriptive combinatorial reference that assumes five uniformly chosen axes. Correlated measurements and choices made across earlier analyses limit its interpretation.
• Selecting axes by this ranking helped a little in one direction and not in the other. Dropping the five breaking axes raised the instrument's average index from 0.510 to 0.547 (0.518, 0.570, 0.551), almost all of it on Camelyon (0.471 to 0.570). Keeping only the six holding axes lowered it to 0.475 (0.534, 0.479, 0.413), with Tolkach tissue accuracy falling to 0.366. The plan had predicted that the six would be at least as robust as the full instrument; that did not hold.

7. Reading

• Coverage, frequency and severity answer different questions. The public controls agree in their headline ordering across three metrics. The Tolkach reversal in nonpositive frequency and the zero floor of Camelyon’s local colour RI show why support and lower-tail severity belong beside a headline score.
• The Gaussian control gives each cohort an empirical reference. Without it, 0.47 on Camelyon and 0.30 on Tolkach both look poor; with it, both are at or near the no-signal level.
• RI alone cannot establish tissue discrimination. Random features reach about 0.5 on two datasets with no tissue information. A representation can score near the no-signal level by carrying nothing, or by carrying tissue and hospital in balance.
• Measured covariation in Phikon-v2. On Camelyon, the 48 predictors explained 31% of total standardised embedding variance in the fitted tiles. This fraction is not a decomposition of centre-specific variance. Removing the fitted component moved RI to 0.494, close to the Gaussian control. The new RI alone cannot establish that the remaining embedding is centre-invariant.
• Scope of the fitted transformation. The largest gains were on tiles used for fitting. The half-centre analysis combined exposed and unexposed centres. The cross-dataset fits missed the declared criterion. ComBat used centre labels and had a higher mean RI across our three in-sample fits.
• Camelyon is the instrument's window. The 42 measurements add beyond colour clearly on Camelyon only, under the in-sample fit (+0.083) and the half-hospital fit (+0.086). An earlier pre-registered test on Camelyon17-WILDS found the same block adding to colour for tumour detection across held-out hospitals, and not in a kidney cohort [9]. Both use Camelyon-derived tiles, so they are two tasks on related data, not two independent datasets.
• Five previously flagged axes also rank highly here. This rank agreement across designs motivates testing their behaviour on measured acquisition changes. It does not identify the source of centre association.

8. Limitations

• One foundation model was read (Phikon-v2). The removal is linear and takes at most 48 directions out of a 2,048-dimensional feature.
• Section 4 and the local ComBat fit used scored-cohort tiles. Section 5 reports the mixed-centre evaluation and the cross-dataset fit separately.
• The first centre-specific design was set aside after applying separate fits changed the index even for random columns. The replacement design was specified after that finding. The publisher’s Supplementary Tables 6–8 provide the reference ComBat values.
• The random-feature level was added after the instrument's placement was seen. The relative view (section 3.3) and the hospital-prediction analysis (section 3.5) were proposed after the first results and have no hashed plan. The instrument's placement, the removal and the checks each had one.
• Random features were 42-dimensional. The no-signal level was not checked at the dimensions of the published models.
• Per-dataset standardisation of the reference columns is our choice; another scaling would change the colour and instrument rows.
• Foundation-model values in section 3.3 come from the results shipped in the benchmark repository; leaderboard values are as listed at the stated commit.
• "Hospital pull" is a difference of index values, not a causal quantity. Matching colour's pull does not show that colour is the cause.
• The index's no-signal level differs by dataset, so averages and ranks in this note inherit the mixing of scales described in section 3.1.
• We did not find a random-feature reference for the index in the benchmark's preprint text; one may exist in material we did not read.

Data and code availability

The runnable example contains the Gaussian and colour controls, pinned public input manifests and checksums, native and matched-metric reference scores, and verification and plotting scripts for section 3.6. It requires the two upstream repositories and public tiles. The ZIP SHA-256 is 61e9fbe8f655175609a377a929adcba1c96f3a18c9cafa1409441dd415eff4a6. Public control example and run instructions.

The wider study used the benchmark code and datasets [1], the released fixed instrument [2] and Phikon-v2 [3]. The foundation-model and fixed-instrument analyses are recorded separately from the example. The TCGA tile subset used by PathoROB carries the CC BY-NC-SA 4.0 licence.

References

1. Kömen J, de Jong ED, Hense J, Marienwald H, Dippel J, Naumann P, Marcus E, Ruff L, Alber M, Teuwen J, Klauschen F, Müller K-R. Towards robust foundation models for digital pathology. Nature Communications 2026;17:5218. doi:10.1038/s41467-026-73923-2. Preprint: arXiv:2507.17845. Code and leaderboard: https://github.com/bifold-pathomics/PathoROB (commit d81a04b).
2. Parrish R. parallax-core42 v1.0.0: a fixed 42-axis H&E measurement instrument. https://github.com/parallaxmetrology/parallax-core42
3. Filiot A, Jacob P, Mac Kain A, Saillard C. Phikon-v2, a large and public feature extractor for biomarker prediction. arXiv:2409.09173, 2024.
4. Bándi P, et al. From detection of individual metastases to classification of lymph node status at the patient level: the CAMELYON17 challenge. IEEE Transactions on Medical Imaging 2019;38(2):550–560. doi:10.1109/TMI.2018.2867350
5. Komura D, et al. Universal encoding of pan-cancer histology by deep texture representations. Cell Reports 2022;38(9):110424. doi:10.1016/j.celrep.2022.110424
6. Tolkach Y, et al. Artificial intelligence for tumour tissue detection and histological regression grading in oesophageal adenocarcinomas: a retrospective algorithm development and validation study. The Lancet Digital Health 2023;5(5):e265–e275.
7. Johnson WE, Li C, Rabinovic A. Adjusting batch effects in microarray expression data using empirical Bayes methods. Biostatistics 2007;8(1):118–127.
8. Parrish R. Which fixed H&E measurements survive a change of laboratory. Technical note, 2026. https://www.parallaxmetrology.com/Field_Transport_NOTE.html
9. Parrish R. Two pre-registered transport tests of fixed structural H&E measurements. Technical note, 2026. https://www.parallaxmetrology.com/Transport_Tests_NOTE.html
10. Grisi C, van der Laak J, Litjens G. A distributional robustness margin for pathology foundation models. arXiv:2607.25497, 2026. Preprint · Scorer at the tested revision.