Diagnosing the Sparse Mechanism Shift Hypothesis: A Graph-Free Invariance Test

1. Introduction

Distribution shift (a change in the data-generating distribution between training and deployment) is a central obstacle to reliable machine learning . A productive way to reason about shift is through causal structure: if the data are generated by a structural causal model (SCM) whose variables \(X_1,\dots,X_d\) obey mechanisms \(P(X_j\mid X_{\mathrm{pa}_j})\), then a shift corresponds to some of these mechanisms changing while others stay invariant . The independent causal mechanisms principle further suggests that natural shifts are localised: a change to the world typically perturbs only a few mechanisms at once. This is the sparse mechanism shift (SMS) hypothesis . To state it precisely, assume the data follow an SCM in which each variable \(X_j\) is produced by a mechanism, its conditional distribution given its causal parents, \(P(X_j\mid X_{\mathrm{pa}_j})\). Comparing a source environment \(s\) with a target environment \(t\), the changed set, its size, and the sparsity ratio are:

and SMS posits that only a small fraction of the \(d\) mechanisms change across environments, i.e. \(\rho\ll 1\); the shift is dense when \(\rho\) is large (a majority of mechanisms change).

SMS is attractive because it licenses powerful procedures. If only a few mechanisms change, one can score and localise them , discover structure from heterogeneous or nonstationary data by treating the environment as an auxiliary variable , and build predictors that transport by relying on the invariant part of the model . Yet across this literature SMS is typically an assumption: methods presuppose that the shift is sparse and proceed, without testing whether sparsity actually holds in the data at hand. When the assumption silently fails (when a shift is in fact dense), the downstream guarantees quietly fail with it.

As a concrete example, consider a protein-signalling network in which each protein’s abundance is generated from its upstream regulators . A targeted drug that inhibits a single protein is an atomic intervention: ideally it alters only that protein’s mechanism and leaves the others invariant, so \(\rho\) is small and SMS holds. A broad-acting drug that perturbs a whole pathway is a fat-hand intervention: many conditionals shift at once, \(\rho\) is large, and SMS is violated. Whether SMS holds is therefore not a modelling convenience but an empirical property of the specific shift, one that, we argue, should be measured rather than assumed.

Objective. The goal of this paper is to make the SMS assumption diagnosable on the data at hand. Concretely, we aim to: (i) characterise when and why localising the changed mechanisms becomes hard once the causal graph must be estimated rather than known; (ii) design a graph-free, label-free invariance diagnostic that flags nodes whose conditional distribution appears environment-dependent (a proxy for the changed set \(S^\star\); see §3.4 and §5 for the causal-localisation caveat); and (iii) build a data-calibrated, falsifiable diagnostic that returns a verdict for SMS together with a discovery-based sparsity statistic \(\hat\rho=\lvert \hat S\rvert/d\), a false-discovery-controlled count of flagged mechanisms rather than a consistent estimator of the true ratio \(\rho\) (see {\S}3.5 and {\S}4.5). Pursuing these objectives, our contributions are:

1. A diagnosis of mechanism-shift detection under an estimated graph. Through a controlled ablation we show the precision bottleneck is systematic false positives at the truly invariant nodes, induced by mis-estimated parent sets. It is not the global skeleton quality, not the changed nodes, and not fixable by conditioning-set stability selection.
2. A graph-free, label-free detector that flags a node only when no candidate conditioning subset renders its conditional invariant across environments, an inverse use of invariant causal prediction . It is robust to the diagnosed failure mode and, with a false-discovery-rate (FDR) decision (as an approximation; see §5(iv)) , matches an oracle that knows the true graph.
3. A data-referenced SMS diagnostic: a false-discovery-controlled discovery statistic \(\hat{\rho}=\lvert \hat{S}\rvert/d\) together with a data-driven null floor (split one environment in half, where no real shift exists) and a bootstrap verdict (no shift / sparse / dense / inconclusive). We present this as a data-calibrated heuristic rather than a hypothesis test with formal Type-I control: only the lower null floor is calibrated, the sparse/dense cut-point remains a semantic choice, and intervals straddling it are reported inconclusive.
4. Evidence across controlled, semi-synthetic, real-structure and real-intervention data. The verdict recovers the correct sparse/dense regime on synthetic data and, on real protein-signalling interventions, returns a dense verdict that clears the data-referenced no-shift floor by a wide margin, which we read as evidence against SMS for these chemical interventions rather than as within-environment detector noise—the source split calibrates the floor against false positives within an environment, so a systematic cross-environment artefact is not excluded by it (Section 4.4); a paired atomic-versus-fat-hand study supplies a mechanistic reading of that verdict and yields a falsifiable prediction.

We deliberately work in a linear-Gaussian, two-environment setting to establish the test cleanly, and provide a kernel-based extension for nonlinear mechanisms. All reported numbers come from executed runs.

2. Related Work

Mechanism-shift scoring and heterogeneous discovery. The Mechanism Shift Score (MSS) of Perry et al. ranks mechanisms by how much their conditionals change across environments, explicitly under the SMS hypothesis. CD-NOD adds the environment as a surrogate node and detects which mechanisms depend on it. Two limitations follow. First, both presuppose that the shift is sparse: they rank or flag mechanisms but never report whether SMS actually holds, so a dense shift is mishandled silently. Second, both lean on a known or reliably discoverable causal graph, and their accuracy degrades once the graph must be estimated from the same shifted data. We use MSS and a CD-NOD-style detector as baselines and supply the missing ingredient, a test of the premise that is also robust to graph-estimation error.

Invariance and transport. Invariant Causal Prediction (ICP) seeks a predictor whose residual distribution is invariant across environments; IRM , anchor regression and the graph-surgery view build robust or transportable predictors from invariant components; DRIG interpolates between empirical-risk minimisation and the causal solution. These methods, however, are oriented toward predicting a single target from an invariant set, not toward localising which mechanisms changed across the whole graph; and several (notably IRM) are known to be sensitive to environment design and optimisation. Our detector inverts ICP: instead of finding one invariant predictor for a target, we ask, for every node, whether any invariant conditioning set exists, and flag the node if none does.

Sparse shift estimation with FDR. Sparse-Joint-Shift / SEES estimates model performance under simultaneous covariate and label shift, assuming the shift is sparse; SGShift attributes concept shift to a sparse set of features with knockoff-based FDR control . Both, however, operate at the feature level and still assume sparsity as an identifiability or FDR condition rather than testing it; and a feature whose marginal merely moved cannot be distinguished from a mechanism that genuinely changed versus an ancestor shifting upstream. We instead localise causal-mechanism changes (conditioning on causal parents separates a changed mechanism from a propagated one), and test the sparsity assumption itself.

Causal representation learning. A complementary recent line lifts causal structure from low-level data such as images: causal representation learning recovers latent factors and their causal graph, with identifiability established mainly from interventions across environments . These are identifiability guarantees under idealised assumptions (e.g. one intervention per latent node, or the infinite-sample limit) and are not aimed at testing which mechanisms shifted: they say how to obtain aligned factors but stop short of the downstream sparsity test. We connect the two: such an encoder is exactly the front-end our image extension relies on, and our finding that accurate changed-set recovery needs a cross-environment-consistent encoder mirrors the role of multi-environment interventional data for identifiability there; intervention-extrapolation guarantees are especially pertinent to the target-domain bias we isolate.

The common gap. Across these lines three issues recur and motivate this work. (i) Sparsity is assumed, not tested: no method reports whether SMS holds on the data at hand, so downstream guarantees fail silently when the shift is in fact dense. (ii) Accurate mechanism localisation is demonstrated only under an oracle (a known graph, a designated prediction target, or generative-factor-aligned latents), leaving the realistic regime (estimated graph, no labels, no aligned encoder) largely untreated. (iii) The failure modes in that regime are uncharacterised, so it is unclear what to repair. We answer these in turn with a test of the premise, a graph-free and label-free detector, and a precise diagnosis of where estimated-graph detection breaks.

3. Method

3.1 Problem setup

We observe data from two environments, a source \(s\) and a target \(t\), generated by SCMs that share a directed acyclic graph \(G\) over variables \(X=(X_1,\dots,X_d)\) with parent sets \(\mathrm{pa}_j\):

with mutually independent noises \(N_j^{(e)}\). The mechanism of node \(j\) in environment \(e\) is the conditional \(P^{(e)}(X_j\mid X_{\mathrm{pa}_j})\). The changed set \(S^\star\), its size \(k=\lvert S^\star\rvert\), and the sparsity ratio \(\rho=k/d\in[0,1]\) are as introduced in Eq. (1); the shift satisfies the sparse mechanism shift hypothesis when \(\rho\) is small.

Note that a changed mechanism at node \(j\) moves the conditional \(P(X_j\mid X_{\mathrm{pa}_j})\), but the marginal of a downstream node can move even if its own mechanism is invariant, because the change propagates through the graph. Distinguishing “this mechanism changed” from “an ancestor changed upstream” is exactly what conditioning on the causal parents buys, and is why feature-level (marginal) detection is insufficient.

For the controlled setting we use linear-Gaussian SCMs: a changed node receives an intercept shift, a parent-coefficient shift, and a noise-variance rescaling, so that every unchanged node’s conditional stays exactly invariant while its marginal may move. This gives ground-truth \(S^\star\) for every run, so detection can be scored.

3.2 A mechanism-change test

Given a candidate conditioning set \(S\) (intended to be the parents of \(j\)), the atomic question is whether node \(j\)’s mechanism changed, that is, whether the conditional \(P(X_j\mid X_S)\) is the same in both environments. We cast this as a likelihood-ratio test (LRT) between two linear-Gaussian models of \(X_j\) regressed on \(X_S\). The reduced model encodes the null hypothesis \(H_0\) “the mechanism is invariant”: it pools the two environments and fits a single shared conditional. The full model encodes the alternative “the mechanism changed”: it fits a separate conditional in each environment. Writing \(\ell_{\mathrm{reduced}}\) and \(\ell_{\mathrm{full}}=\ell^{(s)}+\ell^{(t)}\) for the two maximised log-likelihoods,

The full model can only fit at least as well, so \(\Lambda_j(S)\ge 0\) measures how much a per-environment fit improves on the pooled one. Under \(H_0\) it is asymptotically \(\chi^2\) with \(\lvert S\rvert+2\) degrees of freedom: the free parameters that the second environment’s conditional adds over the shared one (\(\lvert S\rvert+1\) regression coefficients: an intercept and one slope per variable in \(S\), plus one noise variance). Because each environment’s conditional is fitted independently in both its coefficients and its variance, the statistic detects a mean change (a shifted intercept or parent coefficients) and a variance change (rescaled noise), exactly the ways a mechanism can move in our SCM. A small p-value \(p=P(\chi^2_{\lvert S\rvert+2}\ge\Lambda_j(S))\) rejects invariance and flags node \(j\) as changed. Applying the test with \(S=\mathrm{pa}_j\) at every node and controlling the false discovery rate with Benjamini–Hochberg (BH) over the \(d\) nodes yields the estimated changed set \(\hat{S}\) and the sparsity estimate \(\hat{\rho}=\lvert \hat{S}\rvert/d\). With a known graph this is accurate; the difficulty, which the next subsection diagnoses, is that in practice \(G\) (and hence each parent set \(\mathrm{pa}_j\)) must be estimated.

3.3 Diagnosing the estimated-graph bottleneck

When the graph is estimated (e.g. by the PC algorithm on pooled or single-environment data), detection precision drops sharply. We localise the cause with an ablationThe diagnostic variants are named descriptively here. In our code and released results they correspond to labels A (oracle graph), C (pooled-PC), C5--C8 (the ablations), and C10 (the exists-invariant-set detector, used as a shorthand in later sections). that holds the data-generating process and seeds fixed and changes only one ingredient at a time (Table 1):

• A better global skeleton does not help: running the combined detector on a high-quality source-only skeleton does not beat its pooled-graph counterpart, so global skeleton quality is not the bottleneck.
• Repairing the changed nodes’ parents does not help: oracle-repairing the flagged nodes’ parents or doing real local repair there leaves \(F_1\) essentially unchanged.
• Repairing the unchanged nodes’ parents does: oracle-repairing the presumed-stable nodes’ parents jumps \(F_1\) to near the oracle. Hence the false positives that cap precision come from mis-estimated parent sets of the invariant nodes: a wrong parent set makes a stable conditional look environment-dependent.
• Conditioning-set voting cannot fix it: stability selection over resampled plausible parent sets is worse, because the stable-node false positives are systematic (most sampled parent sets miss a true parent), so voters share the bias.

Table 1. Diagnostic ablation: changed-set \(F_1\) versus the true number of changed mechanisms \(k\) (linear-Gaussian SCM, \(d=10\), \(n=3000\) per environment, 15 seeds, FDR \(\alpha=0.1\)). Repairing the stable nodes recovers oracle-level \(F_1\); repairing the changed nodes, using a better (source-only) skeleton, or conditioning-set voting does not. At \(k=0\) the true changed set is empty, so precision and recall are both \(0/0\); we adopt the convention \(F_1=1\) when \(\hat{S}=\varnothing\) and \(F_1=0\) otherwise, so that column reports the fraction of seeds in which a method makes no false discovery. The oracle row is an upper bound only among detectors that condition on a single (here, the true) parent set per node: the exists-invariant-set detector follows a different route and can exceed it, as it does at \(k=0\) and \(k=2\).

3.4 Graph-free detection via existence of an invariant set

Our detector responds to this diagnosis. Rather than commit to one (possibly wrong) parent set per node, we ask whether some conditioning set makes the node’s conditional invariant. This inverts invariant causal prediction : a node is declared changed only if no candidate subset survives the invariance test. Formally, with a candidate pool \(\mathcal{C}_j\) (by default the variables most correlated with \(X_j\), so a true parent missed by an estimated graph can still be tried),

the largest invariance \(p\)-value over candidate subsets. A large \(q_j\) means an invariant set exists (the node is saved); a small \(q_j\) means every set was rejected (the node changed). We apply BH-FDR to \(\{q_j\}_{j=1}^d\) to obtain \(\hat{S}\). Because a stable node is rescued whenever any candidate set (e.g. its true parents) looks invariant, the detector is robust to the parent-set errors that defeat the parent-conditioned test, precisely the stable-node failure mode identified above. We refer to this exists-invariant-set detector as C10. Algorithm 1 summarises it; the per-node subset search is the dominant cost and is evaluated by batched linear algebra.

When does “no invariant set exists” mean the mechanism changed? The detector’s target, the non-existence of an invariant conditioning set, coincides with a genuine change of the causal mechanism \(P(X_j\mid X_{\mathrm{pa}_j})\) only under a candidate-coverage assumption: the pool \(\mathcal{C}_j\) must (a) contain a set that renders a truly unchanged node invariant, for which its parents \(\mathrm{pa}_j\) suffice, and (b) not contain sets that restore invariance spuriously, such as descendants, colliders, or variables whose own shift compensates node \(j\)’s. In the linear-Gaussian SCM this holds by construction, because every unchanged node’s parent conditional is exactly invariant, which is why C10 matches the oracle on the synthetic and real-structure data. On real data (e.g. Sachs) the assumption is not guaranteed, so we read a small \(q_j\) as evidence that \(X_j\) is environment-dependent given every set we tried, rather than as a definitive causal-mechanism localisation; conditioning on a compensating descendant is an adversarial case that can hide a genuine change, and we bound \(\lvert S\rvert\le m\) and rank \(\mathcal{C}_j\) by parent-like association to limit it. Under coverage assumption (a), \(q_j=\max_S p(\Lambda_j(S))\) is super-uniform under the invariance null, since \(\Pr(q_j\le t)\le \Pr\bigl(p(\Lambda_j(S^{\mathrm{inv}}))\le t\bigr)=t\) for the invariant set \(S^{\mathrm{inv}}\in\mathcal{C}_j\), which is what licenses applying BH to \(\{q_j\}\). Because the \(q_j\) are computed on shared data and are therefore dependent across nodes, we use BH as an approximation and note that the Benjamini–Yekutieli correction provides FDR control under arbitrary dependence when a conservative guarantee is required.

3.5 A data-referenced SMS diagnostic

The detector gives a set-valued output \(\hat{S}\) and an associated discovery proportion \(\hat{\rho}=\lvert\hat{S}\rvert/d\)—a detector output, not a consistent estimator of the true sparsity \(\rho\) (BH-FDR controls the false-discovery proportion, not the changed-set count, so \(\hat{\rho}\) can be biased upward by false positives or downward by false negatives; see Section 5). To diagnose SMS we need a reference for “how sparse is sparse”. We avoid a guessed threshold for the lower end by calibrating a null floor from the data: split the source environment in half, where no real shift exists, and run the same detector to obtain \(\rho_{\mathrm{null}}\), the detector’s own false-positive floor. The decision uses bootstrap 90% confidence intervals (Algorithm 1): if the observed \(\hat{\rho}\) interval overlaps the null floor, there is no detectable shift; if the interval is separated from the floor and also overlaps \(\tau_{\mathrm{dense}}\) (i.e., \(\mathrm{CI}_{\mathrm{lo}}(\hat{\rho}) < \tau_{\mathrm{dense}} < \mathrm{CI}_{\mathrm{hi}}(\hat{\rho})\)), the verdict is inconclusive—the uncertainty makes a definite sparse/dense call unreliable; if it is separated from the floor and does not extend above \(\tau_{\mathrm{dense}}\) (i.e. \(\mathrm{CI}_{\mathrm{hi}}(\hat{\rho}) \le \tau_{\mathrm{dense}}\), matching line 15 of Algorithm 1), the shift is sparse and SMS holds; if it is separated and entirely above \(\tau_{\mathrm{dense}}\) (\(\hat{\rho}\ge \tau_{\mathrm{dense}}\) and \(\mathrm{CI}_{\mathrm{lo}}(\hat{\rho}) \ge \tau_{\mathrm{dense}}\), default \(0.5\), “a majority of mechanisms”), the shift is dense and SMS is rejected. The null floor is data-referenced rather than statistically calibrated: it provides an empirical reference for the detector’s within-environment false-positive rate, but the overall verdict does not carry a formal Type-I or coverage guarantee (the sparse/dense boundary \(\tau_{\mathrm{dense}}\) remains a semantic choice, and the bootstrap CIs reflect resampling variation of \(\hat{\rho}\)’s expectation, not a confidence set for the true \(\rho\)). Nevertheless, the null floor serves an important diagnostic role: it certifies that a large observed \(\hat{\rho}\) is a real cross-environment signal rather than detector noise within a single environment.

We emphasise that this three-way procedure is a data-calibrated diagnostic, not a hypothesis test with a formal Type-I or coverage guarantee. Only the lower no-shift floor is calibrated, and it is calibrated against within-environment false positives (the source split), so it does not certify the absence of systematic cross-environment over-flagging; the sparse/dense cut-point \(\tau_{\mathrm{dense}}\) is a semantic choice, not a level. To avoid overclaiming at the boundary we report the verdict as inconclusive whenever the \(\hat{\rho}\) interval straddles \(\tau_{\mathrm{dense}}\) (Algorithm 1, line 13) rather than forcing a label. Finally, \(\hat{\rho}=\lvert\hat{S}\rvert/d\) is a false-discovery-controlled discovery count: BH controls the false-discovery proportion of \(\hat{S}\), not its cardinality, so \(\hat{\rho}\) need not be a consistent or unbiased estimator of \(\rho\) and can be biased downward by missed detections. We therefore use \(\hat{\rho}\) to separate regimes (sparse vs. dense vs. no shift), and read it as a detector output rather than as a point estimate of the true sparsity.

3.6 Nonlinear extension

The invariance test in Eq. (3) is linear-Gaussian. For nonlinear mechanisms we replace it inside Eq. (4) with a kernel conditional-independence test : node \(j\) has an invariant set \(S\) iff \(X_j \perp E \mid X_S\), where \(E\) is the environment indicator. This keeps the exists-invariant-set logic but removes the linear-Gaussian assumption.

3.7 Extension to image data via a fixed factor encoder

Because the test operates on a matrix of variables, it extends to images through a front-end encoder \(g\) that maps each image to a low-dimensional factor vector \(z=g(x)\); the recovered factors are then fed, unchanged, to the same detector and data-referenced verdict. The encoder is frozen after training, a pure preprocessing step that leaves the statistical core untouched, so the SMS machinery is reused verbatim. The catch is that sparsity is basis-dependent: a shift that is sparse in the causal factors can appear dense in an entangled representation, so \(g\) must output factors aligned with the generative ones rather than generic deep features. We therefore evaluate on datasets with known generative factors and study how the choice of \(g\) governs recovery.

4. Experiments

Table 2 summarises the datasets used throughout. They span synthetic SCMs with full ground truth, semi-synthetic and published causal graphs, real protein-signalling interventions, and image data with known generative factors. Sample sizes and seed counts are reported with each experiment below.

Table 2. Datasets used in the experiments. \(d\) counts the variables (graph nodes or generative factors); \(n\) per env. is the per-environment sample size (for Sachs, observational / per-intervention cell counts, 5400 total). “LG” denotes linear-Gaussian mechanisms; the nonlinear synthetic SCM uses \(\tanh\) mechanisms for the kernel (KCI) detector. dSprites (binary) and 3D Shapes (RGB) are \(64\times64\) images with LG factor mechanisms.

4.1 Controlled synthetic SMS

With a known graph (\(d=12\), \(n=4000\) per environment, 30 seeds, FDR \(\alpha=0.1\)), the mechanism-change test recovers \(S^\star\) reliably: recall is \(1.0\) at every \(k\), \(F_1\approx 0.90\)–\(1.0\), and \(\hat{k}\) tracks the truth (e.g. \(k=4\!\to\!\hat{k}\approx4.5\), \(k=8\!\to\!\hat{k}\approx8.2\)). The abstain rate (the fraction of runs in which the designated target’s own mechanism is flagged, so no invariant predictor of it exists) rises from \(0\) to \(1\) with \(k\), tracking \(k/d\), and the abstain decision is correct 97–100% of the time (Figure 1). Certified invariance buys robustness: among target-stable cases the parents predictor’s transport error stays small (\(\approx 0.001\)–\(0.03\)) across all \(k\), whereas an empirical-risk-minimising “ERM-all” predictor degrades sharply as the shift densifies (mean transport error \(\approx 0.007 \to 27 \to 219\)), a heavy-tailed phenomenon across random graphs (Figure 2).

4.2 Diagnostic ablation and the C10 solution

Dropping the known-graph assumption (\(d=10\), \(n=3000\), 15 seeds), Table 1 and Figure 3 establish both the diagnosis and our solution. Two further comparisons sharpen the message. First, FDR control beats fixed-threshold MSS exactly in the sparse regime it targets: at \(k=0\) the MSS baseline has \(F_1\approx0.53\) versus FDR’s \(0.80\) (the gap closes as the shift densifies and false positives matter less). Second, mechanism-level beats feature-level: a graph-free marginal detector flags any variable whose marginal moved, inflating its estimate far beyond the truth (precision \(\approx0.44\) at \(k=2\) versus the oracle’s \(0.92\)), because it cannot tell a changed mechanism from a shifted ancestor. The upgraded C10 (global candidate pool + FDR) reaches recall \(1.0\) at every \(k\) and \(F_1 = 1.00/0.99/0.96/0.98/0.98\) at \(k=0/2/4/6/8\), matching the oracle and even exceeding it at small \(k\), with no labels and no known graph. Figure 4 shows the complementary discovery view: single-environment (source-only) discovery keeps graph recovery flat as the shift densifies, while pooled and CD-NOD discovery degrade; and CD-NOD’s environment adjacency is a high-precision but falling-recall changed-set signal, complementary to the high-recall LRT+FDR.

4.3 Semi-synthetic and real causal structures

On a linear-Gaussian SCM fit from real American Community Survey covariates (\(d=8\), 15 seeds) with planted shifts, C10 attains \(F_1=0.97\)–\(1.0\) (recall \(1.0\)), tying or beating the oracle and far above an estimated-graph baseline, so the result is not an artefact of a clean synthetic SCM. Table 3 repeats the test on four published causal-graph topologies (ASIA, SACHS, CHILD, ALARM; \(d=8\) to \(37\)) with synthetic mechanisms and planted shifts. C10 ties or beats the oracle in every cell but one and stays robust with dimension; the exception is ALARM at \(k=4\) (\(0.88\) against the oracle’s \(0.97\)), where the bounded candidate pool and subset order cap recovery on the largest and densest shift we consider (Section 5, limitation ii). A pooled-PC estimated graph, by contrast, collapses on the denser networks (e.g. SACHS \(F_1\) down to \(0.48\) at \(k=2\)). The nonlinear extension behaves as intended: on a \(\tanh\) SCM the kernel version of C10 restores \(F_1\) to \(1.00/0.98\) at \(k=2/4\) where the linear version is misspecified (\(0.81/0.85\)).

Table 3. Real causal structures: changed-set \(F_1\) for the exists-invariant-set detector (C10), the oracle (A), and an estimated pooled-PC graph (C), at \(k\) changed mechanisms. Across \(d=8\)–\(37\) C10 ties or beats the oracle in every cell except ALARM at \(k=4\) (\(0.88\) against \(0.97\); see Section 5, limitation ii); the estimated graph collapses on the denser networks. Pooled-PC is omitted for ALARM (\(d=37\)), where it is prohibitively slow. At \(k=0\) the changed set is empty and \(F_1\) is undefined in the usual sense; we adopt the convention \(F_1=1\) when \(\hat{S}=\varnothing\) and \(F_1=0\) otherwise, so this column reports the fraction of seeds with no false discovery.

4.4 Real interventions: the Sachs data

We next apply the detector to real protein-signalling data with known chemical interventions : each intervention is contrasted with the observational condition over 11 phospho-proteins on the published consensus topology. The detector flags every intervened target (recall \(5/5\)), and \(\hat{S}\) is large (\(\hat{k}=8\)–\(11\) of \(11\)). The conventional precision computed against the single intervention target as ground truth is low (\(\approx 0.09\)–\(0.17\)), but this number is not strictly interpretable as a false-discovery rate: if a chemical drug is fat-hand, the true changed set exceeds the single target, and nodes flagged beyond the target are not necessarily false positives. The source-split null floor is \(\rho_{\mathrm{null}}\le 0.28\) (Table 4), far below the cross-environment \(\hat{\rho}=0.65\)–\(1.0\), which rules out indiscriminate within-environment over-flagging and indicates that the elevated \(\hat{\rho}\) reflects a genuine cross-environment signal. Together, the evidence supports the interpretation that real chemical drugs are fat-hand (a single drug perturbs an entire pathway), and the observational and interventional conditions differ globally, so the real shift simply is not sparse. The test therefore returns a dense verdict on real interventions—exactly the situation SMS-assuming methods would mishandle silently. We phrase this as a verdict rather than a verified diagnosis because Sachs provides no complete ground-truth changed set against which correctness could be checked. For the same reason, systematic cross-environment over-flagging (e.g., due to batch effects or discretisation artefacts, beyond what the source-split null captures) cannot be fully excluded; the fat-hand explanation remains a hypothesis supported by the synthetic atomic-vs-fat-hand evidence (Section 4.6) rather than a direct measurement.

4.5 The data-referenced SMS test

Table 4 reports the data-referenced verdict (\(d=12\), \(n=3000\), 20-bootstrap 90% CIs, \(\tau_{\mathrm{dense}}=0.5\)). On controlled synthetic data the verdict anchors all three regimes: \(k=0\) is called no shift (\(\hat{\rho}\approx\rho_{\mathrm{null}}\approx 0\), so the detector does not hallucinate a shift), \(k=1\)–\(4\) are sparse, and \(k=8\) is dense (with \(\hat{\rho}\) broadly consistent with, though not numerically equal to, the true \(k/d\); note that \(\hat{\rho}=\lvert\hat{S}\rvert/d\) is a detector output, not a consistent estimator of \(\rho\), and the 90% bootstrap CIs reflect resampling variation of \(\hat{\rho}\)’s expectation—see Sections 3.5 and 5). The synthetic \(k=2\) case (true \(\rho=0.17\), \(\hat{\rho}=0.36\) with CI \([0.25, 0.50]\)) illustrates the upward bias from false positives; the CI upper bound touches \(\tau_{\mathrm{dense}}=0.5\) without exceeding it, so the interval does not strictly straddle the cut-point, the inconclusive condition (Algorithm 1, line 13) does not fire, and the boundary rule \(\mathrm{CI}_{\mathrm{hi}}(\hat{\rho})\le\tau_{\mathrm{dense}}\) returns SPARSE. This case sits exactly on the boundary and should be read as such. On all five Sachs interventions the verdict is dense / reject SMS; critically, the null floor is only \(\rho_{\mathrm{null}}\le 0.28\) while \(\hat{\rho}=0.65\)–\(1.0\), indicating that the dense verdict reflects a genuine cross-environment signal and not merely noise from the discretised, observational measurements.

Table 4. Data-referenced SMS verdict. \(\hat{\rho}=\lvert \hat{S}\rvert/d\) with bootstrap 90% CI; \(\rho_{\mathrm{null}}\) is the data-driven false-positive floor (source split in half), with its CI upper bound in parentheses; \(\tau_{\mathrm{dense}}=0.5\).

4.6 Atomic versus fat-hand interventions

The dense verdict on Sachs raises a mechanistic question: is a more atomic intervention actually sparser? This is the regime of Perturb-seq , where a CRISPR knockout targets a single gene. Lacking the single-cell data here, we test the mechanism-level logic on a controlled SCM (\(d=12\), 6 seeds) by contrasting, on the same graph, an atomic intervention (change only the target’s mechanism) against a fat-hand one (change the target and all its descendants, emulating pathway propagation). Table 5 shows the verdict calls the atomic intervention sparse in \(6/6\) runs (\(\hat{\rho}=0.09\)) and the fat-hand one dense in \(6/6\) (\(\hat{\rho}=0.83\)). This explains the Sachs result (chemical drugs are fat-hand) and yields a falsifiable prediction: a genuinely atomic intervention, such as a CRISPR knockout, should be sparser and more likely to satisfy SMS.

Table 5. Atomic versus fat-hand interventions on the same SCM (\(d=12\), 6 seeds). The verdict calls atomic sparse and fat-hand dense, every time.

4.7 Extension to image data: dSprites and 3D Shapes

As a proof of concept we apply the image pipeline to dSprites , binary \(64\times64\) images with known generative factors. We drive a \(d{=}5\) factor SCM (shape, scale, orientation, \(x\), \(y\)), render the corresponding real images, and plant \(k\) changed factor-mechanisms (known \(S^\star\)). With the oracle factors the verdict is correct in the sparse/dense sense but noisy at this low dimension (the count statistic \(\hat\rho=\hat k/d\) is coarse at \(d{=}5\)), so we report changed-set \(F_1\), which is more stable, and use the verdict only qualitatively.

We now examine how the frozen encoder \(g\) affects recovery (Table 6 and Figure 5). A CNN trained on the source environment only loses \(F_1\) relative to the oracle, despite high reconstruction fidelity (target recon \(\approx 0.87\)). This loss is not a fidelity problem: a controlled experiment that adds independent Gaussian noise of the same magnitude to the oracle factors leaves \(F_1\) essentially unchanged at that recon level, so it is the encoder’s cross-environment systematic bias (extrapolation onto the shifted target), not residual magnitude, that manufactures the false positives. Consistent with this, an encoder trained jointly on both environments (a domain-consistent encoder; recon \(\approx 0.98\)) restores \(F_1\) to the oracle. A frozen encoder thus preserves the coarse verdict, but accurate changed-set recovery requires a representation that is consistent (identifiable) across environments, not merely high-fidelity.

We repeat the comparison on 3D Shapes , RGB \(64\times64\) images with six known factors (Table 6). Here the source-only frozen CNN already matches the oracle (\(F_1\) within \(0.01\)), leaving no gap for a domain-consistent encoder to close: the clean renderings let even a source-trained CNN reach recon \(\approx 0.97\)–\(0.99\) that transfers to the target. This sharpens rather than contradicts the earlier conclusion: the changed-set deficit is governed by the encoder’s cross-environment extrapolation error, which is large on dSprites but small here, so domain-consistent training helps precisely when that error is large. As on dSprites, the low factor count and discretisation cap the oracle itself (e.g. \(F_1\approx0.52\) for sparse shifts).

Table 6. Image data (dSprites, 3D Shapes): changed-set \(F_1\) for oracle factors, a source-only frozen CNN, and a domain-consistent (jointly trained) frozen CNN, averaged over seeds.

5. Discussion and Limitations

We have made the SMS premise diagnosable: a graph-free, label-free invariance test that matches an oracle, and a data-referenced diagnostic with a data-driven null floor that accepts sparsity when it holds and rejects it when it does not, returning a dense verdict on real interventions for which intervention atomicity supplies a mechanistic reading. Several limitations remain. (i) The core test is linear-Gaussian with two environments; the kernel extension addresses nonlinearity but a full multi-environment treatment is future work. (ii) The exists-invariant-set search is bounded in candidate-pool size and subset order, which caps recovery on very dense, high-dimensional shifts (e.g. ALARM at \(k=4\) dips to \(0.88\)). (iii) The C10 detector tests whether an invariant conditioning set exists, not whether the causal mechanism \(P(X_j\mid X_{\mathrm{pa}_j})\) changed. The two coincide under the conditions stated in Section 3.4 (faithful DAG with fixed structure, true parents contained in \(\mathcal{C}_j\), and no confounding by descendants or jointly shifting variables); on real data lacking these guarantees, C10’s set-valued output should be interpreted as an invariance diagnostic rather than a causal-localisation claim. (iv) The \(q_j\) statistic in Eq. (4) is the maximum p-value over candidate subsets, which is super-uniform only when the true parent set (or another invariant set) is contained in \(\mathcal{C}_j\). The application of BH-FDR to the resulting \(\{q_j\}\) across nodes assumes validity under dependence; while BH is known to control FDR under positive regression dependence (PRDS), this condition is not verified for the across-node dependence induced by shared data. These are practical rather than prohibitive concerns—the empirical performance in Tables 1 and 3 is strong—but they mean the detector does not carry a formal FDR guarantee without further assumptions. Two inexpensive checks would quantify the residual risk and we flag them as immediate next steps: reporting the realised false-discovery proportion against the nominal \(\alpha\) in the synthetic settings, where the changed set is known, and re-running the pipeline under the Benjamini–Yekutieli correction, which is valid under arbitrary dependence, to confirm that the verdicts are unchanged. (v) \(\hat{\rho}=\lvert\hat{S}\rvert/d\) is a detector output, not a consistent estimator of the true sparsity \(\rho\). BH-FDR controls the false-discovery proportion, not the size of the rejected set, so \(\hat{\rho}\) can be biased upward by false positives (as seen in the synthetic \(k{=}2\) case, where true \(\rho{=}0.17\) but \(\hat{\rho}{=}0.36\)) or downward by false negatives. The bootstrap CIs reflect resampling variation of \(\hat{\rho}\)’s expectation under the detector, not a confidence set for \(\rho\). We therefore treat \(\hat{\rho}\) throughout as a diagnostic output—the fraction of nodes flagged by the detector—and not as a sparsity estimate with formal estimation guarantees. (vi) The Sachs analysis treats discretised protein levels as continuous and adopts the standard intervention targets and a contested consensus graph; without a complete ground-truth changed set, the target-based precision in Section 4.4 is not strictly interpretable, and systematic cross-environment over-flagging (e.g. due to batch effects or discretisation artefacts) cannot be fully excluded by the source-split null alone. (vii) The sparse/dense cut-point \(\tau_{\mathrm{dense}}\) remains a semantic choice, although the lower null floor is data-referenced; stronger statistical guarantees on the verdict are an open direction. (viii) The atomic-versus-fat-hand study is a synthetic proxy for Perturb-seq; validating the prediction on real single-cell perturbation data is the natural next step. Its replication budget is also modest—6 seeds in Table 5, and 20 bootstrap resamples for the intervals in Table 4—which matters because Table 5 carries the mechanistic reading of the Sachs verdict; the separation there is categorical (every atomic run sparse, every fat-hand run dense), but a larger seed count and a stability analysis would place that reading on firmer ground. (ix) The image extension is a proof of concept: the disentanglement datasets we use (dSprites, 3D Shapes) have only \(d{=}5\)–\(6\) factors, so the verdict is dimension-limited (we rely on \(F_1\)), and the domain-consistent encoder uses target factor labels as a domain-coverage upper bound. A label-free realisation we attempted with an identifiable VAE (with BCE/KL-annealing tuning, \(K{=}16\) pseudo-environments, and a continuous-factor diagnostic) did not reach the labelled upper bound: its disentanglement (MCC) plateaus near \(0.5\), and the continuous-factor test indicates the bottleneck is the iVAE itself rather than discretisation; a stronger, intervention-tailored method (interventional causal representation learning ) is the natural route, which we leave to future work.

6. Conclusion

The sparse mechanism shift hypothesis underwrites a large body of causal approaches to distribution shift, yet is rarely tested. We contribute a diagnostic toolkit for this premise: a precise ablation of why mechanism-shift detection fails under an estimated graph, a graph-free invariance test that resolves it by checking whether any conditioning set restores invariance, and a data-referenced verdict (no shift / sparse / dense / inconclusive). Across controlled, semi-synthetic, real-structure and real-intervention data the verdict is broadly consistent with the true sparsity when it is known, and on real chemical interventions it returns a dense verdict well clear of the data-referenced no-shift floor, which we read as evidence against SMS in that setting. Making SMS falsifiable lets practitioners diagnose the assumption their methods rely on, rather than assume it.