By 2026, the operational reality of frontier model deployment is that . Models are integrated into customer-facing assistants, coding agents, enterprise workflow systems, and tool-using planners. Each deployment setting creates a large stream of interactions that, in principle, could contain safety-relevant failures: policy violations, misinformation with downstream harm, privacy leakage, illicit assistance, or more subtle forms of manipulation. In the small, we can imagine careful human review of each episode; in the large, that is infeasible. The dominant practical response has been to replace (or heavily front-load) human supervision with automated supervision: using another model, or a committee of models, to label outputs, critique reasoning traces, score compliance, and generate training signals.
This shift is visible in the learning pipelines that have become standard. Constitutional AI (CAI) and related critique-and-revise frameworks reduce reliance on expensive human labels by using a fixed set of principles to guide self-critique and refinement. Reinforcement learning from AI feedback (RLAIF) replaces direct human preference comparisons with model-generated preferences, often distilled from a smaller amount of human input. More broadly, ``Safe RLHF’’ in practice increasingly means: (i) train a policy model to be helpful, (ii) train a reward or evaluator model to detect unsafe behavior, and then (iii) optimize the policy against that evaluator at scale. These methods can be remarkably effective at improving behavior under common prompts, and they are economically attractive: they amortize supervision cost over huge volumes of interactions and enable continuous iteration.
However, the central risk we want to isolate is not that AI-based supervision is ``worse than humans’’ in some generic sense. The risk is that automated supervision can fail on the very regions of the interaction space that matter most for safety: low-probability, high-impact, strategically elicited failures. In other words, the oversight system may have . When the same families of models, training data, and evaluation templates are used across both the acting policy and the supervising evaluator, their errors need not be independent. They can be aligned in the worst possible way: the policy discovers behaviors that look safe to its evaluator, precisely because the evaluator shares the same vulnerabilities. This is not merely an ML generalization story; it is also an incentives story, because correlated blind spots alter the effective probability that harmful behavior is detected and punished.
We can phrase the problem in deployment terms. A platform wants its assistant to be responsive; responsiveness drives retention and revenue, and (in many applications) is intrinsically valuable to users. Let u > 0 denote the per-interaction value of compliance or answering. Safety interventions that force refusals, hedging, or safe redirections incur an opportunity cost: the assistant becomes less useful and the platform captures less value. Meanwhile, the most direct lever a regulator or internal governance process typically has is to impose penalties (formal fines, contractual liability, reputational loss, or mandated remediation) conditional on harmful events. This creates a familiar expected-cost tradeoff: comply and earn u, but risk a penalty L multiplied by the probability that the event is detected by oversight. The crucial detail is that detection is not a constant: it depends on the oversight pipeline, and it may collapse precisely on the tail events that are adversarially targeted.
This is where ``tail harms, not averages’’ becomes the right lens. Many evaluation regimes implicitly optimize expected performance under a natural prompt distribution. If adversarial prompts are rare—call their probability mass π—then the expected contribution of these cases to conventional metrics is small. Yet the security posture is governed by what happens when an attacker can concentrate queries on that tail region (or when a small but non-negligible user segment persistently induces those behaviors). In such regimes, a model can score highly on aggregate benchmarks while still admitting a coherent cluster of failures that is both exploitable and harmful. A correlated blind spot means this cluster is also poorly measured: the same automated evaluator that is used to gate deployment, trigger penalties, or generate training feedback may miss exactly those failures. From an economic standpoint, this reduces the effective expected penalty in the region that matters most, and so it weakens the deterrent effect of liability.
The interaction between optimization and correlated blind spots adds a further failure mode. Once a policy is trained against a fixed evaluator (or a slowly updated one), it has an instrumental incentive—in the narrow sense of gradient-based optimization—to find outputs that are scored as safe, not necessarily outputs that are safe. If the evaluator has a stable weakness on some cluster, the training process can amplify that weakness by steering behavior toward it. This is analogous to specification gaming, but the key point for governance is that the platform may believe it is ``paying’’ safety costs via penalties and audits, while in fact the penalty channel is informationally shut off for the important tail. In such a regime, increasing the nominal size of penalties L may not help: if detection probability is near zero on the relevant failures, the expected penalty remains small no matter how large L is (short of making L effectively unbounded or introducing alternative detection mechanisms).
The natural remedy is oversight diversification: introduce an
independent signal that breaks the correlation. In practice this could
mean human audits, external red-teaming, independent third-party
evaluation, or even a second AI evaluator trained on different data,
architectures, or objectives. The defining property is not
human-ness'' but \emph{error independence} relative to the acting policy and its primary evaluator. Independence matters because it converts tail failures fromalmost
never detected’’ into
detected with some positive probability,'' which can be enough to flip the platform's incentives on the margin. If a harmful behavior yields benefit $u$ but is detected with probability $\delta$ and punished by $L$, then the relevant quantity is $\delta L$; increasing $\delta$ through independent oversight can have disproportionate impact when the platform's decision is threshold-like (comply vs refuse). This is the intuitiveindependence
premium’’: small amounts of truly independent auditing can buy large
reductions in worst-case harm.
A key conceptual distinction we emphasize is between and . Platforms often adopt AI-based supervision because they must: interaction volumes are too large, and latency and privacy constraints make ubiquitous human review impractical. But if we accept that AI oversight is the scalable default, then we must ask what residual risks it creates at the system level. Correlated blind spots are exactly such a residual risk: they do not show up as a smooth degradation in average accuracy; they show up as concentrated vulnerabilities that remain undetected and therefore undeterred. This makes them uniquely dangerous in settings where an attacker can select prompts adaptively, where harmful behaviors can be chained across steps, or where downstream systems rely on the assistant’s outputs in ways that magnify rare failures.
From a policy design standpoint, this motivates two complementary approaches. First, we can view independent audits as a ``portfolio’’ choice: allocate some fraction of oversight budget to mechanisms whose errors are deliberately decorrelated from the mainline evaluator. Second, we can treat penalties and audit rates as coupled instruments. Penalties without detection are toothless; detection without meaningful penalties may not change incentives. The right unit of analysis is the expected marginal deterrence on the adversarial cluster, not the nominal strictness of the policy. This is also why transparency about evaluation pipelines matters: if the evaluator is closely coupled to the deployed policy (shared weights, shared data, shared training process), then the system may be overconfident about its own safety posture.
We should also be explicit about scope and limitations. Oversight failures are not the only reason a platform might comply with unsafe requests; misclassification of prompts, ambiguous policies, or genuine uncertainty about harm can all matter. Moreover, real systems are repeated games with reputation, dynamic updates, and heterogeneous users, not one-shot interactions. Nonetheless, the one-shot perspective is valuable because it isolates a minimal mechanism: when detection is imperfect and correlated with the policy’s failure modes, an incentive gap opens specifically on tail risks. The formalism we develop next abstracts away many complexities to spotlight this gap and to make a crisp comparative-statics claim: increasing independence in oversight can reduce worst-case harm more efficiently than either increasing penalties alone or improving an evaluator that shares the same blind spot.
Finally, we connect this framing back to training and verification practice. Many contemporary safety pipelines implicitly assume that stronger automated evaluation, more test coverage, and more fine-tuning will monotonically reduce risk. Our claim is more conditional: these efforts help in the regions the evaluator can see, but they may have sharply diminishing returns (or even perverse effects) on the regions it cannot. This pushes us toward governance mechanisms that explicitly target the tail: adversarially generated test distributions, mandated independent audits, evaluator diversity, and procedures that treat ``unknown unknowns’’ as first-class objects. In the next section, we make these intuitions precise by writing down a stylized deployment game in which a platform trades off the benefit of compliance against expected penalties under an oversight portfolio with correlated and independent components.
We model a single deployed interaction between a platform and a user as a one-shot game. The goal is not to capture the full complexity of repeated deployment, but to isolate a minimal mechanism by which oversight failures can create a persistent tail-risk incentive gap. The modeling choice that does most of the work is that detection is imperfect and depends on which oversight channel is applied; the platform internalizes penalties only when harms are .
There are two prompt types, indexed by t ∈ {B, A}. Type B denotes a benign interaction in which answering does not create a harmful event in our baseline abstraction. Type A denotes an : a coherent family of prompts for which answering would lead to a harm event (e.g., illicit instruction, privacy breach, or other policy-violating content). Under the prompt distribution, t = A occurs with probability π ∈ (0, 1) and t = B occurs with probability 1 − π. This π is meant to represent ``tail mass’’ under ordinary usage: small, but nonzero.
Because we also want to capture security-relevant behavior, we allow an attacker to target the cluster. Formally, we consider two regimes. In an regime, t is drawn from the natural distribution. In a regime, an attacker chooses an action a that sets t = A (equivalently, the attacker conditions its queries on being in the adversarial cluster). The latter is intended to capture settings where an adversary can concentrate attempts on a vulnerability class, rather than being forced to sample from the benign majority. We will focus on worst-case harm in later results, but we include π in the model to keep contact with conventional expected-risk evaluation.
After observing t, the
platform chooses a policy action σ ∈ {C, R}, where
C denotes
comply/answer'' and $R$ denotesrefuse/safe engage.’’ We
interpret R broadly: it
includes explicit refusals, safe completions that do not satisfy the
harmful request, requests for clarification, or escalation to a human
operator. The platform’s choice is allowed to depend on t in the baseline model, i.e., the
platform is not confused about whether the prompt belongs to the
adversarial cluster. This is a deliberate simplification: we want to
attribute the failure mode to (imperfect detection of harm conditional
on answering), rather than to (imperfect identification of which prompts
are dangerous). In practice, both matter; separating them lets us make
clean statements about the role of independence in evaluation.
Complying yields a gross per-interaction benefit u > 0 to the platform. This u is a reduced-form proxy for revenue, retention, or product value associated with being responsive. Refusing yields payoff 0 in the baseline. We can interpret the difference as the opportunity cost of safety interventions that reduce helpfulness.
If the platform answers on the adversarial cluster, harm occurs. We
encode this by a harm-event indicator
h(t, σ) = 1{t = A, σ = C}.
In the baseline, h(B, C) = 0 and
h(B, R) = 0;
the benign type never produces a harm event. This binary structure is
intentionally stark: it captures a setting where the adversarial cluster
is ``safety-relevant’’ in a way that benign prompts are not, and where
the platform faces a threshold decision (answer vs do not answer) rather
than a continuous degree of harm. Later extensions could relax this by
allowing partial harms or noisy harm generation, but the binary case
already captures the incentive discontinuity we care about.
Let ε ∈ [0, 1] denote the independent-audit probability (audit rate). With probability 1 − ε, the AI evaluator is used; with probability ε, the independent auditor is used. We emphasize that ε is a , not a capacity constraint: it can be interpreted as the fraction of interactions sampled for stronger review (ex post), or the fraction of deployments routed through an independent red-team or verification pipeline (ex ante), provided the detection event relevant for penalties is well approximated by a Bernoulli draw.
Detection is modeled conditional on a harm event occurring. If h(t, σ) = 0, there
is nothing to detect and no penalty is triggered. If h(t, σ) = 1,
detection occurs with a mechanism-dependent probability. The AI
evaluator detects harms on the adversarial cluster with
probability
δAI = 1 − ρ,
where ρ ∈ [0, 1] is the AI
evaluator’s miss probability on this adversarial cluster. We interpret
ρ as a reduced-form measure of
a : the evaluator systematically fails on the same cluster of cases that
the deployed policy may fail on (or may even learn to exploit under
optimization). In the extreme ρ → 1, the AI evaluator is almost
perfectly blind to the relevant tail harms; in the opposite extreme
ρ → 0, the AI evaluator
reliably flags such harms.
The independent auditor detects harms with probability δH ∈ (0, 1]. Crucially, we assume the independent auditor’s errors are independent of the AI evaluator’s errors on the adversarial cluster, and in particular that δH does not degrade as ρ increases. This is the formal way we express ``independence’’ as the scarce resource: the second signal is valuable insofar as it is not subject to the same failure correlation that undermines AI-only oversight.
Given the audit portfolio, the detection probability on a harm event
is the mixture
δ(ε) = (1 − ε) δAI + ε δH.
We can equivalently define a detection indicator Z(ε) ∼ Bernoulli(δ(ε))
that is realized only when a harm event occurs. The platform does not
observe Z(ε) at
decision time; it chooses σ
knowing only the detection probabilities.
If harm occurs and is detected, the platform pays a penalty L > 0. We treat L as encompassing regulatory fines, legal liability, mandated remediation costs, or reputational losses that are triggered by substantiated incidents. The key is that L is assessed on harmful events; undetected harms do not directly enter the platform’s payoff in the baseline model. This captures the practical governance constraint that enforcement is mediated by measurement.
The platform’s per-interaction utility is thus
UP(σ, t; ε) = u ⋅ 1{σ = C} − L ⋅ 1{t = A, σ = C} ⋅ Z(ε).
Taking expectations over detection conditional on t and σ, we obtain the expected payoff
comparison on type A:
𝔼[UP(C, A; ε)] = u − δ(ε)L, 𝔼[UP(R, A; ε)] = 0.
On type B, the expected payoff
is 𝔼[UP(C, B; ε)] = u
and 𝔼[UP(R, B; ε)] = 0.
For completeness, one can add a per-audit cost borne by the platform, caε with ca ≥ 0, to capture the resource burden of independent review. We will mostly omit this term in the baseline because our main comparative statics focus on the deterrence channel (how ε changes detection and hence expected penalties), but the extension is straightforward.
In the worst-case regime, an attacker selects t = A (or otherwise focuses
its query distribution on the adversarial cluster). The attacker
benefits from the platform answering on the adversarial cluster; we can
write
UA(σ) = 1{t = A, σ = C}.
Nothing essential depends on the attacker having this exact payoff; any
attacker objective that is increasing in the probability of harmful
compliance yields the same worst-case logic. The point is that, under
attack, the relevant risk metric is not expected harm under π but the platform’s action on A.
The interaction proceeds as follows. First, an oversight regime is specified: a regulator or internal governance process may set L and may mandate or recommend an audit rate ε; alternatively, the platform may choose ε endogenously subject to cost or capacity constraints. Second, the prompt type t is realized (either from the natural distribution or chosen adversarially). Third, the platform observes t and selects σ. Fourth, if h(t, σ) = 1, an oversight mechanism is applied (AI evaluator with probability 1 − ε, independent audit with probability ε), producing a detection outcome Z(ε) and triggering a penalty if Z(ε) = 1.
The informational assumption that the platform observes t is a modeling choice to isolate the effect of correlated blind spots in detection. If the platform also had imperfect knowledge of t, then refusals could occur on benign prompts and compliance could occur on adversarial prompts even absent incentive problems. Those effects are important in practice, but they are orthogonal to the mechanism here: even with perfect type recognition, the platform can rationally choose harmful compliance if the expected penalty is small because detection is weak on the adversarial cluster.
Although the baseline analysis is about platform incentives, it is
often useful to also represent social welfare. Let H > 0 denote the social harm
magnitude of an undeterred harmful event (which may exceed the penalty
actually imposed), and let kε denote the resource cost
of audits from a planner perspective. Then welfare can be written
as
W(σ, t; ε) = u ⋅ 1{σ = C} − H ⋅ 1{t = A, σ = C} − kε.
This makes explicit a familiar gap: the platform trades off u against an δ(ε)L, while
society trades off u against
the H (and may also value
audits differently). Our main focus in what follows is the private
incentive threshold created by δ(ε)L, because
correlated blind spots enter precisely through δAI = 1 − ρ
and hence through the effectiveness of AI-only enforcement.
In the next section we will solve for the platform’s best response on A as a function of δ(ε)L and show how even small independent auditing can move the system across a compliance/refusal threshold when δH is meaningfully larger than δAI. This is the sense in which independence functions like a portfolio diversification device for oversight: it does not need to be perfect or ubiquitous to change behavior on the tail, provided it is sufficiently decorrelated from the failure mode it is meant to detect.
We now solve the platform’s decision on the adversarial cluster and extract a simple bound that formalizes the ``independence premium’’: when the default evaluator has a correlated blind spot, even very large penalties may fail to deter harmful compliance unless we introduce some oversight that is with that blind spot. The logic is intentionally elementary—the point is that the safety failure mode arises from a threshold incentive created by imperfect detection, not from any subtle equilibrium refinement.
Fix an audit rate ε ∈ [0, 1]. Conditional on t = A, the platform
compares refusing (payoff 0) to
complying (benefit u minus an
expected penalty). Since a penalty is assessed only when harm is
detected, the relevant object is the effective detection probability
δ(ε). Using the
mixture structure from the previous section,
δ(ε) = (1 − ε)δAI + εδH.
Hence the expected payoff difference between compliance and refusal on
type A is
Δ(ε) := 𝔼[UP(C, A; ε)] − 𝔼[UP(R, A; ε)] = u − δ(ε)L.
The platform’s best response is therefore a cutoff rule:
$$
\sigma^*(A) \;=\;
\begin{cases}
C & \text{if } u > \delta(\varepsilon)L,\\
R & \text{if } u < \delta(\varepsilon)L,\\
\text{any mixture} & \text{if } u = \delta(\varepsilon)L.
\end{cases}
$$
On the benign type B,
compliance strictly dominates refusal in the baseline (no harm is
generated), so σ*(B) = C.
This pins down the unique pure-strategy equilibrium whenever u ≠ δ(ε)L.
In the attacker-chosen regime, the attacker sets t = A, so worst-case harm
is governed entirely by the platform’s action on A. Let Hwc(ε)
denote the worst-case harm probability. In the pure regimes above,
Hwc(ε) = 1{σ*(A) = C} = 1{u > δ(ε)L},
with the usual knife-edge caveat that when u = δ(ε)L
any mixed strategy yields intermediate harm probability. This
step-function dependence on ε
is the source of the ``disproportionate gains’’ phenomenon: once ε pushes δ(ε)L above u, behavior flips discretely from
always-comply to always-refuse on the adversarial cluster.
Consider AI-only oversight, ε = 0. Then δ(0) = δAI = 1 − ρ, and the compliance condition becomes u > (1 − ρ)L. This yields an immediate lower bound on worst-case safety.
If ε = 0 and u > δAIL, then σ*(A) = C and Hwc(0) = 1.
The platform internalizes only the portion of harm. When the evaluator has a correlated blind spot on the adversarial cluster (large ρ, small δAI), the expected penalty δAIL can be far below the nominal penalty L. In particular, as ρ → 1 we have δAI → 0, so no penalty can deter compliance: u > δAIL holds for any fixed u > 0 and L < ∞. This is the precise sense in which ``turning up the fine’’ is not enough when the measurement channel is informationally shut off on the tail.
This bound is not meant to claim AI oversight is always weak; rather, it isolates a failure mode that survives even if (i) the platform perfectly recognizes t = A, and (ii) the nominal penalty is large. The only ingredient is that detection is systematically low on the particular cluster that an attacker will target. Put differently, correlated miss probability ρ functions like an applied to L.
We now ask: how much independent auditing is required to deter harmful compliance on A? Since δ(ε) is affine in ε, and the platform’s best response depends only on whether δ(ε)L crosses u, we can solve for the minimal audit rate that guarantees refusal.
Assume δH > δAI
(otherwise audits provide no detection improvement, and the exercise is
vacuous). Solving u ≤ δ(ε)L
for ε yields
$$
\varepsilon \;\ge\; \frac{u/L - \delta_{AI}}{\delta_H - \delta_{AI}}.
$$
Clipping to [0, 1] gives the minimal
feasible threshold:
$$
\varepsilon^* \;:=\; \max\Bigl\{0,\ \min\Bigl\{1,\ \frac{u/L -
\delta_{AI}}{\delta_H - \delta_{AI}}\Bigr\}\Bigr\}.
$$
Suppose δH > δAI. Then for any ε ≥ ε* we have σ*(A) = R and hence Hwc(ε) = 0.
The threshold ε* is increasing in u and ρ (since δAI = 1 − ρ decreases with ρ), and decreasing in L and δH. These monotonicities have a direct incentive interpretation: higher compliance benefit or weaker default detection requires more independent auditing; higher penalties or stronger audit detection reduces the necessary audit rate.
The premium is not that audits are perfect, but that they are from the blind spot. Formally, the marginal effect of auditing on deterrence is governed by the detection gap δH − δAI. When the default evaluator is already good on the adversarial cluster (δAI high), audits buy little. When the default evaluator is systematically weak (δAI low), the same audit mechanism can dramatically increase δ(ε) even at small ε.
A particularly important regime is the one we are most concerned
about in practice: the adversarial cluster is precisely where automated
oversight has low sensitivity (large ρ), while independent mechanisms
retain nontrivial detection power (δH bounded away
from 0). In the extreme δAI ≈ 0,
the formula simplifies to
$$
\varepsilon^* \;\approx\; \frac{u/L}{\delta_H}.
$$
Two qualitative facts fall out.
First, it is entirely possible for AI-only oversight to fail in the worst case (because u > δAIL), while a small amount of independent auditing eliminates worst-case harm (because ε* can still be modest). Second, the dependence on ε is essentially kinked: below ε* the platform complies on A; at ε* it is indifferent; above ε* it refuses. In consequence, the social value of moving ε from 0 to a small positive number can be vastly larger than moving it from, say, 0.5 to 0.6, even though δ(ε) itself increases linearly.
From a governance perspective, this is the core actionable message: if we can identify clusters where ρ is high (evaluation blind spots that could be exploited), we should not interpret ``low observed incident rate’’ under AI-only pipelines as evidence of low worst-case risk. Instead, we should interpret it as a measurement artifact, and ask whether the system is above or below the deterrence threshold u ≤ δ(ε)L.
It is helpful to reframe ε not as ``how much human labor we throw at the system,’’ but as a portfolio allocation across evaluators with different error correlations. In our baseline mixture model, an audit draw is a . The only thing that matters for deterrence on the adversarial cluster is the induced detection probability δ(ε), which is linear in the portfolio weights.
This portfolio framing immediately suggests generalizations beyond a
single human auditor. For instance, suppose we add a second evaluator
whose miss events are approximately independent of the first on the
cluster of interest (whether or not either is ``human’’). If we flag
harm when evaluator flags, then under independence the joint miss
probability multiplies, yielding an effective detection
probability
δ = 1 − ρ1ρ2.
Replacing δH with 1 − ρ2 in the threshold
expression shows that functions as a substitute for a traditional audit.
The scarce resource is not a particular evaluator type, but a signal
that does not share the same blind spots as the policy and its default
monitor.
This also clarifies a central limitation of ``self-evaluation’’ training loops. If the policy and its evaluator are co-trained, or if evaluation uses the same model family with similar inductive biases, we should expect correlated errors on precisely those rare, high-stakes adversarial clusters that the policy can benefit from exploiting. In our notation, that correlation manifests as a high ρ on A, and therefore a low δAI exactly where we most need the enforcement signal. The independence premium bound quantifies why adding more optimization pressure to such a loop does not necessarily close the gap: incentives are controlled by δAIL, and δAI may be the bottleneck.
The independence premium bound is, at heart, a single inequality: harmful compliance is privately optimal on the adversarial cluster when u > δ(ε)L. AI-only oversight fails whenever δAI is sufficiently small relative to u/L, and in the limiting correlated-blind-spot regime (ρ → 1) no finite penalty can restore deterrence. Introducing an independent auditing channel raises δ(ε) linearly, but because the platform’s action is thresholded, the safety outcome can change discontinuously at the minimal audit rate ε*. This is why small, targeted independent audits can have outsize impact on worst-case harm, and why audit design is naturally understood as a portfolio allocation problem over partially decorrelated evaluators.
Our cutoff characterization makes an uncomfortable point about has incentives to supply independent oversight. The audit rate ε is a control knob that increases detection on the adversarial cluster, hence increasing the expected penalty conditional on harmful compliance. From a safety perspective this is exactly what we want; from a profit-maximizing platform perspective it is (in the one-shot model) almost purely a self-imposed tax. This wedge is what turns ``audit design’’ into a governance problem rather than a purely technical tuning problem.
To make the wedge explicit, suppose (as an optional extension of the
primitives) that the platform can choose the audit rate ε itself, paying a per-interaction
audit cost caε.
Holding fixed the platform’s action on type A, the expected payoff on A under compliance is
𝔼[UP(C, A; ε)] = u − δ(ε)L − caε,
while under refusal it is
𝔼[UP(R, A; ε)] = 0 − caε.
Two immediate observations follow. First, conditional on refusing on
A, auditing is strictly
dominated: it has no deterrence role (since harm never occurs) and only
incurs cost, so the platform sets ε = 0. Second, conditional on
complying on A, auditing
weakly reduces profit even if ca = 0, because
δ(ε) is increasing in
ε whenever δH > δAI,
and the expected penalty term δ(ε)L therefore
increases with auditing.
Formally, if δH > δAI,
then
$$
\frac{\partial}{\partial \varepsilon}\Bigl(u - \delta(\varepsilon)L -
c_a\varepsilon\Bigr)
\;=\; -(\delta_H-\delta_{AI})L - c_a \;<\; 0,
$$
so on the adversarial cluster, its optimal audit choice is again ε = 0. Putting both cases together,
in the baseline one-shot model the platform’s privately optimal audit
intensity is
εP* = 0,
independently of L, u, ρ, and δH (except for
the vacuous case δH = δAI
where auditing changes nothing). This is the cleanest statement of
``private incentives to under-audit’’: unless auditing creates some
countervailing private benefit not in the model (reputation, customer
retention, reduced litigation risk outside the detected-harm channel,
model improvement from audit feedback, etc.), an optimizing platform
either (i) refuses and does not audit, or (ii) complies and does not
audit. In neither case does it voluntarily purchase detection that only
increases expected penalties.
A regulator (or social planner) values compliance benefits but also
internalizes social harm and audit resource costs. In our notation, a
simple welfare objective is
W(σ, t; ε) = u ⋅ 1{σ = C} − H ⋅ 1{t = A, σ = C} − kε,
where kε captures
social audit costs (which can differ from caε).
The regulator’s is not σ
directly but rather a pair (L, ε) (or a mandate on
ε plus a penalty schedule),
anticipating the platform’s cutoff best response on A.
In the attacker-chosen (worst-case) regime, the relevant comparison
is stark: if the platform complies on A, welfare is u − H (no audit cost if
ε = 0); if the regulator
mandates a sufficient audit rate ε ≥ ε* so that
the platform refuses on A,
welfare is 0 − kε.
Hence, under worst-case targeting the welfare-optimal mandate has a
corner structure:
mandate ε = ε*
(to induce refusal) if and only if
H − u ≥ kε*,
and otherwise mandate ε = 0
and tolerate harmful compliance. This ``kink’’ mirrors the platform’s
threshold response: because behavior changes discontinuously at ε*, welfare also tends to
be optimized at corners (no audits vs. the minimal audits that cross the
deterrence threshold).
Under a natural prompt distribution with tail mass π (rather than an attacker forcing
t = A), the same
logic applies but the benefit-cost comparison is scaled by prevalence.
Since the platform observes t,
it can refuse only on A while
continuing to comply on B. If
audits are mandated at rate ε ≥ ε*, the
platform refuses on A,
yielding welfare (1 − π)u − kε*.
If audits are not mandated and the platform complies on A, welfare is u − πH. Thus, in
the expected-welfare variant the regulator prefers the deterrence
mandate when
(1 − π)u − kε* ≥ u − πH ⇔ π(H − u) ≥ kε*.
This condition makes two deployment-relevant points. First, the appetite
for audit mandates increases with tail mass π even when the worst-case analysis
is unchanged. Second, when H
is very large (catastrophic harm), a small π can still justify a mandate, but
only if ε* is not
itself prohibitively costly.
One might hope to avoid audit mandates by simply setting L high. Our earlier ``tail-risk lower bound’’ shows why this can fail: if AI-only detection δAI is small on the adversarial cluster, the platform internalizes only δAIL, so deterrence requires L ≥ u/δAI, which diverges as δAI → 0 (i.e., ρ → 1). In other words, penalties without measurement can be economically meaningless: the enforcement channel is informationally bottlenecked.
Even when δAI is not tiny, very large L can be administratively or politically infeasible (limited liability, due process constraints, insurance constraints, or the risk of over-deterrence leading to excessive refusal). In that regime, ε is the more direct lever because it changes via δ(ε) rather than attempting to compensate for low detection with unbounded fines.
The model suggests a useful translation layer between abstract parameters and governance instruments.
are naturally interpreted as lower bounds on ε (and/or requirements that δH − δAI
be nontrivial). For example, a standard might require that a fixed
fraction of interactions in specified high-risk domains be routed to an
evaluator that is operationally independent (separate team, separate
model family, separate data access), which in our notation is exactly
set $\varepsilon$ above $\varepsilon^*$ on the relevant cluster.'' The portfolio framing matters here: what the standard should demand is nothuman
review’’ per se but verifiable , i.e., independence relative to the
system’s dominant blind spots.
can implement the same constraint: large buyers can require certification that the vendor runs independent audits at rate at least ε̄ and can document detection performance δH on red-team suites representative of targeted clusters. Procurement is often a faster lever than legislation, and it can be tailored (higher ε̄ for higher-stakes deployments). In our terms, procurement is a way of selecting equilibria by constraining feasible (ε, L) pairs even if the statutory penalty L is modest.
primarily instantiate L,
but crucially can also be used to make auditing privately attractive
rather than privately dominated. The baseline has a per-detected-harm
penalty, which perversely makes higher detection privately worse. A
negligence-style design can invert this: impose large penalties for
harms , while offering safe harbor or reduced penalties when the
platform can demonstrate compliance with an approved independent-audit
process. In reduced form, such a rule makes the effective penalty
schedule depend on ε,
e.g.,
L(ε) decreasing in
ε (safe harbor),
or equivalently adds a separate penalty for under-auditing. This changes
the platform’s audit calculus by introducing a benefit to auditing
(penalty mitigation) that can dominate the ``detection tax’’ effect. Put
differently, the regulator can use liability not only to set the
magnitude of punishment but to reshape incentives so that the platform
internalizes the value of .
Several practical issues sit just under the surface of the toy model.
First, verifying that an
independent'' auditor is actually decorrelated from the system's blind spots is nontrivial: institutional independence (separate org chart) does not guarantee error independence. Second, if audits are predictable (non-random, announced in advance, or easily inferred), platforms may engage inaudit
gaming’’—deploying a different policy when they expect scrutiny—which
would break the mixture model and effectively reduce δ(ε). Randomization, secure
logging, and post-hoc reproducibility checks can be understood as
mechanisms to keep ε
meaningful.
Third, the most important under-audit incentive is often not the direct cost ca but the strategic cost of learning bad news: internal audits can create discoverable evidence, trigger incident response obligations, or increase expected future penalties. These channels are outside our one-shot formulation but point in the same direction: absent countervailing incentives, platforms will avoid generating adverse signals about tail behavior.
The independence premium bound tells us small ε can matter (high-ρ clusters); the audit design lens tells us we should not expect such ε to appear endogenously. In the baseline, the platform’s optimal audit choice is ε = 0, while the regulator’s optimal choice is typically a corner at either 0 or the minimal deterrence level ε*. That gap motivates policy instruments that (i) mandate independent oversight directly, or (ii) redesign liability so that auditing is privately beneficial rather than purely punitive.
The baseline cutoff model is intentionally stylized: it isolates the role of correlated evaluator blind spots without confounding it with classification limits (the platform observes t) or with multi-period learning. Here we sketch several extensions that remain analytically clean while addressing deployment-relevant deviations from the idealized assumptions. The common theme is that our main qualitative conclusions are robust: what matters for deterring tail harms is not merely , but evaluation whose errors are sufficiently from the system’s failure modes on the relevant adversarial clusters.
Independent audits are not omniscient. In practice, even
well-designed red-team pipelines miss attacks due to limited coverage,
ambiguous policy boundaries, and the difficulty of reproducing rare
failure modes. Modeling this is as simple as allowing δH ∈ (0, 1)
rather than δH = 1. The
effective detection probability remains
δ(ε) = (1 − ε)δAI + εδH,
so the platform’s best response on the adversarial cluster continues to
be governed by the same comparison: comply on A iff u > δ(ε)L.
Consequently, the minimal audit rate to deter harmful compliance is
unchanged in form,
$$
\varepsilon^* \;=\;
\max\Bigl\{0,\min\Bigl\{1,\frac{u/L-\delta_{AI}}{\delta_H-\delta_{AI}}\Bigr\}\Bigr\},
$$
but the interpretation becomes sharper. First, if δH ≤ δAI,
then independent audits are not actually
independent enough'' in the relevant sense: increasing $\varepsilon$ does not improve detection on the adversarial cluster, so there is no independence premium and no finite $\varepsilon$ can flip the cutoff (unless $L$ itself changes). Second, even when $\delta_H>\delta_{AI}$, imperfect audits imply a hard upper bound $\delta(\varepsilon)\le \delta_H$, so deterrence requires $u\le \delta_H L$. If $u>\delta_H L$, then no audit mixture can deter compliance in a one-shot setting because the enforcement channel is too weak even under maximal auditing. This upper-bound phenomenon is a useful reminder thatmandate
audits’’ is not a universal fix; one needs either sufficiently accurate
independent evaluation (raise δH),
sufficiently strong penalties (raise L), or a change in product design
that reduces the marginal value of unsafe compliance (lower u on harmful continuations via
policy constraints or capability shaping).
A natural substitute (or complement) to human or third-party audits
is evaluator diversity: run multiple evaluators and flag harm if
evaluator flags. When evaluators’ misses are independent on the
adversarial cluster, detection improves multiplicatively. Concretely,
suppose we have m evaluators
indexed by i, each with miss
probability ρi on the
adversarial cluster (so detection δi = 1 − ρi),
and we flag if at least one flags. Under conditional independence of
misses,
$$
\delta_{\mathrm{ens}} \;=\; 1-\prod_{i=1}^m \rho_i.
$$
This formula makes explicit why
more of the same evaluator'' does not help: if the additional evaluators share the same correlated blind spot, the effective $\rho_i$ are not independent and the product-form gain collapses. In deployment terms, we should not think of diversity astwo
copies of the safety model,’’ but as evaluators drawn from different
model families, training data regimes, or oversight objectives, ideally
with demonstrably different error surfaces on adversarial suites.
Once we have an ensemble detection rate δens, the earlier cutoff analysis carries through with the substitution δAI ⇝ δens (or, in a portfolio with occasional human audits, δ(ε) = (1 − ε)δens + εδH). This yields a simple engineering governance implication: if we can certify that the joint miss probability ∏iρi is small on the relevant tail cluster, then we can reduce the required independent audit rate ε*, sometimes dramatically. But the certification burden shifts to demonstrating —an empirical claim about joint errors, not just marginal accuracies. That is precisely the kind of claim we will need to operationalize in the validation plan that follows.
In real systems there is rarely a single adversarial cluster. Instead we can index clusters by j ∈ {1, …, J}, with potentially different evaluator miss rates and audit detectability: δAI, j = 1 − ρj and δH, j, and potentially different harm magnitudes Hj (for social welfare) or even different platform benefits uj (if some harmful behaviors are also especially engagement-driving). An attacker who can adaptively choose the cluster will target whichever cluster maximizes their success probability.
In the one-shot worst-case regime, this adaptation is captured by
treating nature/attacker as selecting the type after observing the
platform’s policy: t = Aj
is chosen to maximize 1{σ(Aj) = C}
(or, in mixed/continuous variants, to maximize the probability/extent of
harmful compliance). If the platform observes j and can condition its action on
it, then deterrence requires satisfying the cutoff constraint for
cluster:
uj ≤ δj(ε)L for
all
j, δj(ε) := (1 − ε)δAI, j + εδH, j.
This immediately yields a
max of thresholds'' structure: \[ \varepsilon \;\ge\; \max_{j}\varepsilon_j^*, \qquad \varepsilon_j^*:=\max\Bigl\{0,\min\Bigl\{1,\frac{u_j/L-\delta_{AI,j}}{\delta_{H,j}-\delta_{AI,j}}\Bigr\}\Bigr\}. \] The interpretation is that adaptive attackers push us toward the hardest-to-deter cluster (high $u_j$, low $\delta_{AI,j}$, and/or low $\delta_{H,j}$). This formalizes awaterbed’’
failure mode: improvements in oversight that raise δAI, j
or δH, j
for some clusters may simply shift attacks to other clusters unless the
entire tail is covered above threshold. From a governance perspective,
it suggests that reporting a single aggregate detection number is not
decision-relevant; we need cluster-conditional (or
attack-family-conditional) estimates, and we should expect the limiting
factor to be the worst uncovered slice.
One can also consider the non-worst-case regime where clusters occur under a distribution and the attacker has limited ability to concentrate mass. Then the objective becomes trading off expected harm versus audit costs, but the same structural point persists: as adversaries become more adaptive, the effective tail mass concentrates on whichever region has the largest residual compliance probability, amplifying the importance of the minimum detection guarantee across clusters.
Binary comply/refuse is a useful abstraction, but many deployed
safety mitigations are better modeled as : partial answers,
safe-completion templates, or constrained tool access. We can capture
this by letting the platform choose an action x ∈ [0, 1] on the adversarial
cluster, where x indexes
``degree of compliance.’’ Let the private benefit be b(x) with b(0) = 0, b′(x) > 0,
and let the harm probability (or expected harm indicator) be h(x) with h(0) = 0, h′(x) ≥ 0. Under
detection probability δ(ε) and penalty L, the platform solves
maxx ∈ [0, 1] b(x) − δ(ε)L h(x).
When b and h are smooth and strictly
increasing, the optimal x*(ε) typically
satisfies the first-order condition
b′(x*) = δ(ε)L h′(x*),
with boundary solutions when the equality cannot be met in the interior.
This makes the role of auditing transparent: increasing ε increases δ(ε) (when δH > δAI),
which increases the effective marginal price of harm and therefore
reduces x*(ε). In
contrast to the binary model, the response can be continuous rather than
a discrete flip from C to
R.
However, the ``small ε big gain’’ logic can reappear whenever h(x) is . For instance, if h(x) is close to 0 for x ≤ x̄ but rises sharply beyond x̄ (capturing a regime where low-effort refusals or guarded responses are safe, but higher-fidelity compliance unlocks harmful capability), then modest increases in δ(ε) can push the optimizer from x > x̄ to x ≤ x̄, producing a large drop in harm with relatively small additional oversight. Put differently, the discontinuity in the binary model is an idealization of a steep region in the harm curve; the independence premium is the ability to shift behavior leftward across that steep region by improving enforceability on the tail.
This continuous formulation also clarifies what it would mean to ``pay for safety without sacrificing all utility’’: instead of inducing full refusal on A, the regulator might aim to induce a safe partial-compliance level x̃ satisfying b′(x̃) ≤ δ(ε)Lh′(x̃). That kind of target is closer to many product policies, and it suggests that empirical estimates should measure not only whether harms happen, but how harm rates vary with response fidelity and tool access.
Across all four extensions, the governing object remains the effective detection probability on the relevant tail events—possibly cluster-indexed, possibly joint across evaluator portfolios, and possibly interacting with a smooth policy choice. What changes is to make the model operational: not just marginal detection, but cluster-conditional detection; not just single-model accuracy, but joint miss rates across ``families’’ of evaluators; not just binary harm incidence, but harm as a function of response policy. These are exactly the empirical questions we take up next when we discuss how to validate (and potentially falsify) the correlated-error parameterization via evaluator families, adversarial suites, and adaptive red teaming.
Our theoretical claims hinge on an empirical object that is easy to name but nontrivial to measure: the evaluator miss probability on the relevant tail cluster, ρ, and the extent to which misses are across the oversight tools we deploy. In practice, this is a question about failure rather than pointwise accuracy. The validation problem is therefore twofold: (a) identify and maintain a representation of adversarial clusters that is stable enough to track over time, and (b) estimate cluster-conditional detection for each oversight channel (AI evaluators, independent auditors, and portfolios), including miss rates that witness (de)correlation.
The symbol A in the model should be read as an equivalence class of prompts and interaction traces that share a common exploit mechanism. Empirically, we do not want to define A as ``whatever the evaluator missed’’ (which would be tautological). Instead, we propose to define clusters via annotations (e.g., jailbreak style, tool misuse pattern, policy boundary ambiguity, multi-turn coercion) and then refine them with (e.g., clustering embeddings of prompts, tool traces, and model rationales). Concretely, we maintain a catalog {Aj}j = 1J with:This ensures that δAI, j and δH, j are estimands tied to a stable slice of behavior, not to a moving target defined by the evaluator itself.
In the baseline notation, δAI = 1 − ρ is the probability the AI oversight channel detects harm on the adversarial cluster. The key subtlety is that production logs are filtered by prior defenses (rate limits, refusals, etc.) and by the evaluator under measurement (if the evaluator gates escalation). To avoid selection artifacts, we recommend a data design:Within each cluster Aj, we
estimate
δ̂AI, j = Pr (sAI = 1 ∣ h = 1, t = Aj),
where h = 1 is established by
a higher-fidelity reference process (see Step 2). Because h = 1 can be rare even within a
cluster definition, we recommend stratified sampling plus importance
weights when aggregating across clusters (especially if we later move
from worst-case to expected-welfare variants). We also recommend
reporting on δAI, j,
since governance decisions based on expected penalties are sensitive to
optimistic detection estimates.
We then estimate
δ̂H, j = Pr (sH = 1 ∣ h = 1, t = Aj),
and, crucially, we estimate a miss rate to test decorrelation:
ρ̂j joint = Pr (sAI = 0, sH = 0 ∣ h = 1, t = Aj).
If AI and auditors were conditionally independent given h = 1 and t = Aj,
we would expect ρjjoint ≈ (1 − δAI, j)(1 − δH, j).
Deviations quantify correlation directly. A convenient summary is the
log-odds ratio (or mutual information) of misses; we can also report the
implied ``effective independence’’ penalty by comparing observed joint
miss to the independence baseline.
The parameter ρ is best
understood not as a fixed scalar, but as a property of an evaluator on a
particular failure surface. We therefore propose to instantiate a set of
evaluators ℰ = {E1, …, Em}
that vary along axes plausibly affecting error correlation: base model
lineage, training data provenance, prompting templates, reward models,
and explicit policy formalizations. For each cluster Aj we
estimate:
ρ̂i, j = Pr (sEi = 0 ∣ h = 1, t = Aj), ρ̂i, k, j joint = Pr (sEi = 0, sEk = 0 ∣ h = 1, t = Aj).
This supports two practical outcomes. First, it lets us
correlated errors'' as a measurable overlap structure (which pairs fail together, on what clusters, and how often). Second, it enables portfolio design: for anOR’’
ensemble that flags if any evaluator flags, the empirical detection rate
is
δ̂ens, j = 1 − Pr (∀i, sEi = 0 ∣ h = 1, t = Aj),
which we can estimate directly without assuming independence. This is
exactly the quantity that matters for enforcement in the one-shot
deterrence inequality.
Our model’s starkest guarantee is worst-case: an attacker can target
the cluster where compliance remains attractive. Empirically, this means
we should not only report average harm rates under a fixed test
distribution, but also measure the system under an adversary that
reallocates effort toward uncovered clusters. We propose an
adaptive red team as bandit'' protocol: \begin{itemize} \item maintain a menu of attack generators indexed by cluster $j$; \item run sequential rounds where the red team chooses $j_t$ based on past success; \item measure success as harmful compliance (or graded harm) under the deployed policy; \item report the achieved maximum over time and the identifiedhardest’’
clusters.
\end{itemize}
This procedure gives an empirical counterpart to the theoretical max
over clusters. Importantly, adaptivity introduces statistical dependence
(the test set is not i.i.d.), so we recommend sequential confidence
procedures (e.g., always-valid p-values or anytime confidence sequences)
to avoid over-claiming improvements when the attacker adapts.
Given empirical estimates δ̂AI, j
and δ̂H, j,
the model predicts an audit threshold (clusterwise)
$$
\widehat{\varepsilon}^{\,*}_j
\;=\;
\max\Bigl\{0,\min\Bigl\{1,\frac{u_j/L-\widehat{\delta}_{AI,j}}{\widehat{\delta}_{H,j}-\widehat{\delta}_{AI,j}}\Bigr\}\Bigr\},
$$
with the worst-case requirement ε̂ * = maxjε̂j *
under an adaptive attacker. Two practical notes matter here. First, we
should treat uj as (or
bounded) rather than assumed: it can be proxied via engagement,
retention, or revenue attributable to compliance on the cluster, but
those proxies are noisy and can perversely reward harmful content.
Second, we should propagate uncertainty: because the policy cutoff
depends on products like δL, conservative governance
should use lower bounds on detection and upper bounds on benefit (or
directly compute a high-confidence upper bound on ε*).
The key empirical signature is a steep change in qAj(ε)
(or a sharp drop in harm) around a small audit rate. We can further test
mechanism by verifying that the detection rate changes as
predicted:
δ̂j(ε) ≈ (1 − ε)δ̂AI, j + εδ̂H, j,
and that the platform’s action responds primarily through this channel
rather than through unrelated shifts (e.g., changes in user mix). In
high-stakes settings we may need to run these experiments in sandboxed
environments or with synthetic-but-realistic attacks to avoid exposing
users to harm; the point is to validate the of behavior with respect to
enforceability.
If these issues dominate, the right conclusion is not that the framework is useless, but that the empirically relevant object is larger than a scalar ρ: we need a time-indexed, cluster-indexed map of joint errors, with explicit controls for adaptivity.
Taken together, this plan turns ρ from a rhetorical gesture at ``blind spots’’ into an auditable measurement program: define clusters, estimate cluster-conditional detection and joint misses across evaluator families, stress-test under adaptive red teaming, and then run small-ε interventions to see whether improved independence produces the predicted nonlinear safety gains. This sets up the governance discussion that follows, because it clarifies what third parties would actually have to measure—and what kinds of independence claims can, in principle, be certified.
The empirical program above is not just ``how to measure ρ’’; it also clarifies what governance can and cannot buy in 2026. The central governance implication of the model is that oversight is valuable primarily through an channel: it raises the expected cost of harmful compliance by increasing the effective detection probability δ(ε). This framing is deliberately agnostic about whether oversight is performed by humans, AI systems, or hybrids. What matters is whether the oversight portfolio introduces on the relevant tail clusters, so that a targeted attacker cannot reliably route around the penalty trigger.
A common failure mode in deployment governance is to treat evaluation
as an epistemic exercise
(do we know the model is safe?'') rather than an incentive instrument (is
harm privately dominated for the platform under attack?’’). Our one-shot
analysis makes the enforcement character explicit: on adversarial type
A, compliance is chosen
whenever u > δ(ε)L.
If the detection channel is weak on the tail (low δAI due
to high ρ), then increasing
internal effort on policies, prompting, or even red-teaming can fail to
change the platform’s decision in the worst case, because the expected
penalty remains too small. This is one reason why purely internal
AI-only oversight can look strong on average-case benchmarks yet remain
brittle under targeted attack: the governance-relevant object is not
average accuracy, but the lower tail of detection on attack-relevant
clusters.
In 2026, we expect regulators and enterprise buyers to increasingly
operationalize requirements in terms of and rather than general
statements of
safety processes.'' The model suggests why: when the platform’s product incentives push toward compliance (high $u$), any credible harm constraint must either increase $L$ (liability/penalties) or increase $\delta(\varepsilon)$ (independent detection). Absent one of these levers,voluntary
commitments’’ are structurally fragile against both attackers and
competitive pressures.
If independence is the scarce resource, then an immediate governance question is whether it can be purchased reliably. We should expect a real audit market to form around three services: (i) (labeling, incident review, cluster-specific test suites), (ii) (attestations about processes and independence properties), and (iii) (forensics, root-cause, and remediation verification). The risk is that the market equilibrates to audits that satisfy checkbox compliance but do not meaningfully increase δ(ε) on the clusters that matter. In our notation, such a failure corresponds to δH ≈ δAI, or worse, high joint-miss probability even when marginal detection looks adequate.
This suggests a design principle for audit markets: the product
should be priced and regulated around on tail clusters, not around labor
hours or report length. Conceptually, the
independence premium'' is $\delta_H-\delta_{AI}$ (or more precisely the reduction in joint miss relative to an AI-only baseline). Auditors with genuine independence and competence will be capacity-constrained; if governance regimes simply mandatethird-party
audits’’ without specifying independence properties, the market will
predictably fill with low-signal offerings. Conversely, if procurement
and regulation explicitly reward demonstrated reductions in joint miss
on pre-specified adversarial clusters, the market has a target that
aligns with the deterrence logic.
A second predictable market failure is . Platforms have strong incentives to select auditors who minimize disruption to shipping cadence and public risk. The natural countermeasure is to separate who pays from who selects: e.g., regulator-approved auditor rosters, randomized auditor assignment for high-risk systems, or pooled funding mechanisms. Even lighter-touch interventions help: mandatory disclosure of financial relationships, limits on consulting services provided by auditors to auditees, and rotation requirements. These are standard tools from financial auditing, but our model highlights a distinct technical reason they matter: capture tends to increase error correlation, reducing the very independence that makes ε > 0 valuable.
A related 2026 trend is the emergence of third-party evaluator organizations that are neither purely auditors nor purely red teams. Their comparative advantage is maintaining diverse evaluator stacks and continuously updated cluster catalogs {Aj}, then selling (or publishing) portfolio-level detection curves. In our framework, a third-party evaluator can be understood as a supplier of of δAI, j, δH, j, and joint-miss structure, along with operational protocols that reduce induced correlation (blinding, tool separation, and controlled access to platform artifacts).
This role matters because platforms face a commitment problem. Internally, they can always claim that their evaluator is ``good enough’’; externally, stakeholders need comparability across providers and across time. Third parties can provide that comparability by standardizing: (i) cluster taxonomies, (ii) reference labeling protocols, (iii) reporting of lower confidence bounds on detection, and (iv) versioning of suites to track nonstationarity. In particular, when attackers adapt, static benchmark scores are weak evidence. What a third party can credibly sell is an evaluation regime that approximates the worst-case behavior we model.
The governance challenge is that independence is easy to assert and
hard to verify. We therefore want standards that are both (a)
technically meaningful in terms of reducing correlated misses, and (b)
auditable themselves. In our terms, independence claims should be
grounded in evidence about joint errors on the relevant tail. At
minimum, an independence standard should require reporting (by cluster)
of quantities like
Pr (sAI = 0, sH = 0 ∣ h = 1, t = Aj),
and should treat high joint-miss on high-severity clusters as a red flag
even if marginal detection δH appears
strong.
None of these guarantees perfect independence, but they are concrete enough to be inspected and, importantly, they target correlation mechanisms rather than superficial organizational separation.
The model’s ``AI-only tail-risk lower bound’’ is not a claim that AI evaluators are useless; it is a claim about the limits of oversight under worst-case selection. In 2026 deployments, there are at least four reinforcing reasons this limitation persists.
First, shared training and shared incentives induce common blind spots. If the policy model and evaluator are trained on similar data and optimized with similar reward signals, their errors will be correlated on precisely the regimes that are underrepresented, ambiguous, or strategically rare. This is the informal content of high ρ.
Second, adaptive attackers search for vulnerabilities. Even if the evaluator improves on average, the attacker reallocates to the clusters where detection remains weakest. This selects for tail events, effectively driving the interaction toward the worst-case A in our model rather than the average-case mixture.
Third, internal evaluators are susceptible to Goodharting through operational feedback. Once evaluator scores become release gates, teams optimize to pass the evaluator (raising apparent δAI on the measured distribution) without necessarily reducing harm under distribution shift. Without an independent check, it is difficult to distinguish genuine robustness from evaluation overfitting.
Fourth, AI-only oversight tends to become self-referential at scale: the same model family generates content, judges content, and proposes mitigations. This may be efficient, but it amplifies correlation. In our notation, it pushes the system toward a regime where δAI is precisely the quantity most likely to be overestimated and least likely to generalize to the tail.
These limitations motivate a governance stance that treats AI-only oversight as a layer (cheap, high coverage, fast iteration), but not as a sufficient condition for worst-case safety in high-stakes settings. The formalism makes the sense in which this is true: if δAIL cannot exceed u on the relevant tail clusters, then no amount of internal scoring changes the equilibrium action under attack.
In practice, governance will rely on a mix of mandates (minimum audit rates for certain capability tiers), procurement requirements (buyers demanding attestations), and liability regimes (increasing L for certain harms). Our framework suggests that disclosure should be designed to support two downstream decisions: (i) whether a given oversight portfolio plausibly achieves u ≤ δ(ε)L on the worst relevant clusters, and (ii) whether the independence claims are credible (low joint-miss).
This pushes toward reporting standards that are more like ``safety balance sheets’’ than marketing claims: cluster-indexed detection estimates with uncertainty, joint-miss measures, and clear statements about what adversarial protocols were used. Importantly, we do want to require disclosure of exploit details that would trivially enable attackers; the right abstraction level is precisely the cluster mechanism description and aggregate detection statistics. The measurement plan above is compatible with controlled disclosure: third parties can verify in private while publishing only aggregate metrics and confidence bounds.
Finally, we should be explicit about what this framework does not settle. First, the one-shot deterrence model ignores dynamics: in reality, audits also generate training signals, and attackers and defenders co-evolve. This can either amplify or dampen the independence premium depending on whether auditing causes evaluator convergence (reducing independence over time) or continual diversification (maintaining it). Second, we abstract away from partial compliance and nuanced mitigations (e.g., safe completion, tool restrictions). These interventions effectively change u and the mapping from σ to harm; they are crucial in practice and can be incorporated by enriching the action space, but the basic correlation issue remains.
Third, there is a strategic layer in how audits are allocated. If audits are predictable, attackers may time or shape attacks to avoid them; if audits are adaptive, platforms may route traffic in ways that change what auditors see. This argues for governance that treats audit allocation as an adversarially robust design problem, not merely a sampling problem. Fourth, independence itself can be eroded by standardization: shared rubrics and shared benchmarks can unintentionally synchronize evaluators. The governance challenge is to standardize while maintaining diversity in .
These considerations motivate the more prescriptive discussion that follows: if independence is the relevant scarce resource, then we need concrete minimum requirements, portfolio-level diversity metrics, and reporting norms that make tail-risk legible to regulators and buyers without leaking sensitive exploit information. The next section therefore translates the above logic into policy recommendations that are implementable under 2026 institutional constraints.
The core takeaway of the formalism is that many ``safety evaluations’’ fail precisely when they are most needed: on attacker-selected tail clusters where oversight errors are correlated. In that regime, additional internal effort can increase average scores without changing the platform’s privately optimal deployment action on the relevant adversarial type. The governance question is therefore not merely whether a system be evaluated, but whether the deployed oversight portfolio makes harmful compliance under plausible worst-case targeting. In our notation, the operational objective is to ensure that for the relevant high-severity clusters, the platform faces a regime where u ≤ δ(ε)L holds with high confidence, using a detection portfolio whose joint-miss structure is itself robust to Goodharting and adaptation.
We close by translating that objective into three concrete, 2026-implementable recommendations: (i) minimum independent audit requirements expressed as portfolio constraints (not process checklists), (ii) evaluator-diversity metrics that directly penalize correlated blind spots, and (iii) reporting standards that make tail-risk legible without leaking exploit details.
A useful governance mistake to avoid is specifying ``third-party audits’’ as a binary attribute. The model makes clear why: the deterrence lever is δ(ε), and ε > 0 is only valuable to the extent that the alternative evaluator is with the platform’s failures (i.e., δH > δAI on the clusters that matter, and joint-miss is low). We therefore recommend that minimum audit requirements be framed in terms of and , rather than auditor identity.
Concretely, regulators and enterprise buyers can implement a tiered
requirement of the form:
$$
\varepsilon \;\ge\; \varepsilon_{\min} \quad \text{and} \quad
\underline{\delta}(\varepsilon) \, L_{\text{tier}} \;\ge\;
u_{\text{tier}},
$$
where $\underline{\delta}(\varepsilon)$
is a conservative (e.g., lower-confidence) estimate of portfolio
detection on specified adversarial clusters, and (Ltier, utier)
are tier-dependent parameters reflecting harm severity and the
operational pressure to comply. The conceptual point is not that a
regulator can directly observe u, but that governance can
approximate the compliance pressure via proxies (product-critical flows,
latency constraints, revenue concentration, customer SLAs) and set
conservative utier
accordingly. For high-stakes tiers (biosecurity, cyber offense
enablement, child safety, critical infrastructure), we should treat
utier as high,
because platforms are predictably punished by the market for false
refusals in these lucrative segments.
When a single inequality is needed for enforceability, the closed
form from the model suggests a simple compliance test: require that the
platform’s declared audit rate meet or exceed a calibrated
threshold
$$
\varepsilon \;\ge\; \varepsilon^\star \;=\;
\max\Bigl\{0,\min\Bigl\{1,\frac{u/L-\delta_{AI}}{\delta_H-\delta_{AI}}\Bigr\}\Bigr\},
$$
but with δAI and
δH
replaced by $\underline{\delta}_{AI}$
and $\underline{\delta}_H$ computed on
pre-registered adversarial suites (or cluster catalogs) and evaluated
under blinding and access controls. This substitution matters: if the
governance regime allows point estimates without uncertainty, platforms
will rationally push measurement variance into the tail and overstate
deterrence.
We also recommend two operational complements to minimum ε mandates. First, audits should be : the sampling policy that determines which interactions are audited should be unpredictable to the platform’s routing logic and to external attackers. A simple rule is to require that a regulated fraction of audits be assigned via a verifiable random process over a well-defined eligible set (e.g., all sessions in a capability tier), with strict limits on platform-side filtering. Second, audit requirements should include a small but nonzero budget for audits triggered by external reports, since worst-case harms frequently arise in precisely the regions that pre-deployment test suites miss.
Finally, minimum requirements should be coupled to credible penalties. In the model, L is the other lever, and weak liability regimes predictably push governance toward impractically high ε. A pragmatic policy is therefore ``moderate ε plus meaningful L’’ rather than attempting to sample away risk. This pairing better matches institutional capacity constraints in 2026: independent auditors are scarce, and governance should spend their effort where it buys the most reduction in joint miss.
If independence is the scarce resource, we need metrics that reward it directly. Accuracy-style metrics (overall detection rate, AUC, benchmark pass rates) are compatible with synchronized failures: two evaluators can each look ``good’’ marginally while missing the same tail events. We therefore recommend that standards explicitly include and measures, reported by cluster and severity.
A minimal required statistic is the cluster-conditional joint-miss
probability:
Jj := Pr (sAI = 0, sH = 0 ∣ h = 1, t = Aj),
together with the marginals Pr (sAI = 0 ∣ h = 1, t = Aj)
and Pr (sH = 0 ∣ h = 1, t = Aj).
The governance-relevant object is not only that δH exceeds δAI,
but that Jj is small on
the high-severity clusters. Where privacy or operational constraints
prevent direct observation of h = 1, an acceptable surrogate is to
define Aj
suites with ground-truth labeling via controlled red-teaming, escrowed
incident review, or structured expert elicitation under audit.
To summarize diversity into a single scalar suitable for procurement,
we recommend a normalized ``independence premium’’ score for each
cluster:
$$
\mathrm{IP}_j \;:=\; 1 \;-\;
\frac{J_j}{(1-\delta_{AI,j})(1-\delta_{H,j})},
$$
where IPj ≈ 0
indicates near-independence (joint miss close to the product of misses)
and negative values indicate harmful correlation (joint miss worse than
independence would predict). This form is intentionally diagnostic: it
does not merely punish low detection, but punishes correlation
mechanisms that undermine the value of adding evaluators.
For regimes that use multiple AI evaluators (or multiple audit
providers), standards should additionally require reporting the as
evaluators are added:
δport(m) := 1 − Pr (all
m evaluators
miss ∣ h = 1, t = Aj),
because this curve directly measures the returns to evaluator diversity.
If adding evaluators yields little increase in δport, then the system is
in a correlated-blind-spot regime even if each evaluator appears strong
in isolation.
Two practical cautions matter in 2026. First, diversity is not a property of organizational charts; it is a property of error surfaces. We should therefore treat ``different vendors’’ as weak evidence unless accompanied by measured reductions in joint miss. Second, evaluator diversity can decay over time due to standardization and feedback loops (shared rubrics, shared datasets, model distillation from common sources). We accordingly recommend that diversity metrics be re-estimated on a cadence aligned with major model releases and evaluator updates, with change logs that make convergence legible.
The third recommendation is to move from narrative assurance to quantitative tail-risk reporting that is both (i) decision-relevant for deterrence and (ii) safe to publish. We propose a reporting template with three layers.
Platforms (or third-party evaluators) should report, for each high-severity cluster Aj, conservative estimates such as lower confidence bounds $\underline{\delta}_{AI,j}$, $\underline{\delta}_{H,j}$, and $\underline{\delta}_j(\varepsilon) = (1-\varepsilon)\underline{\delta}_{AI,j} + \varepsilon \underline{\delta}_{H,j}$. These should be accompanied by sample sizes, suite construction procedures, and drift indicators. The key norm is that reported detection should be : it is acceptable (and expected) that detection varies sharply across clusters, and governance should focus on the minimum over relevant j rather than the mean.
To make independence claims auditable, reports should include Jj (or a conservative upper bound) and a description of the mechanisms used to reduce induced correlation: blinding, tooling separation, restrictions on cross-training, and controls on evaluator access to platform artifacts. Importantly, this disclosure is about and , not exploit details. One can report that ``cluster Abio has Jbio ≤ 0.05 under protocol P’’ without describing the exact prompts that instantiate Abio.
Finally, reports should include a small set of standardized summary numbers that support procurement and regulatory triage. A plausible minimum set is:These are deliberately simple: governance failures in practice often arise not from lack of data, but from metrics that cannot be operationalized into go/no-go decisions.
We should be explicit about what these recommendations do and do not guarantee. They do not eliminate harm in an absolute sense; rather, they create a governance regime in which (i) the platform has credible incentives to refuse or safely engage on attacker-selected tail clusters, and (ii) independence claims are measured rather than asserted. The hard remaining problems are dynamic: maintaining diversity under feedback, preventing audit gaming as audits become predictable, and defining cluster catalogs that are stable enough to regulate yet adaptive enough to track attacker innovation.
Our claim is nonetheless that these recommendations are feasible under 2026 constraints and are the right next step: they directly target the correlated-blind-spot failure mode that makes AI-only oversight brittle. If we adopt minimum independent audit requirements expressed as portfolio constraints, measure evaluator diversity via joint-miss and detection curves, and standardize tail-risk reporting with uncertainty, then the safety case becomes less about narratives of ``best effort’’ and more about verifiable deterrence on the clusters that adversaries will actually choose.