feat(disposition): separate the assessment axis from the action
`decide` returned a `Disposition` — WARN / QUARANTINE_REVIEW / FAIL_SECURE —
which names an ACTION. But BRIEF design principle 4 says the library reports
and the pipeline decides, and disposition.py admitted the gap in its own
docstring: "It imposes no blocking of its own." So we returned an action we
cannot enforce, having discarded the judgement that produced it. A consumer
wanting different behaviour had to reinterpret the action itself — which is
why a consumer ends up pinning our GRADING: the action was all they got.
`Risk` (NONE/LOW/ELEVATED/SEVERE) now carries that judgement, and
`Policy.action_map` lets a caller map it to their own action. Both overlays
move the assessment rather than the action, so a custom map cannot silently
drop the compound escalation or the quarantine floor. `guard`'s fail-closed
path pins both axes and deliberately bypasses the map: downgrading SEVERE
means "I accept this class of finding", never "I accept a crashed scanner".
`DispositionResult.assessment` is required with no default. `Risk.NONE` is the
natural-looking default and the wrong one — a site that forgot the field would
report clean, and the axis would fail open.
MEASURED ADDITIVE, not assumed:
- 703 -> 715 tests, no existing test changed
- coverage matrix 128/128 recall, 6/6 documented gaps still hold
- the PRESET_USER_UPLOAD grading table locked in 0.3.1 re-measured row by
row: ordinary link/image/autolink/refdef -> warn on BOTH doors, unchanged
Both locked consumer promises in docs/PLAN-v1.md were checked against that
measurement and neither fires: the grading is untouched (linkedin-studio), and
the relative-target asymmetry is untouched (llm-ingestion-okf).
Scope held to disposition, per PLAN-v1.md:380. Version stays 0.4.0; the 0.5.0
bump lands in its release commit with all five version surfaces at once —
that is the fix for the defect where the v0.4.0 tag carried a 0.3.4 README.
Records limitation 32: `Severity` still carries disposition intent on the
DETECTION side, which this change does not address and cannot without moving
the grading.
This commit is contained in:
parent
3c56d50e05
commit
de097110d2
7 changed files with 329 additions and 38 deletions
51
CHANGELOG.md
51
CHANGELOG.md
|
|
@ -7,7 +7,56 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
|
||||
## [Unreleased]
|
||||
|
||||
Nothing yet.
|
||||
### Added — the axis separation: assessment (`Risk`) vs action (`Disposition`)
|
||||
|
||||
> **Additive, and measured to be so.** Every disposition 0.4.0 rendered is
|
||||
> rendered identically: the full suite went 703 → 715 with no test changed, the
|
||||
> coverage matrix holds at 128/128 recall with 6/6 documented gaps, and the
|
||||
> `PRESET_USER_UPLOAD` grading table locked in 0.3.1 was re-measured row by row
|
||||
> and is unchanged. A caller that never reads the new field sees no difference.
|
||||
|
||||
`decide` and `guard` returned a `Disposition` — `WARN` / `QUARANTINE_REVIEW` /
|
||||
`FAIL_SECURE` — which names an **action**. But BRIEF design principle 4 says the
|
||||
library reports and the *pipeline* decides, and `disposition.py` admitted the
|
||||
gap in its own docstring: *"It imposes no blocking of its own."* So the library
|
||||
returned an action it cannot enforce, while discarding the judgement that
|
||||
produced it. A consumer wanting different behaviour had to reinterpret the
|
||||
action itself, which is why a consumer ends up pinning our *grading* — the
|
||||
action was all they got.
|
||||
|
||||
- **`Risk`** — the new assessment axis: `NONE` / `LOW` / `ELEVATED` / `SEVERE`.
|
||||
It answers *how dangerous is this artifact given its source context*, and is
|
||||
trust-aware exactly as BRIEF §4.7 describes the domain: the same finding
|
||||
genuinely is a different judgement in authored prose than in a code fence.
|
||||
- **`DispositionResult.assessment`** — carries that judgement alongside the
|
||||
action. The field is **required, with no default**: `Risk.NONE` would be the
|
||||
natural-looking default and is the wrong one, since a construction site that
|
||||
forgot it would report *clean* and the axis would fail open.
|
||||
- **`Policy.action_map`** — an optional `Risk -> Disposition` override, so
|
||||
"hold for review where you would block" is a policy statement rather than a
|
||||
reason to pin our grading. Defaults to `None`, which means
|
||||
`DEFAULT_ACTION_MAP` and keeps an untouched `Policy` hashable as before. A
|
||||
partial map falls back per-level instead of raising.
|
||||
- Both overlays — compound escalation and the quarantine floor — now move the
|
||||
**assessment**, so a custom action map cannot silently drop them.
|
||||
- The fail-closed path in `guard` pins both axes to their most severe value and
|
||||
deliberately does **not** route through the action map: a policy that
|
||||
downgrades `SEVERE` means "I accept this class of finding", never "I accept a
|
||||
scanner that crashed on crafted input" (§4.6).
|
||||
|
||||
`NONE` and `LOW` both map to `WARN`, which is the point rather than an
|
||||
oversight: a clean document and one carrying only low-severity findings were a
|
||||
single indistinguishable value through 0.4.0.
|
||||
|
||||
### Known limitation recorded (32, was 31)
|
||||
|
||||
`Severity` still carries disposition intent on the *detection* side — the
|
||||
separation above is caller-side only. Two places say so outright:
|
||||
`ACTIVE_CONTENT_ORDINARY_SEVERITY = LOW` exists because grading an ordinary
|
||||
external image `HIGH` fail-secured ordinary uploads, and the quarantine floor
|
||||
was raised to `MEDIUM+` to repair the same regression from the other end.
|
||||
Closing it changes the grading and so fires a consumer-notification promise;
|
||||
deferred deliberately. See `docs/LIMITATIONS.md`.
|
||||
|
||||
|
||||
## [0.4.0] — 2026-08-10
|
||||
|
|
|
|||
|
|
@ -236,7 +236,7 @@ a green scan means safe content. The highest-impact items:
|
|||
egress, semantic poisoning, trusted-prose lone-HIGH, lexicon dedup (`count=1`),
|
||||
pure beaconing, and short opaque URL segments.
|
||||
|
||||
**Full list — 31 items, each with the mechanism, plus the out-of-scope boundary:**
|
||||
**Full list — 32 items, each with the mechanism, plus the out-of-scope boundary:**
|
||||
[`docs/LIMITATIONS.md`](docs/LIMITATIONS.md). Several carry field measurements from
|
||||
consumer corpora, including the false positives the URL-shape rule actually produces.
|
||||
|
||||
|
|
|
|||
|
|
@ -334,6 +334,25 @@ fails the test, forcing this doc to be updated:
|
|||
5. **Pure beaconing** — a bare-path remote image on a hostile host is LOW (above).
|
||||
6. **Short opaque URL segment (<24 chars)** — below what entropy can resolve (above).
|
||||
|
||||
- **`Severity` still carries disposition intent on the detection side.** The
|
||||
0.5.0 axis separation split the *assessment* (`Risk`) from the *action*
|
||||
(`Disposition`), but only on the caller side of the boundary. Inside the
|
||||
detectors, a finding's `Severity` is still calibrated partly for the
|
||||
disposition it will produce rather than purely for what was observed, and two
|
||||
places in the tree say so outright: `calibration.py`'s
|
||||
`ACTIVE_CONTENT_ORDINARY_SEVERITY = LOW` exists because grading an ordinary
|
||||
external image `HIGH` fail-secured ordinary uploads (measured on v0.3.0), and
|
||||
`disposition.py`'s quarantine floor was raised from *any finding* to *MEDIUM+*
|
||||
to repair the same regression from the other end. Both fixes are correct for
|
||||
the dispositions they produce; the cost is that an ordinary external image is
|
||||
recorded as low-severity rather than as *a real outward-fetch capability that
|
||||
is not evidence of an attack* — so no policy, however strict, can act on that
|
||||
capability, because the detector already decided it did not matter. Closing
|
||||
this means giving detectors a channel that says what was seen separately from
|
||||
how bad it is, which changes the grading and therefore fires the
|
||||
consumer-notification promise in `docs/PLAN-v1.md`. Deferred deliberately, not
|
||||
overlooked.
|
||||
|
||||
## Out-of-scope (documented boundary)
|
||||
|
||||
Embedding/vector-layer defenses (OWASP LLM08, downstream of persist); multimodal
|
||||
|
|
|
|||
|
|
@ -40,8 +40,10 @@ from .disposition import (
|
|||
Policy,
|
||||
Trust,
|
||||
Provenance,
|
||||
Risk,
|
||||
Disposition,
|
||||
DispositionResult,
|
||||
DEFAULT_ACTION_MAP,
|
||||
PRESET_TRUSTED_SOURCE,
|
||||
PRESET_USER_UPLOAD,
|
||||
)
|
||||
|
|
@ -135,9 +137,9 @@ __all__ = [
|
|||
"neutralize", "NeutralizeResult",
|
||||
# output-side
|
||||
"scan_output", "scan_secret_egress", "scan_active_content",
|
||||
# disposition
|
||||
# disposition — `Risk` is the assessment axis, `Disposition` the action
|
||||
"decide", "guard", "Policy", "Trust", "Provenance",
|
||||
"Disposition", "DispositionResult",
|
||||
"Risk", "Disposition", "DispositionResult", "DEFAULT_ACTION_MAP",
|
||||
"PRESET_TRUSTED_SOURCE", "PRESET_USER_UPLOAD",
|
||||
# contract asserters
|
||||
"assert_tool_less", "assert_credential_allowlist",
|
||||
|
|
|
|||
|
|
@ -90,6 +90,34 @@ DISPOSITION_RANK = {
|
|||
"fail_secure": 2,
|
||||
}
|
||||
|
||||
# --- assessment: the risk axis (0.5.0 axis separation) ----------------------
|
||||
# Rank of each risk level, same value-keyed convention as DISPOSITION_RANK.
|
||||
# ``Risk`` answers *how dangerous is this artifact given its source context*;
|
||||
# ``Disposition`` answers *what should the pipeline do*. They were one enum
|
||||
# through 0.4.0, which meant a consumer wanting a different action had to
|
||||
# re-derive it from the action itself — the assessment was already discarded.
|
||||
RISK_RANK = {
|
||||
"none": 0,
|
||||
"low": 1,
|
||||
"elevated": 2,
|
||||
"severe": 3,
|
||||
}
|
||||
|
||||
# The default risk -> disposition mapping, keyed by both enum *value* strings.
|
||||
# This map is what keeps the separation additive: it reproduces every
|
||||
# disposition 0.4.0 rendered, so a caller that ignores the new axis sees no
|
||||
# change at all. A ``Policy`` may override it; ``None`` means "use this".
|
||||
#
|
||||
# NONE and LOW both map to ``warn`` deliberately — that collapse is precisely
|
||||
# the information 0.4.0 could not express, since a clean document and one
|
||||
# carrying only low-severity findings were the same single value.
|
||||
DEFAULT_ACTION_MAP = {
|
||||
"none": "warn",
|
||||
"low": "warn",
|
||||
"elevated": "quarantine_review",
|
||||
"severe": "fail_secure",
|
||||
}
|
||||
|
||||
# --- active_content: per-construct severities -------------------------------
|
||||
# Zero-click auto-fetch / auto-execute constructs are HIGH; click-required ones
|
||||
# are MEDIUM. Mirrors ``neutralize``'s defang classes. These are the severities
|
||||
|
|
|
|||
|
|
@ -23,12 +23,35 @@ from __future__ import annotations
|
|||
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from typing import Callable, Optional
|
||||
from typing import Callable, Mapping, Optional
|
||||
|
||||
from .calibration import DISPOSITION_RANK
|
||||
from .calibration import DEFAULT_ACTION_MAP as _DEFAULT_ACTION_MAP_VALUES
|
||||
from .calibration import DISPOSITION_RANK, RISK_RANK
|
||||
from .report import Report, Severity, severity_rank
|
||||
|
||||
|
||||
class Risk(str, Enum):
|
||||
"""How dangerous the artifact is *given its source context* — the assessment.
|
||||
|
||||
The other half of the 0.5.0 axis separation. :class:`Disposition` names an
|
||||
**action**; ``Risk`` names the **judgement that action was derived from**.
|
||||
Through 0.4.0 only the action was returned, which had two costs: a consumer
|
||||
who wanted different behaviour had to reinterpret an action whose reasoning
|
||||
was already gone, and ``NONE`` vs ``LOW`` — a clean document versus one
|
||||
carrying only low-severity findings — were indistinguishable, since both
|
||||
render as ``WARN``.
|
||||
|
||||
Risk is **trust-aware**, exactly as BRIEF §4.7 describes the domain: the
|
||||
same finding genuinely *is* a different judgement in authored prose than in
|
||||
a code fence, not merely a different action taken on one shared judgement.
|
||||
"""
|
||||
|
||||
NONE = "none" # no findings at all
|
||||
LOW = "low" # findings present, none dispositive in this context
|
||||
ELEVATED = "elevated" # suspicious enough to hold for a human
|
||||
SEVERE = "severe" # treat as a real payload
|
||||
|
||||
|
||||
class Disposition(str, Enum):
|
||||
"""The gate decision, ordered by :data:`_DISPOSITION_RANK`."""
|
||||
|
||||
|
|
@ -54,19 +77,41 @@ class Provenance(str, Enum):
|
|||
|
||||
@dataclass(frozen=True)
|
||||
class Policy:
|
||||
"""A named source-trust policy. See the ``PRESET_*`` constants."""
|
||||
"""A named source-trust policy. See the ``PRESET_*`` constants.
|
||||
|
||||
``action_map`` overrides how an assessed :class:`Risk` becomes a
|
||||
:class:`Disposition`. ``None`` — the default — means
|
||||
:data:`DEFAULT_ACTION_MAP`, which reproduces every disposition 0.4.0
|
||||
rendered. It is deliberately ``None`` rather than the map itself so an
|
||||
untouched ``Policy`` stays hashable exactly as before.
|
||||
|
||||
This is the seam a consumer needs: wanting *hold for review* where we
|
||||
render *fail secure* is now a policy statement, not a reason to pin our
|
||||
grading.
|
||||
"""
|
||||
|
||||
trust: Trust
|
||||
quarantine_default: bool = False # any finding -> at least QUARANTINE_REVIEW
|
||||
action_map: Optional[Mapping[Risk, Disposition]] = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DispositionResult:
|
||||
"""The decision plus an auditable trail of which rules fired."""
|
||||
"""The decision plus an auditable trail of which rules fired.
|
||||
|
||||
``assessment`` is the risk the rules actually established; ``disposition``
|
||||
is that risk mapped through the policy's action map. Consumers that gate on
|
||||
the assessment are insulated from a later recalibration of the mapping.
|
||||
"""
|
||||
|
||||
disposition: Disposition
|
||||
reasons: tuple[str, ...]
|
||||
max_severity: Optional[Severity]
|
||||
assessment: Risk
|
||||
# Deliberately *required*, with no default. ``Risk.NONE`` would be the
|
||||
# natural-looking default and is precisely the wrong one: a construction
|
||||
# site that forgot the field would report a clean assessment, so the axis
|
||||
# would fail open. Fail-loud beats fail-silent for a gate (BRIEF §4.7).
|
||||
|
||||
|
||||
# Invisible carriers have no legitimate place in a reference file: they block in
|
||||
|
|
@ -83,20 +128,52 @@ _CARRIER_LABELS = frozenset({
|
|||
"lexicon:unicode-tags-present",
|
||||
})
|
||||
|
||||
# Enum-keyed rank rebuilt from calibration's value-keyed source of truth
|
||||
# (calibration is a leaf module and cannot import the Disposition enum without a
|
||||
# cycle). Higher = more severe.
|
||||
# Enum-keyed ranks rebuilt from calibration's value-keyed source of truth
|
||||
# (calibration is a leaf module and cannot import these enums without a cycle).
|
||||
# Higher = more severe.
|
||||
_DISPOSITION_RANK = {d: DISPOSITION_RANK[d.value] for d in Disposition}
|
||||
_RISK_RANK = {r: RISK_RANK[r.value] for r in Risk}
|
||||
|
||||
DEFAULT_ACTION_MAP: Mapping[Risk, Disposition] = {
|
||||
Risk(risk_value): Disposition(disposition_value)
|
||||
for risk_value, disposition_value in _DEFAULT_ACTION_MAP_VALUES.items()
|
||||
}
|
||||
"""The default :class:`Risk` -> :class:`Disposition` mapping.
|
||||
|
||||
Reproduces every disposition 0.4.0 rendered, so the axis separation is additive:
|
||||
a caller that never reads ``assessment`` sees no behavioural change.
|
||||
"""
|
||||
|
||||
|
||||
def _more_severe(a: Disposition, b: Disposition) -> Disposition:
|
||||
return a if _DISPOSITION_RANK[a] >= _DISPOSITION_RANK[b] else b
|
||||
def _more_severe(a: Risk, b: Risk) -> Risk:
|
||||
return a if _RISK_RANK[a] >= _RISK_RANK[b] else b
|
||||
|
||||
|
||||
def _escalate(disposition: Disposition) -> Disposition:
|
||||
if disposition is Disposition.WARN:
|
||||
return Disposition.QUARANTINE_REVIEW
|
||||
return Disposition.FAIL_SECURE
|
||||
def _escalate(risk: Risk) -> Risk:
|
||||
"""Escalate one tier on the assessment axis.
|
||||
|
||||
``NONE`` is unreachable here — the only caller is the compound overlay,
|
||||
which needs two MEDIUM+ findings and so implies at least ``LOW`` — but it
|
||||
escalates rather than being a no-op, so the function is total.
|
||||
"""
|
||||
if risk is Risk.SEVERE:
|
||||
return Risk.SEVERE
|
||||
return Risk(next(
|
||||
r for r in Risk if _RISK_RANK[r] == _RISK_RANK[risk] + 1
|
||||
))
|
||||
|
||||
|
||||
def _action(risk: Risk, policy: Policy) -> Disposition:
|
||||
"""Map an assessed ``risk`` to an action under ``policy``.
|
||||
|
||||
An action map that omits a risk level falls back to the default rather than
|
||||
raising: a partial override is a likely way to use this, and a ``KeyError``
|
||||
from inside the gate would be turned into a fail-closed by :func:`guard`
|
||||
anyway — silently, and with a useless reason.
|
||||
"""
|
||||
if policy.action_map is not None and risk in policy.action_map:
|
||||
return policy.action_map[risk]
|
||||
return DEFAULT_ACTION_MAP[risk]
|
||||
|
||||
|
||||
def _carrier_label(report: Report) -> Optional[str]:
|
||||
|
|
@ -126,21 +203,24 @@ def decide(
|
|||
reasons: list[str] = []
|
||||
max_sev = report.max_severity()
|
||||
|
||||
def result(risk: Risk) -> DispositionResult:
|
||||
return DispositionResult(_action(risk, policy), tuple(reasons), max_sev, risk)
|
||||
|
||||
# Overlay A — compound forced-fallback (§4.6): a scan hit plus a failed
|
||||
# transform is a probable forced-fallback attack. Overrides everything.
|
||||
if transform_failed and report.found:
|
||||
reasons.append("compound-forced-fallback: transform failed with active findings")
|
||||
return DispositionResult(Disposition.FAIL_SECURE, tuple(reasons), max_sev)
|
||||
return result(Risk.SEVERE)
|
||||
|
||||
# Any-tier exceptions (§4.7): invisible carriers and CRITICAL findings block
|
||||
# regardless of trust or provenance.
|
||||
carrier = _carrier_label(report)
|
||||
if carrier is not None:
|
||||
reasons.append(f"any-tier: invisible carrier ({carrier})")
|
||||
return DispositionResult(Disposition.FAIL_SECURE, tuple(reasons), max_sev)
|
||||
return result(Risk.SEVERE)
|
||||
if max_sev is Severity.CRITICAL:
|
||||
reasons.append("any-tier: CRITICAL finding")
|
||||
return DispositionResult(Disposition.FAIL_SECURE, tuple(reasons), max_sev)
|
||||
return result(Risk.SEVERE)
|
||||
|
||||
# Effective low-trust: an untrusted source, or a low-trust region within an
|
||||
# otherwise-trusted document (a code fence or a localized string).
|
||||
|
|
@ -149,38 +229,47 @@ def decide(
|
|||
Provenance.LOCALIZED,
|
||||
)
|
||||
|
||||
disposition = _base_disposition(report, max_sev, low_trust, policy, reasons)
|
||||
risk = _base_risk(report, max_sev, low_trust, policy, reasons)
|
||||
|
||||
# Overlay B — compound escalation (§4.6): several weaker signals escalate one
|
||||
# tier even when each alone would only WARN.
|
||||
# tier even when each alone would only WARN. Escalating the *assessment*
|
||||
# rather than the action is what keeps the overlay from being silently lost
|
||||
# under a custom action map.
|
||||
if _is_compound(report):
|
||||
escalated = _escalate(disposition)
|
||||
if escalated is not disposition:
|
||||
escalated = _escalate(risk)
|
||||
if escalated is not risk:
|
||||
reasons.append("compound: >=2 findings at MEDIUM+ -> escalated one tier")
|
||||
disposition = escalated
|
||||
risk = escalated
|
||||
|
||||
return DispositionResult(disposition, tuple(reasons), max_sev)
|
||||
return result(risk)
|
||||
|
||||
|
||||
def _base_disposition(
|
||||
def _base_risk(
|
||||
report: Report,
|
||||
max_sev: Optional[Severity],
|
||||
low_trust: bool,
|
||||
policy: Policy,
|
||||
reasons: list[str],
|
||||
) -> Disposition:
|
||||
) -> Risk:
|
||||
"""Assess ``report`` before the overlays, on the risk axis.
|
||||
|
||||
The reason strings still name the *disposition* each branch yields, because
|
||||
they are an audit trail consumers already read; the axis separation must not
|
||||
silently reword it. They are rendered through :func:`_action` so a policy
|
||||
that remaps an action gets a trail that matches what it actually did.
|
||||
"""
|
||||
tier = "low" if low_trust else "high"
|
||||
if max_sev is None:
|
||||
disposition = Disposition.WARN
|
||||
risk = Risk.NONE
|
||||
reasons.append("clean: no findings")
|
||||
elif max_sev is Severity.HIGH:
|
||||
disposition = Disposition.FAIL_SECURE if low_trust else Disposition.WARN
|
||||
reasons.append(f"HIGH under {tier}-trust -> {disposition.value}")
|
||||
risk = Risk.SEVERE if low_trust else Risk.LOW
|
||||
reasons.append(f"HIGH under {tier}-trust -> {_action(risk, policy).value}")
|
||||
elif max_sev is Severity.MEDIUM:
|
||||
disposition = Disposition.QUARANTINE_REVIEW if low_trust else Disposition.WARN
|
||||
reasons.append(f"MEDIUM under {tier}-trust -> {disposition.value}")
|
||||
risk = Risk.ELEVATED if low_trust else Risk.LOW
|
||||
reasons.append(f"MEDIUM under {tier}-trust -> {_action(risk, policy).value}")
|
||||
else: # LOW or INFO
|
||||
disposition = Disposition.WARN
|
||||
risk = Risk.LOW
|
||||
reasons.append(f"{max_sev.value} -> WARN")
|
||||
|
||||
# quarantine_default floor: a finding at MEDIUM+ is held for review.
|
||||
|
|
@ -194,12 +283,12 @@ def _base_disposition(
|
|||
if policy.quarantine_default and max_sev is not None and (
|
||||
severity_rank(max_sev) >= severity_rank(Severity.MEDIUM)
|
||||
):
|
||||
floored = _more_severe(disposition, Disposition.QUARANTINE_REVIEW)
|
||||
if floored is not disposition:
|
||||
floored = _more_severe(risk, Risk.ELEVATED)
|
||||
if floored is not risk:
|
||||
reasons.append("quarantine-floor: MEDIUM+ finding -> QUARANTINE_REVIEW")
|
||||
disposition = floored
|
||||
risk = floored
|
||||
|
||||
return disposition
|
||||
return risk
|
||||
|
||||
|
||||
def guard(
|
||||
|
|
@ -221,10 +310,16 @@ def guard(
|
|||
report = scan_fn()
|
||||
return decide(report, policy, provenance=provenance, transform_failed=transform_failed)
|
||||
except Exception as exc: # noqa: BLE001 — fail closed on ANY scan/dispose error
|
||||
# Both axes are pinned to their most severe value, and deliberately NOT
|
||||
# routed through the policy's action map: an un-scannable artifact is
|
||||
# not a risk judgement a caller gets to remap. A policy that downgrades
|
||||
# SEVERE means "I accept this class of finding", never "I accept a
|
||||
# scanner that crashed on crafted input" (BRIEF §4.6, fail closed).
|
||||
return DispositionResult(
|
||||
Disposition.FAIL_SECURE,
|
||||
(f"fail-closed: scan/dispose error: {type(exc).__name__}",),
|
||||
None,
|
||||
Risk.SEVERE,
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -5,10 +5,12 @@ import pytest
|
|||
|
||||
from llm_ingestion_guard.report import Finding, Report, Severity, Source
|
||||
from llm_ingestion_guard.disposition import (
|
||||
DEFAULT_ACTION_MAP,
|
||||
Disposition,
|
||||
DispositionResult,
|
||||
Policy,
|
||||
Provenance,
|
||||
Risk,
|
||||
Trust,
|
||||
decide,
|
||||
guard,
|
||||
|
|
@ -258,3 +260,99 @@ def test_floor_and_escalation_compose_to_fail_secure():
|
|||
_finding(severity=Severity.MEDIUM, label="entropy:base64", detector="entropy"),
|
||||
)
|
||||
assert decide(report, PRESET_USER_UPLOAD).disposition is Disposition.FAIL_SECURE
|
||||
|
||||
|
||||
# --- 0.5.0 axis separation: assessment (how dangerous) vs disposition (what to
|
||||
# --- do). PLAN-v1.md:294/:380 — change DISPOSITION, never the grading.
|
||||
#
|
||||
# `Disposition` is three *actions*, but BRIEF principle 4 says the library
|
||||
# reports and the *pipeline* decides. `decide` therefore returned an action it
|
||||
# cannot enforce, and a consumer wanting a different action had to re-derive it
|
||||
# from that action — the assessment which produced it was already gone. That is
|
||||
# why a consumer ends up pinning the *grading*: the action is all they get. The
|
||||
# assessment axis hands them the input instead of the verdict.
|
||||
|
||||
|
||||
def test_result_carries_an_assessment_distinct_from_the_disposition():
|
||||
# The payoff: a clean report and a LOW-severity report are BOTH `WARN`
|
||||
# today, and indistinguishable without re-reading the report. The assessment
|
||||
# axis separates them while leaving the action identical.
|
||||
clean = decide(_report(), UNTRUSTED)
|
||||
low = decide(_report(_finding(severity=Severity.LOW, label="active:markdown-link")),
|
||||
UNTRUSTED)
|
||||
|
||||
assert clean.disposition is Disposition.WARN
|
||||
assert low.disposition is Disposition.WARN # action: identical
|
||||
assert clean.assessment is Risk.NONE
|
||||
assert low.assessment is Risk.LOW # assessment: distinct
|
||||
|
||||
|
||||
def test_default_action_map_reproduces_todays_outcomes():
|
||||
# The default mapping is a no-op by construction: this is the contract that
|
||||
# keeps `PRESET_USER_UPLOAD` grading untouched (locked promise 1).
|
||||
assert DEFAULT_ACTION_MAP == {
|
||||
Risk.NONE: Disposition.WARN,
|
||||
Risk.LOW: Disposition.WARN,
|
||||
Risk.ELEVATED: Disposition.QUARANTINE_REVIEW,
|
||||
Risk.SEVERE: Disposition.FAIL_SECURE,
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("severity,policy,provenance,expected_risk", [
|
||||
(Severity.CRITICAL, TRUSTED, Provenance.PROSE, Risk.SEVERE),
|
||||
(Severity.HIGH, TRUSTED, Provenance.PROSE, Risk.LOW),
|
||||
(Severity.HIGH, TRUSTED, Provenance.CODE_FENCE, Risk.SEVERE),
|
||||
(Severity.HIGH, UNTRUSTED, Provenance.PROSE, Risk.SEVERE),
|
||||
(Severity.MEDIUM, TRUSTED, Provenance.PROSE, Risk.LOW),
|
||||
(Severity.MEDIUM, UNTRUSTED, Provenance.PROSE, Risk.ELEVATED),
|
||||
(Severity.LOW, UNTRUSTED, Provenance.PROSE, Risk.LOW),
|
||||
])
|
||||
def test_assessment_tracks_danger_given_trust(severity, policy, provenance, expected_risk):
|
||||
# The assessment is trust-aware, exactly as BRIEF §4.7 describes: the *same*
|
||||
# hit is a different assessment in prose vs a code fence — not merely a
|
||||
# different action on one shared assessment.
|
||||
result = decide(_report(_finding(severity=severity)), policy, provenance=provenance)
|
||||
assert result.assessment is expected_risk
|
||||
assert result.disposition is DEFAULT_ACTION_MAP[expected_risk]
|
||||
|
||||
|
||||
def test_custom_action_map_changes_the_action_not_the_assessment():
|
||||
# THE point of the separation. A consumer that wants to hold for review
|
||||
# rather than block says so in the policy, and the assessment it was derived
|
||||
# from is unchanged — so they never have to pin our grading to get their
|
||||
# behaviour.
|
||||
report = _report(_finding(severity=Severity.HIGH))
|
||||
strict = Policy(trust=Trust.UNTRUSTED)
|
||||
lenient = Policy(trust=Trust.UNTRUSTED, action_map={
|
||||
**DEFAULT_ACTION_MAP,
|
||||
Risk.SEVERE: Disposition.QUARANTINE_REVIEW,
|
||||
})
|
||||
|
||||
assert decide(report, strict).disposition is Disposition.FAIL_SECURE
|
||||
assert decide(report, lenient).disposition is Disposition.QUARANTINE_REVIEW
|
||||
assert decide(report, strict).assessment is Risk.SEVERE
|
||||
assert decide(report, lenient).assessment is Risk.SEVERE
|
||||
|
||||
|
||||
def test_overlays_escalate_the_assessment_not_only_the_action():
|
||||
# Compound escalation and the quarantine floor are assessment-level moves;
|
||||
# if they only moved the action, a custom action_map would silently drop
|
||||
# them. Two MEDIUM findings untrusted: ELEVATED escalated to SEVERE.
|
||||
report = _report(
|
||||
_finding(severity=Severity.MEDIUM, label="lexicon:config"),
|
||||
_finding(severity=Severity.MEDIUM, label="entropy:base64", detector="entropy"),
|
||||
)
|
||||
result = decide(report, PRESET_USER_UPLOAD)
|
||||
assert result.assessment is Risk.SEVERE
|
||||
assert result.disposition is Disposition.FAIL_SECURE
|
||||
|
||||
|
||||
def test_guard_fails_closed_with_a_severe_assessment():
|
||||
# The fail-closed path must not leave the assessment unset, or a consumer
|
||||
# mapping on the assessment alone would read a scanner crash as clean.
|
||||
def boom() -> Report:
|
||||
raise RuntimeError("detector exploded")
|
||||
|
||||
result = guard(boom, PRESET_USER_UPLOAD)
|
||||
assert result.disposition is Disposition.FAIL_SECURE
|
||||
assert result.assessment is Risk.SEVERE
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue