open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Releases 4 Activity

docs(limits): seven shipped behaviours were only in the code, and one of them was written down wrong

Browse source 
TRACK 2. All seven were tested and released; none had a line in
docs/LIMITATIONS.md. Written where each belongs topically, not appended.
36 -> 43 items; README count moved with it. No code change, no release.

- The carrier split LOOSENS a lone anchor, not only tightens. Measured as
  shipped: <a href="https://ext.example/p"> alone is active:raw-html-link at
  MEDIUM -> quarantine_review on PRESET_USER_UPLOAD (warn on trusted); through
  0.6.1 it was HIGH -> fail_secure. An <img src> to the same host is untouched.
  Only the tightening direction had been documented.
- The no-URL narrowing's name set is a judgement about affordance no test can
  derive. A name wrongly placed in _URL_AFFORDANCE_TAGS goes silently inert,
  and no corpus can catch it: what it produces is an absence.
- hybrid-xss:script-tag and active:raw-html do not match the same strings.
  <script src="a<b"> raises active:raw-html and NO XSS label -- the lexicon's
  [^><] is the 0.3.3 ReDoS fix, HTML_TAG_RE consumes quoted runs atomically.
  Disposition never moves; the label does, so that id is not a script census.
- A green ReDoS row is evidence only if seen red, and three rows never were.
  Corrected while writing: the third row is
  test_gate_is_bounded_on_the_long_attribute_arm, killed by 0.7.0's own
  narrowing (separation 1.0x, 0.028s, no findings) -- not active-url-attr-value,
  which is a payload-design rule, not a dead row.
- Every suite bound runs on process CPU time (tests/redos_clock.py); the
  residual is that a BLOCKING hang would hang the suite. docs/redos-sweep.py
  still times on time.monotonic(), so the published 1.5ms floor at N=8000 and
  the "~23s at the cap" figure come from a different instrument than the bounds
  they justify. Not re-derived; stated as not directly comparable.
- Under DEFAULT_ACTION_MAP the assessment axis adds exactly one judgement over
  the disposition (NONE vs LOW). Consequence for this document: the published
  FP rates are a risk statement only while NONE+LOW are the WARN pre-image.
- The ZWJ context test is one predicate on two surfaces, and the symbol is
  private -- output imports sanitize's _is_joiner_in_emoji_sequence, so a
  consumer importing it pins a name 1.x does not promise.

Verified: 802 passed, 129/129 classes, 6/6 gaps hold, grep -c '^- \*\*' = 43,
and the doc scanned for stray format characters (one U+200D remains, the
pre-existing composed emoji -- an accidental U+200C of my own was removed).
This commit is contained in:
Kjell Tore Guttormsen 2026-08-17 12:15:00 +02:00
commit 4472f209a4
2 changed files with 135 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

2
README.md
View file

@ -261,7 +261,7 @@ a green scan means safe content. The highest-impact items:
two of the three corpora are living, so the cells are not rewritten in place.
Method and before/after: [`docs/rawhtml-census.py`](docs/rawhtml-census.py).
**Full list — 36 items, each with the mechanism, plus the out-of-scope boundary:**
**Full list — 43 items, each with the mechanism, plus the out-of-scope boundary:**
[`docs/LIMITATIONS.md`](docs/LIMITATIONS.md). Several carry field measurements from
consumer corpora, including the false positives the URL-shape rule actually produces.

134
docs/LIMITATIONS.md
View file

@ -304,6 +304,21 @@ items; this is the full list, each with the mechanism.
`test_split_tightens_the_trusted_tier_when_both_carriers_are_present` constructs
and pins. Read the zero as "not observed in any of the three populations, each
counted against its own denominator", never as "cannot happen".
- **The split also LOOSENS the upload door for a lone anchor — the direction it was
built for, and the one with a residual worth naming.** Measured as shipped:
`<a href="https://ext.example/p">t</a>` on its own emits one
`active:raw-html-link` at MEDIUM and disposes `quarantine_review` under
`PRESET_USER_UPLOAD` (`warn` under `PRESET_TRUSTED_SOURCE`); through 0.6.1 the
same document graded HIGH and `fail_secure`d. An `<img src>` to the same host is
untouched — `active:raw-html`, HIGH, `fail_secure`. So an anchor pointing at an
attacker-controlled host, arriving on an untrusted upload, is now a human decision
rather than a halt. The trade is deliberate and it removes an asymmetry that came
from syntax rather than affordance: following an anchor needs a click, exactly like
the markdown inline link that has graded MEDIUM since 0.3.1, so the same URL no
longer grades two different ways depending on which syntax carries it. It is
recorded here so 0.7.0's "frees N documents" is not read as free — what was freed
is the click-required class, and MEDIUM is a real grade drop on the door where
every finding is trust-escalated.
- **Raw-HTML findings no longer count end tags, and that moved a published field.**
Through 0.6.1 `</a>` was active by name on its own, so `count` ran roughly 1.6×
the opening-tag total (measured on one corpus) and a start/end pair counted 2.
@ -312,6 +327,21 @@ items; this is the full list, each with the mechanism.
every document carrying `</a>`, on a field whose meaning did not change. The
finding count is unaffected: the class still collapses to one finding per class
per document, and `count` was never a document count.
- **Which tags the no-URL narrowing may render inert is a judgement about affordance,
and no test in this repo can derive it.** `_URL_AFFORDANCE_TAGS` holds the nine
names whose entire active affordance *is* the URL they name — `a`, `area`, `img`,
`video`, `audio`, `source`, `track`, `frame`, `frameset` — so carrying no URL
attribute they name no target and grade inert. Every other name in the active set
stays active with no attributes at all, because it does something a URL cannot
describe: `<script>` executes its body, `<style>` restyles, `<form>` submits. That
boundary is asserted, not measured. A name placed in the set whose affordance does
*not* reduce to its URL would go silently invisible, and no corpus can catch it,
because what it produces is an absence — the census counts findings, and a tag that
stopped firing contributes nothing to count. The fail-secure choice one branch
further in holds the other way and is worth reading beside it: a URL attribute whose
*value* this module cannot resolve keeps the tag active
(`active_content.py:320-324`), a branch the three corpora exercise **0** times.
That zero is empirical, so the predicate is written not to depend on it.
- **"Clean" means *graded, no finding raised* — never *cleaned bytes* — and at one
measured consumer's door, `warn` is the floor a document must clear to be
persisted rather than rejected.** The word is the library's own: a WARN
@ -461,6 +491,23 @@ items; this is the full list, each with the mechanism.
HIGH under a low-trust preset is `fail_secure`. Report-only means the text is
never mutated — it does not mean the finding cannot block.
- **The two labels a `<script>` tag raises come from patterns that do not match the
same strings.** `hybrid-xss:script-tag` is `<script\b[^><]*>`; `active:raw-html`
reads the same tag through `HTML_TAG_RE`, which consumes quoted attribute runs
atomically and so tolerates both `<` and `>` inside a quoted value. The lexicon's
`<` exclusion is not a modelling choice — it is the 0.3.3 ReDoS fix, and widening
it back to `[^>]` restores a quadratic arm (the row below carries the numbers).
Measured through both scanners: `<script src="a<b">` raises `active:raw-html` and
**no** XSS label, while `<script data-t="a>b">` raises both, the lexicon's match
simply ending at the quoted `>`. The disposition never moves — the raw-HTML branch
grades `<script>` HIGH with no attributes at all, so every shape here still
`fail_secure`s under `PRESET_USER_UPLOAD` — so what the divergence costs is the
*label*: a consumer filtering findings on the XSS id sees a subset of the script
tags the gate actually caught, and must not read that id as the gate's script-tag
census. The residual is practically dead in prose (a `<` inside a script tag's
quoted attribute region is not an ordinary shape) and is recorded because the
asymmetry is invisible from either scanner alone.
- **A connection-string password longer than 256 chars is not matched.** The
password run in the `*-connstr` egress patterns is bounded by
`MAX_CONNSTR_VALUE`; unbounded, it sits in front of a mandatory `@` and makes
@ -495,6 +542,60 @@ items; this is the full list, each with the mechanism.
*closes* around a long body, and a run of plain characters carrying no anchor
at all.
- **A green ReDoS row is evidence only if it has been seen red, and three rows in
this suite had never been.** The class is not a bad bound but a payload that cannot
reach the defect, and it leaves the row passing under the vulnerable form too. The
sub-agent row is the clearest case: the seed's unbounded lazy run costs per *prefix
match*, not per character — each start position where `spawn an agent that ` matches
drives its own O(N) scan to end-of-string looking for a capability keyword the
payload never supplies, so K prefix matches cost K×O(N), and the bounded
`(?:\S+\s+){0,12}?` port caps each scan at 12 tokens for K×O(1). A payload that
matches the prefix **once** and then pads pays a single lazy run and is linear
however long the pad is — two earlier shapes did exactly that, and the row sat
measured-dead at 1.2× until the payload was rebuilt as
`"spawn an agent that " * 3000`. (The nesting an older comment blamed is a red
herring: the inner `.*?` sits in an optional group, never a repeated one.) Measured
through `scan_lexicon` with the seed form patched back in — exponent **1.92** against
the shipped **1.01**, and **4.091s vs 0.190s** at 12 000 words, the seed breaking the
2.0s bound outright. Two siblings were dead for different reasons.
`lexicon-script-tag` had to be given its own N=200 000: at the shared N=100 000 the
vulnerable `[^>]` form measured only ~1.2–1.4s — under the 2.0s assert, so the row
was green under both forms and proved nothing. And
`test_gate_is_bounded_on_the_long_attribute_arm` was killed by **this repo's own
narrowing**: 0.7.0 put `<a>` in `_URL_AFFORDANCE_TAGS`, so its `<a ` + 100k + `>`
payload became inert and returned *before* the body ever reached the arm the row
exists to guard — separation 1.0×, 0.028s and no findings, against 12.475s for the
same payload carried by `<script `. The carrier was moved to `<script `, which is
active by name with no attributes, so no future URL-shaped narrowing can hollow it
out the same way. The general rule the three share: a payload must **deny** the
literal the vulnerable run sits in front of — a unit that supplies it matches
immediately and never exercises the run. **Nothing but hand measurement finds this
class.** The row is green either way, so
the suite cannot report its own blind spot, and every bound in it should be read as
"verified red under the vulnerable form" only where a comment says it was.
- **Every ReDoS bound in the suite is measured on process CPU time; the sweep that
sets the published sensitivity floor is not.** `tests/redos_clock.py` is the one
clock all six test files import — `time.process_time()` — because a blowup is spent
cycles while a loaded machine steals wall clock without adding any. On
`time.monotonic()` two 0.7.0 rows failed at **2.24s / 3.66s** against a 2.0s bound
while two census processes held the CPU, and passed 3/3 on an idle machine: they had
been descheduled, not slowed. It lives in one module rather than five copies because
`test_output.py::test_the_redos_clock_ignores_time_this_process_did_not_spend` pins
one implementation, and four unpinned copies would be free to drift back to a wall
clock with nothing going red. **What the CPU clock gives up, stated: a scan that
BLOCKS forever burns no CPU, so it would hang the suite instead of failing it.**
That is acceptable only because every scanner it measures is pure regex over an
in-memory string, with no I/O and no locks — the last wall-clock holdout was retired
by auditing its path for anything that could block, not by assumption, and a wall
clock guarding a mode that cannot occur still charges the false-red premium
(measured there at 21.6s against a 10.0s bound, on a scan that spent 7.6s). **The
divergence to know about:** `docs/redos-sweep.py` still times on `time.monotonic()`
(`redos-sweep.py:139`), so the 1.5 ms floor at N=8000 and the "~23 s at the cap"
figure above are wall-clock numbers, produced by a different instrument than the
bounds they justify. They have not been re-derived on the CPU clock; treat them as
the noisier of the two measurements rather than as directly comparable.
- **Every surface now bounds its input, but not all of them the same way.**
`sanitize`, `fence` and `neutralize` raise `OversizeInputError` above
`MAX_INPUT_CHARS` (1 000 000) rather than returning a partially transformed
@ -557,6 +658,23 @@ fails the test, forcing this doc to be updated:
names the outward fetch whatever severity it carries — and must not infer
"nothing was seen" from a low `Severity`.
- **Under the default action map the assessment axis carries exactly one judgement
the disposition does not.** `DEFAULT_ACTION_MAP` sends `NONE` and `LOW` to `WARN`,
`ELEVATED` to `QUARANTINE_REVIEW` and `SEVERE` to `FAIL_SECURE` — the last two 1:1.
So for any document that carries a finding at all, `assessment` is a relabelling of
`disposition` and nothing more; the only thing it adds is *clean* versus *findings
present, none dispositive in this context*, which 0.4.0 rendered identically. That
collapse is the point (the map is what keeps the separation additive, so a caller
ignoring the new axis sees no change), and it is also the limitation: reading
`assessment` buys a consumer nothing until it supplies its own `action_map` or needs
the clean/low distinction. **The second consequence is on this document.** The
published false-positive rates are counts of documents *disposed non-WARN*, and they
are a statement about assessed risk only while `NONE` + `LOW` are exactly the WARN
pre-image. `tests/test_corpus.py::test_the_published_fp_metric_is_a_risk_statement`
pins that equivalence — but it pins it for `DEFAULT_ACTION_MAP`. A caller running its
own map makes "disposed non-WARN" a different claim from the one measured here, with
nothing in either repo failing to say so.
- **A ZWJ hidden between two emoji is exempt, and ZWNJ's own false-positive
class is untouched.** U+200D composes emoji (👩‍💻 is WOMAN + ZWJ + PERSONAL
COMPUTER), so testing it on codepoint membership alone flagged *and stripped*
@ -575,6 +693,22 @@ fails the test, forcing this doc to be updated:
not pictographic) and no corpus is available here to verify it against, so it
is parked as a known false-positive class rather than guessed at.
- **That context test is one predicate on two surfaces, and the symbol carrying it is
private.** `sanitize` owns `_is_joiner_in_emoji_sequence`; `output` imports it
(`output.py:74`) instead of restating it, because the same defect had to be fixed on
both surfaces and a split would let the input side stop flagging while the output
side kept hard-blocking — or the reverse, which is how a carrier reaches a persisted
artifact after passing the input gate. The agreement is pinned by
`tests/test_output.py::test_output_zwj_narrowing_matches_the_sanitize_side`, which
asserts `stripped == flagged` across six shapes — half-context on either side, a
leading and a trailing joiner, one genuine in-sequence joiner, and a word split.
Two things that pin does not give. The six shapes are hand-written rather than
drawn from a corpus, so everywhere outside them
the surfaces agree by *shared implementation*, not by test — which is the stronger
guarantee only for as long as the import survives. And the leading underscore means
the predicate is **not** part of the surface frozen under semver: a consumer that
imports it is pinning a private name 1.x makes no promise about.
## Out-of-scope (documented boundary)
Embedding/vector-layer defenses (OWASP LLM08, downstream of persist); multimodal