llm-ingestion-pipeline-security/CHANGELOG.md

# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

Nothing yet.


## [1.0.0] — 2026-08-13

### Changed — the exported Python surface is frozen under semver

No code changed in this release. `1.0.0` is a governance promise, not a claim that
the library is finished: **no name exported from `llm_ingestion_guard` is removed,
renamed or given a different meaning without a `2.0.0`.** Measured before the tag,
the surface has been stable in form since `0.3.4` — four names added, none removed
or renamed — while behaviour moved across five releases (`0.4.0` … `0.7.0`).

**Detection behaviour is deliberately outside the freeze.** Severities, thresholds,
lexicon entries and the dispositions they produce are calibration, and calibration
moves in minor and patch releases. A payload that disposes `WARN` here may dispose
`FAIL_SECURE` in a later `1.x`; that is a fix, not a break. Assert on the
disposition your policy requires, not on a severity you observed.

The behaviour changes this freeze rests on are not repeated here — see `[0.3.0]`
for the active-content gate and the OKF adapter, and `[0.3.1]` for the
ordinary-link/image calibration that the two consumer promises pin.

### Changed — three limitations are conceded for `1.x` rather than deferred

`docs/LIMITATIONS.md` no longer says "deferred" or "pending" about any of them:

- `Severity` still carries disposition intent on the detection side. Separating
  *what was seen* from *how bad it is* changes `Finding` and `Severity`, so it is a
  `2.0.0` change. Read a finding's `id` for the capability.
- The input-cap asymmetry at `MAX_INPUT_CHARS` is permanent in `1.x`: surfaces that
  return content raise `OversizeInputError`, surfaces that return findings truncate
  and emit `active:oversize-input`.
- The multilingual homoglyph false positive is conceded more narrowly — no fix is
  promised, but it is calibration, so one may land in any `1.x` release.

`SECURITY.md` carries all three as documented boundaries and states the support
window for a `1.x` line.

### Known at the freeze, deliberately not blocking it

`docs/LIMITATIONS.md` §`:43` — an OKF block sequence with exactly one key per
element misparses silently in `okf.import_bundle`, so a pointer can ride through in
a key the `resource` allowlist never inspects. Closing it tightens what the adapter
admits: behaviour, not form, and shippable in a `1.x` minor. It is recorded here
because "we knew, and froze first" is a defensible position and "we forgot" is not.

Runtime coverage at the freeze: `llm-ingestion-okf` has measured `0.3.4` and run a
`0.3.4`→`0.6.1` differential on its own door across two Python versions;
`llm-security-commons` differentially tested its independent reconstruction of the
raw-HTML classifier against ours over 42 probe tags with 0 disagreements. **No
external consumer has run the `0.7.0` runtime**; the four symbols added since
`0.3.4` are additive, so a caller that does not invoke them is unaffected.


## [0.7.0] — 2026-08-13

### Added — `active:raw-html-link`, a click-required carrier class for raw HTML

Raw HTML graded on activity alone: every active tag was HIGH. So the *same URL*
was LOW as `[t](https://example.com/guide)` and HIGH as
`<a href="https://example.com/guide">` — an asymmetry produced by syntax, not by
affordance. Following an anchor needs a human, exactly like the markdown inline
link that has been MEDIUM since 0.3.1.

`<a>` and `<area>` now report as **`active:raw-html-link` at MEDIUM**. Everything
a renderer fetches or executes unattended keeps `active:raw-html` at HIGH, and the
event-handler test runs *first*, so `<a onclick=...>` is graded as the
execute-class carrier it is rather than downgraded with the anchors.

The URL-attribute branch deliberately stays on the HIGH side: a name outside the
active set has unknown rendering, and `href` is not the only URL attribute it may
carry. Grading `<Card src="...">` as a link would be reasoning, not measurement.

**This is a new label, and labels are a contract surface consumers pin against.**
A document that previously produced one `active:raw-html` finding may now produce
two findings, one per carrier class.

### Changed — a tag whose whole affordance is a URL it does not carry is inert

`</a>`, `<Frame>`, `<video />` and `<img alt="...">` without `src` were active by
*name* while naming no target at all. This is `<base />`'s argument from 0.6.0 —
"attribute-less, therefore no affordance in any renderer" — applied to the rest of
the name branch. The test is for the URL attribute's **presence**, not for a
readable value: a value the parser cannot resolve keeps the tag active, mirroring
the fail-secure gap `_url_attr_is_external` already leaves open.

Every other member of the active name set does something a URL cannot describe —
`<script>` executes its body, `<style>` restyles, `<form>` submits — and stays
active with no attributes at all.

### Changed — `active_tag_class` is the classification point; `is_active_tag` wraps it

`docs/rawhtml-census.py` measures candidates by patching this symbol, and a
boolean could only express a narrowing, never a regrade. Left as a boolean, every
carrier candidate would have measured equal to PRODUCTION — silently, and in the
direction that reads as "no change helps".

### Measured

`docs/rawhtml-census.py`, three populations, each at one corpus state and each
against its own denominator — the two wiki corpora share content and are never
summed. Documents that stop being `fail_secure` under `PRESET_USER_UPLOAD`, from
0.6.0 as shipped to 0.7.0, with the ceiling being the raw-HTML detector switched
off entirely:

| population | documents | 0.6.0 → 0.7.0 | ceiling | share of achievable |
|---|---|---|---|---|
| reference-corpus | 389 | 54 → 53 | 53 | 1 of 1 |
| vendor-harvest | 187 | 62 → 20 | 18 | 42 of 44 (95%) |
| generated-notes | 552 | 59 → 15 | 13 | 44 of 46 (96%) |

**Neither change alone is worth shipping, and the census is why they went out
together.** Alone, the split frees 8 documents in each wiki corpus and the
narrowing 21 and 23 — but 8+21 measures 42 and 8+23 measures 44. The residual is
**13 documents in both corpora**: the narrowing strips a document's `</a>` and
`<Frame>`, and what is left is the `<a href=...>` the split grades down, so each
change alone leaves the document blocked by the other's residue.

**Tightening, measured: 0 documents on both trust tiers, in all three
populations.** That zero is empirical and thinner than it looks — the split
*alone* tightens 13 documents on the trusted tier in vendor-harvest and 14 in
generated-notes, and the narrowing cancels each one. See `docs/LIMITATIONS.md`
for why it must not be read as "cannot happen".

The `PRODUCTION (as shipped)` row matched `C1 + D (0.7.0)` field for field in
every population, which is the check that the census and the shipped predicate
have not drifted apart.

### Known behaviour change

**`count` drops on documents containing `</a>`.** Through 0.6.1 an end tag was
active by name, so `count` ran roughly 1.6× the opening-tag total and a start/end
pair counted 2. It is now the opening-tag total. The field's meaning did not
change and the finding count is unaffected — the class still collapses to one
finding per class per document.


## [0.6.1] — 2026-08-11

### Fixed — the zero-width check tested identity, so emoji-composed documents were hard-blocked

`_ZERO_WIDTH` (sanitize, input) and `_ZERO_WIDTH_CPS` (output,
`_scan_invisible_carriers`) tested U+200D on codepoint membership alone.
`disposition._CARRIER_LABELS` grades a carrier as any-tier `fail_secure` with no
appeal, so **any** first-party document containing a ZWJ-composed emoji —
professions, families, skin tones, flag variants — was blocked permanently, with
no preset able to release it. Reported by `ms-ai-architect`, confirmed here
against the code.

The worse half was not in the report: sanitize *removed* the joiner, silently
decomposing one emoji into two unrelated ones. A module whose published contract
is "only ever removes carriers" was corrupting content.

The fix is the shape our own lexicon row `unicode:zero-width-in-word`
(`\w[ZW]\w`) already used — **judge the joiner by context, not identity**. A ZWJ
is exempt only when *both* neighbours are emoji-context codepoints. Half-context
is not context, so `a<ZWJ>😀` stays a carrier and an attacker cannot buy
exemption with a single trailing emoji.

**Blocks, not an emoji table.** Measured against Unicode 17.0's
`emoji-zwj-sequences.txt`: 1614 RGI sequences use 122 distinct codepoints
adjacent to a ZWJ, and five block ranges cover 122/122. The measurement earned
its keep — the hand-reasoned candidate table missed U+2194, U+2195 and U+2B1B.
Shipping the RGI list itself would be exact the day it landed and stale at the
next Unicode release, reopening this false positive for every new emoji; whole
blocks carry the unassigned headroom (458 `Cn` codepoints) that future emoji are
allocated into, so the table does not age. The predicate is defined once in
`sanitize` and imported by `output`; a cross-surface test asserts the two halves
agree on six inputs.

### Known behaviour change

**Documents whose only finding was an emoji-context ZWJ now persist unattended.**
On `PRESET_USER_UPLOAD` they move from `fail_secure` to `WARN`. Measured across
the three false-positive populations at one corpus state: **1 document of 1126**
(reference-corpus 1/389, vendor-harvest 0/187, generated-notes 0/550). This is a
loosening of the *upload door*, not of detection — recall is unchanged at 128/128
demonstrated classes with 6/6 documented gaps holding, and a ZWJ anywhere else,
including between a word character and an emoji, is graded exactly as before.

**Patch, not minor.** 0.6.0 called itself minor for loosening the same door, but
that was a policy choice — `<base>` left the active name set by decision. This
one restores a contract the module already published, against a class that was
never meant to be blocked. A fix whose observable effect is the point of the fix
is what the patch level is for.

### Residuals — both documented (`docs/LIMITATIONS.md`, 33 → 34 items)

- **A ZWJ between two emoji is now exempt**, so it can carry a narrowband covert
  channel: one emoji per bit, and it cannot split a word. A deliberate narrowing,
  stated rather than hidden.
- **U+200C (ZWNJ) still has no context test.** It is orthographically *required*
  in Persian, Arabic and Devanagari, so those documents stay hard-blocked. The
  criterion has to be script-based rather than pictographic, and no corpus is
  here to verify one against — parked as a known false-positive class rather than
  guessed at.

736 tests pass (was 727).


## [0.6.0] — 2026-08-11

### Changed — `active:raw-html` stops firing on two things that carry no affordance

`is_active_tag` had two over-reaching branches, both measured on consumer corpora
rather than argued from the code:

- **The URL-attribute branch was a presence test.** Any element carrying `href=`,
  `src=`, `action=` … graded HIGH regardless of where the URL pointed. An MDX
  `<Card href="/en/agent-sdk/quickstart">` — an internal doc route — reaches no
  attacker-controlled host, and neither does Azure APIM policy XML's `<set-header>`.
  The branch now requires an **external** target (absolute scheme or
  protocol-relative), the rule the markdown paths have applied since 0.3.1.
- **`<base>` left the active name set.** HTML's `<base>` has its entire affordance in
  its `href`, which the URL-attribute branch still catches. APIM's attribute-less
  `<base />` means "run the inherited policy" and is inert in every renderer.

**Measured before and after in one session, against one corpus state** — the two
wiki corpora are living, so a before/after split across sessions would mix this
change with re-harvest drift:

| population | n | before | after | ceiling (raw-HTML off) |
|---|---|---|---|---|
| reference-corpus | 389 | 133 | **108** | 107 |
| vendor-harvest | 187 | 100 | **98** | 62 |
| generated-notes | 550 | 90 | **88** | 49 |

96% of the achievable reduction in reference-corpus, 5% in the two wiki corpora:
the over-reach was nearly the whole raw-html cost in APIM policy XML and nearly none
of it in vendor documentation, where what remains is real HTML — `<a>` 298, `<frame>`
94, `<img>` 63 — caught correctly by the name branch.

**The two classes had to be measured together.** Alone they free 3 and 13 documents
in reference-corpus; together, 25. A document carrying one usually carries the other,
so closing either alone leaves it blocked by its twin. `docs/rawhtml-census.py` now
carries a `PRODUCTION` row that re-measures the shipped predicate instead of a
hypothetical, so a doc number and the code cannot drift apart unnoticed.

### Known behaviour change

**Documents whose only finding was one of these two classes now persist unattended.**
On `PRESET_USER_UPLOAD` they move from `fail_secure` / `quarantine_review` to `WARN`
— 25 documents in the reference corpus, 2 in each wiki corpus. This is a deliberate
loosening of the *upload door*, not of detection: recall is unchanged at 128/128
demonstrated classes with 6/6 documented gaps holding, and a tag that is active by
name, carries an `on*=` handler, or points anywhere external is graded exactly as
before. An element outside the active name set whose only URL attribute is
doc-relative is the whole of what changed.

### Fixed — the scanner and the mutator no longer share one predicate

`neutralize` imported `is_active_tag` from `active_content` by name, so narrowing the
scanner would have silently narrowed the opt-in mutator as well — and **no test in
the suite discriminated the two halves**: every `neutralize:raw-html` payload stayed
active under each narrowing considered. The predicates are now separate symbols,
`is_active_tag` (scanner, external-target rule) and `is_defangable_tag` (mutator,
unchanged broad behaviour), and the mutator half is pinned by its own test. Over-
defanging costs nothing there — `neutralize` is opt-in and blocks no disposition —
while under-defanging would hand a human a live construct.

### Self-safety (OWASP LLM10)

Reading a URL attribute's *value* needs a pattern the presence test does not provide.
It reuses the same literal alternation with the value attached, so no new run shape
enters the table, and `_REDOS_PAYLOADS` gains a row (`active-url-attr-value`) whose
unit **denies** the `=` the pattern requires — a unit supplying it matches at once and
never exercises the run. Measured at 100_000 chars: 0.031–0.046s across five attack
shapes, against the suite's 2.0s bound. A gap between the two patterns fails secure:
an attribute seen by the presence test but unreadable by the value parser counts as
external, so it over-blocks rather than under-blocks.

727 tests pass (was 717).


## [0.5.0] — 2026-08-11

### Added — the axis separation: assessment (`Risk`) vs action (`Disposition`)

> **Additive, and measured to be so.** Every disposition 0.4.0 rendered is
> rendered identically: the full suite went 703 → 715 with no test changed, the
> coverage matrix holds at 128/128 recall with 6/6 documented gaps, and the
> `PRESET_USER_UPLOAD` grading table locked in 0.3.1 was re-measured row by row
> and is unchanged. A caller that never reads the new field sees no difference.

`decide` and `guard` returned a `Disposition` — `WARN` / `QUARANTINE_REVIEW` /
`FAIL_SECURE` — which names an **action**. But BRIEF design principle 4 says the
library reports and the *pipeline* decides, and `disposition.py` admitted the
gap in its own docstring: *"It imposes no blocking of its own."* So the library
returned an action it cannot enforce, while discarding the judgement that
produced it. A consumer wanting different behaviour had to reinterpret the
action itself, which is why a consumer ends up pinning our *grading* — the
action was all they got.

- **`Risk`** — the new assessment axis: `NONE` / `LOW` / `ELEVATED` / `SEVERE`.
  It answers *how dangerous is this artifact given its source context*, and is
  trust-aware exactly as BRIEF §4.7 describes the domain: the same finding
  genuinely is a different judgement in authored prose than in a code fence.
- **`DispositionResult.assessment`** — carries that judgement alongside the
  action. The field is **required, with no default**: `Risk.NONE` would be the
  natural-looking default and is the wrong one, since a construction site that
  forgot it would report *clean* and the axis would fail open.
- **`Policy.action_map`** — an optional `Risk -> Disposition` override, so
  "hold for review where you would block" is a policy statement rather than a
  reason to pin our grading. Defaults to `None`, which means
  `DEFAULT_ACTION_MAP` and keeps an untouched `Policy` hashable as before. A
  partial map falls back per-level instead of raising.
- Both overlays — compound escalation and the quarantine floor — now move the
  **assessment**, so a custom action map cannot silently drop them.
- The fail-closed path in `guard` pins both axes to their most severe value and
  deliberately does **not** route through the action map: a policy that
  downgrades `SEVERE` means "I accept this class of finding", never "I accept a
  scanner that crashed on crafted input" (§4.6).

`NONE` and `LOW` both map to `WARN`, which is the point rather than an
oversight: a clean document and one carrying only low-severity findings were a
single indistinguishable value through 0.4.0.

### Known limitation recorded (32, was 31)

`Severity` still carries disposition intent on the *detection* side — the
separation above is caller-side only. Two places say so outright:
`ACTIVE_CONTENT_ORDINARY_SEVERITY = LOW` exists because grading an ordinary
external image `HIGH` fail-secured ordinary uploads, and the quarantine floor
was raised to `MEDIUM+` to repair the same regression from the other end.
Closing it changes the grading and so fires a consumer-notification promise;
deferred deliberately. See `docs/LIMITATIONS.md`.

### Measured — what the upload door costs on benign documents (33rd limitation)

Every field measurement this project had published was **per URL**. None of them
answered the question a consumer actually asks: *how often does an ordinary
document cost me a human?* Three benign populations were run through
`screen_output` under `PRESET_USER_UPLOAD` and counted at document granularity —
98 of 185 vendor-published doc pages (53.0%), 88 of 547 model-written notes
(16.1%), and 133 of 389 first-party reference documents (34.2%) disposed to
something other than WARN.

The number is bad and is published as measured. `docs/PLAN-v1.md` committed to
that in advance — *"et rødt FP-resultat er like verdifullt"* — and the response
here is a documented limitation, not a recalibration: moving the grading would
fire a locked consumer-notification promise, and the drivers are residuals this
document already concedes rather than anything newly discovered.

- **`docs/fp-sweep.py`** — the method, re-runnable, corpus roots as arguments.
  It refuses to print a pooled total (the populations have different provenance
  and different denominators) and it aborts if the default action map stops
  sending exactly `NONE` and `LOW` to WARN, since the published count is a
  statement about *assessed risk* and only equals one while that holds.
- **`tests/test_corpus.py::test_the_published_fp_metric_is_a_risk_statement`** —
  the same equivalence, pinned in the suite. An `action_map` override is a
  supported feature as of the axis separation above, so without this pin a
  consumer-facing number could change meaning with nothing failing.

### Corrected — 0.3.3 listed two behaviour-change classes and there were three

Sweeping the same population against the **v0.3.1 tag a consumer actually pins**
(scratch venv, `git+file://…@v0.3.1`, resolved version asserted) returned 99 of
185 (53.5%) where the current tree returns 98 (53.0%). One document moved, and
chasing it corrects a claim rather than confirming one.

`docs-en-fullscreen.txt` disposed QUARANTINE_REVIEW at 0.3.1 and WARN now,
because `markdown:link-anchor-injection` no longer fires on it. Under 0.3.1 that
pattern matched **300 characters of ordinary prose**: it opened at a `[`, ran
across intervening text containing the word *execute*, and closed at a distant
`](…)` belonging to a different construct. The 0.3.3 ReDoS fix excluded `[` from
the anchor class and `(` from the target class, which telescopes the runaway —
and, as a side effect nobody measured at the time, deletes this false-positive
class too.

0.3.3's *Known behaviour changes* said **"None measured"** and then named two
exceptions: URLs with a literal `(` in the target, and comment bodies with a
literal `(` before the keyword. It missed the third: an anchor can no longer span
a `[`, so a match that used to bridge two separate markdown constructs no longer
forms. The correction is in our favour — one fewer false positive per 185
documents of vendor documentation — but it was a behaviour change presented as
none, and it took a field sweep to find it.

### Fixed

- A **retracted** number was still living in a test comment.
  `tests/test_wiring.py` credited a consumer's capture store with 35 of 35
  query-carrying URLs. That consumer retracted it the next day and re-measured 28
  of 28 on the same 81-URL corpus; `docs/LIMITATIONS.md` was corrected then and
  the comment was not. Corrected, with the retraction written into the comment so
  it cannot read as a second, disagreeing measurement.

- **Five current-state version claims had never been updated by any release.**
  The 0.4.0 release commit touched three files — `CHANGELOG.md`,
  `pyproject.toml`, `src/llm_ingestion_guard/__init__.py` — and deferred the
  README deliberately, so that the install block would not point at a tag before
  a clean-venv install had proven it resolved. That proof step never ran, so tag
  `v0.4.0` permanently carries a README advertising `v0.3.4`. The tag is not
  moved; the ordering is.

  Sweeping *every* tracked file for a version claim, rather than the four
  surfaces the release checklist named, found four more that no release had ever
  touched — plus a stale test count:

  - `SECURITY.md` — "The project is pre-1.0 (`0.2.x`, alpha). Only the latest
    published version receives fixes." The only one with a consequence for an
    outsider: it named a support window two minor lines behind the code.
  - `README.md` — `**Status:** v0.3`, stale since 0.4.0.
  - `docs/BRIEF.md` and `CLAUDE.md` — "v0.2 (alpha)", stale since 0.3.0. The
    latter also claimed 12 modules where `src/` has 15.
  - `docs/ADOPTION-BRIEF.md` — "**703 passing**", where the suite is at 717.

  Every one of them is a *current-state* claim. Measurement provenance — "New in
  `v0.4.0`", "verified identical on 0.2.0 and 0.3.1", "measured against the
  v0.3.1 tag" — is left exactly as written, because bumping those would falsify
  the record rather than update it. From here all current-state surfaces move in
  the release commit itself and are verified by `git show <sha>` *before* the tag
  exists, since that is the only check the previous ordering could not perform.

  Found because `llm-ingestion-okf` took our report of this defect class as a
  hypothesis about their own repo, measured it, found a worse instance, and sent
  back the generalization: writing down a trap is not the same as applying it.


## [0.4.0] — 2026-08-10

> **Behaviour change, not a pure fix — and that is why this is 0.4.0 and not
> 0.3.5.** The three transform surfaces gain a refusal path they did not have. A
> caller that passes a document larger than 1 000 000 characters now gets an
> exception where it previously got a result. Adding a raise to a function that
> was previously total is breaking under SemVer whatever the measured blast
> radius turns out to be, so the number follows the change, not the survey.
>
> **The measured blast radius, for the record: zero.** `linkedin-studio` pins an
> exact tag, so nothing reaches it until it re-pins. `llm-ingestion-okf` moved to
> the range `>=0.3,<0.4` (their `f536e13`), so a 0.3.5 would have landed on them
> at their next resolve without an action on their part — but they answered our
> query (`20260802T193351Z`) with a measured **no**: zero call sites for
> `sanitize` / `fence` / `neutralize` / `prepare_input` anywhere in their `src/`.
> Their `screen_output` path reaches only `scan_output`, which truncates and does
> not raise. Releasing as 0.4.0 puts this outside their ceiling regardless, so
> they cross it deliberately rather than by resolving.

### Added — input-size cap on the transform surfaces (OWASP LLM10)

`sanitize`, `fence` and `neutralize` now raise `OversizeInputError` above
`MAX_INPUT_CHARS` (1 000 000) instead of accepting text of any length. Since
`sanitize` is step 1 of `prepare_input` and only ever *removes*, that single
refusal bounds the whole input path.

They **reject** where the scanners **truncate**, and the asymmetry is the point:

- `scan_lexicon` / `scan_output` return findings. Reading a prefix costs
  detection in the tail and nothing else — a lossy answer, but an answer.
- `sanitize` / `fence` / `neutralize` return *content*. Truncating would return
  a shortened document (silent data loss for anything that persists the result)
  or a transformed prefix followed by an untransformed tail — a bypass, since an
  attacker chooses where in the document the payload sits.

The invariant the three now keep: **returned text is always fully transformed,
or not returned at all.**

`OversizeInputError` subclasses `ContractViolation`, so a pipeline already
bracketing its quarantined stage in `except ContractViolation` keeps failing
closed. Like its parent it is alert-routable: the message carries the size and
the cap, `details` names the refusing surface, and neither carries input.
`max_input_chars` is a per-call parameter, defaulting to the single calibrated
constant.

### Added — the last two detection surfaces bound their input too

`scan_active_content` **called directly** and `okf.link_graph` were the two
surfaces still reading attacker-supplied text with no cap. Both truncate and
record, the way the other scanners do:

- `scan_active_content(text, source, max_scan_chars=MAX_SCAN_CHARS)` emits one
  `active:oversize-input` finding (MEDIUM, LLM10) and scans the prefix. Reached
  through `scan_output` the text is already under that surface's cap, so the flag
  is raised once, there — `max_scan_chars` is now passed down.
- `link_graph(bundle, max_scan_chars=MAX_SCAN_CHARS)` caps each body and records
  `(from_id, body_length)` in the new `LinkGraphResult.truncated` field. The
  field is additive with a default, so existing positional construction and
  attribute access are unaffected.

**What truncation costs is named rather than implied:** past the cap, "no
finding" means "not looked at". That is precisely what a silent truncation would
hide, and why `truncated` exists as a field instead of a log line — it is what
separates "no links past here" from "no links *read* past here".

Recorded in `docs/LIMITATIONS.md`.


## [0.3.4] — 2026-08-01

> **Denial-of-service fix on the INPUT path. Upgrade from 0.3.3.** 0.3.3 swept
> the 83 lexicon patterns arm by arm and left every other table on 0.3.2's
> hand-written rows. Generalising the sweep over all eleven regex-bearing modules
> found three more quadratic patterns — two of them on the input path, one in
> `sanitize`, the first thing every ingested document touches. No disposition
> changes: recall was measured case by case and nothing was lost. Earlier tags
> are not moved.

### Fixed — three quadratic patterns, two on the input path

Same class as everything 0.3.2 and 0.3.3 fixed: a run in front of a **required**
literal, so crafted input that never supplies the literal makes every start
position rescan the tail. Each exponent is read across four doublings, not from a
two-point ratio.

| Pattern | Crafted payload | Measured @ 100 000 | Exponent |
|---|---|---|---|
| `sanitize._HTML_COMMENT_RE` | `<!--` × N | **20.1 s** | 1.96–2.14 |
| `active_content.URL_IN_TEXT_RE` | `<a ` + `A` × N + `>` | 12.99 s / 14.9 s | 1.87–2.22 |
| `okf._MD_LINK_RE` | `[` × N | 7.1 s | 1.99–2.05 |

These are worse than the 0.3.3 findings, and the reason is a separate finding of
its own: `MAX_SCAN_CHARS` is applied in `scan_lexicon` and `scan_output` **only**.
`sanitize`, `neutralize`, `scan_active_content` and the okf link graph accept
input of any size, so there is no cap to extrapolate to. Now documented as a
residual in `docs/LIMITATIONS.md`; extending the cap into the input path changes
the contract for existing callers and is deliberately not done in a ReDoS patch.

Each fix is the one the pattern's own shape allows — the 0.3.3 lesson that a fix
choice must not be copied blindly from a neighbouring table:

- **`sanitize`** drops the regex for a `str.find` scan, semantically identical to
  the lazy `<!--.*?-->` it replaces. Excluding `<` from the run would lose every
  comment containing markup (`<!-- <b>x</b> -->` is the ordinary case); bounding
  the run would be a one-line carrier bypass of the exact construct the stripper
  exists to remove. The module's own "no catastrophic backtracking" comment was
  wrong in the same way `output`'s was before 0.3.2, and is corrected in place.
- **`URL_IN_TEXT_RE`** bounds its scheme run to an RFC 3986 scheme (`{0,63}`).
  Bounding is safe *here* only because this is a defanger applied inside a tag
  already flagged `active:raw-html`, so padding shifts where the match starts
  rather than evading detection. A lookbehind killing interior start positions
  was measured too and **rejected**: it drops `-http://evil.com` and
  `.http://x.com`, a one-character evasion of the defanger. Bounded, the pattern
  runs in 0.185 s at the full 1 000 000-char cap.
- **`okf._MD_LINK_RE`** excludes `[`, matching `active_content.MD_LINK_RE`
  exactly, including the nested-label trade already documented there.

### Changed — the sweep covers every regex surface, not one table

`docs/redos-sweep.py` now sweeps **150 patterns across 11 tables** (0.3.3 covered
83 in one). The collector is mechanical on both axes so no one has to remember to
list anything: it walks each module's namespace for compiled patterns, and it
derives each pattern's call mode from the module source, because `.sub()` and
`.finditer()` visit every start position where `.match()` cannot. A pattern
reachable only through a helper parameter gets the worst mode, marked `*` — the
fallback can over-measure but never miss.

Two arm shapes the generator cannot express are pinned by hand as a result: a tag
that *closes* around a long body (repeating-unit payloads never close it), and a
run of plain characters carrying no anchor at all. The `okf` destination run gets
no row on purpose: `[^)\s]+` cannot fail, so a pin for it could never go red.

A lexicon candidate flagged at ×2.8 measured **linear** across four doublings
(exponent 0.96–1.03) — the near-noise-floor false flag the script's own docstring
warns about, confirmed a second time.

676 tests (+10). Coverage matrix unchanged at 128/128 caught, 6/6 gaps holding.

## [0.3.3] — 2026-07-31

> **Denial-of-service fix, and a correction to 0.3.2. Upgrade from 0.3.2.** The
> sweep 0.3.2 shipped was incomplete, and it said otherwise. Two lexicon patterns
> were still quadratic — reachable through `scan_output`, not only on the input
> path. No disposition changes: recall was measured case by case and nothing was
> lost. The v0.3.2 tag is not moved.

### Fixed — two quadratic patterns in the lexicon table

`8deca93` scoped the remaining ReDoS duty to the lexicon path, and this is that
work: all 83 patterns measured, arm by arm. Two are quadratic, same shape as
everything 0.3.2 fixed — a run in front of a **required** literal, where the run
may cross the pattern's own opening anchor.

| Pattern (arm) | Crafted unit | Measured | At the 1 000 000-char cap |
|---|---|---|---|
| `markdown:link-anchor-injection` (anchor text) | `[` | 1.91 s @ 8 000 | **~8.3 hours** |
| `markdown:link-anchor-injection` (URL run) | `[system](` | 0.006 s @ 8 000 | ~89 seconds |
| `markdown:link-ref-comment` (`.*` run) | `[//]: # (` | 0.22 s @ 8 000 | **~1.0 hour** |

Exponent measured over five points (1 000 → 16 000): **1.98** — quadratic, not
exponential. Legitimate content of the same size is unaffected: 0.316 s at
N=100 000 (prose 0.316 / html 0.315 / markdown 0.297 / connection-string 0.296).

**These were not input-path-only, and that is the correction.** `scan_lexicon`
runs on the output path too, so 0.3.2's *"the last quadratic-backtracking site on
the output path"* was false when written. Measured through the public gate before
this fix: `scan_output("[" * 100_000)` took **334.7 s**. The claim was too broad
because the sweep behind it drove the `[` payload only through
`scan_active_content` — no row ever drove it through the lexicon. The statement is
corrected in `docs/LIMITATIONS.md`.

The fix is anchor exclusion, per the rule `active_content` already documents —
bounding attacker-controlled content would be a one-line detection bypass. The
excluded character is `(`, not `[`:

```
markdown:link-anchor-injection
  \[[^\]\[]*(?:system|…)[^\]\[]*\]\([^)(]+\)
markdown:link-ref-comment
  \[//\]:\s*#\s*\([^(\n]*(?:ignore|…)
```

`[` was the obvious choice and it was measurably worse. Excluding `[` from the
URL run drops `[override your rules](https://[::1]/x)` — still covered, three
other patterns fire on it — but excluding `[` from the link-ref comment run drops
`[//]: # (see [x] then ignore this)`, which **no other pattern catches**. The
anchors contain `(` as well, so excluding `(` telescopes just as effectively at
zero measured recall cost. Both forms verified linear (×1.99–2.02 on doubling).

### Known behaviour changes

- **None measured.** Every case that matched before still matches, except URLs
  containing a literal `(` inside a markdown link target and comment bodies
  containing a literal `(` before the keyword. No corpus, showcase, or coverage
  row moved; 666 tests pass.

### Tests

Four rows added. Three name the guilty pattern per arm
(`test_crafted_redos_payload_stays_bounded_in_the_lexicon`), one covers the
composed gate (`test_gate_is_bounded_on_the_payload_the_first_sweep_missed`).
Pre-fix they failed at 297 s, 8.1 s, 55 s and 334.7 s.

`N` is per row deliberately. The URL arm is quadratic with a small constant and
ran 0.9 s **unfixed** at N=100 000 — under the 2.0 s bound, so that row would have
passed whether or not the pattern was fixed. It is measured at N=300 000 instead,
where crafted (8.10 s) and legitimate (0.926 s) separate 8.8×.

### Residual

The sweep flags on timing and ignores measurements below a 1.5 ms noise floor at
N=8 000. An arm hiding just under it could still cost **~23 s** at the cap, so what
this supports is *"no arm worse than ~23 s"*, not *"no quadratic arm remains"*.
The blind spot is not hypothetical: a generic-payload pass found only one of the
two patterns. The second appeared only once payloads were synthesised per run
from each pattern's own skeleton. Recorded in `docs/LIMITATIONS.md`.

## [0.3.2] — 2026-07-31

> **Denial-of-service fix. Upgrade from 0.3.1.** The output gate could be made to
> spend hours on a single call by crafted input it accepts by design. No
> disposition changes for ordinary documents — the one measured exception is
> listed under *Known behaviour changes* below. The v0.3.1 tag is not moved.

### Fixed — 19 quadratic regex runs on the output path

`scan_output` claimed LLM10 self-safety on the grounds that its patterns contain no
nested quantifiers. That is true and it is not the property that matters. A run in
front of a **required** literal, reachable from a short anchor, is enough: crafted
input repeats the anchor and never supplies the literal, so every start position
rescans the tail. Quadratic, not exponential — and quadratic is sufficient here.

Measured, not argued (Python 3.14, this machine):

| Input | Time through `scan_output` |
|---|---|
| `<a:` × 100 000 (300 KB) | **458.7 s** |
| size-matched ordinary prose | 0.31 s |
| same payload extrapolated to the 1 000 000-char input the gate itself accepts | **~5.7 hours for one call** |

`max_scan_chars` does not mitigate this. It bounds the *input*; quadratic work on a
bounded input is still hours. That claim was stated in both `output.py` and
`calibration.py` and is corrected in both.

The fix is per pattern, not uniform:

- **`active_content` + the lexicon table (15 runs)** — exclude the character that
  opens the pattern's own anchor (`[` for markdown, `<` for tags), so a run cannot
  reach past the next start position and the per-start costs telescope. Verified to
  cost no recall: long URLs, long alt text, and `<` inside a quoted attribute all
  still match. Bounding instead would have been linear too, but wrong here — the
  content is attacker-controlled, so padding past a bound would be a one-line bypass
  of the EchoLeak class this table exists to catch.
- **`*-connstr` secret egress (4 runs)** — bound the password at the new
  `MAX_CONNSTR_VALUE` (256) in `calibration`. The exclusion fix is unavailable: the
  anchor character is `/`, and passwords containing `/` are the common case
  (measured — they match today).
- **`hybrid-xss:script-tag`** — had neither option, since its run is the script
  *body*, which may legitimately contain `<`. It now matches the opening tag and no
  longer requires `</script>`.

### Known behaviour changes

Two, both measured against the v0.3.1 tag rather than reasoned about:

- **A JWT used as a DB password, over 256 chars, is no longer CRITICAL.** The
  remaining detections (`entropy:base64-blob` HIGH, `egress:jwt-token` MEDIUM) top
  out below CRITICAL, so the any-tier block is lost: under `PRESET_TRUSTED_SOURCE`
  such a document moves from `fail_secure` to `quarantine_review`. Under
  `PRESET_USER_UPLOAD` it still `fail_secure`s, and a *generic* long password still
  trips `entropy:base64-blob` at CRITICAL with no change at all. The credential is
  never silently missed; on one preset it is held for review instead of halted.
- **`hybrid-xss:script-tag` now fires on prose that merely mentions `<script>`** —
  and this costs no consumer a disposition. Any text containing a literal
  `<script>` already produced `active:raw-html` at HIGH on 0.3.1, so the same
  document disposed identically before and after. The label is new; the outcome is
  not. By the same measurement, the fail-open this closed (an unclosed
  `<script>alert(1)`) was confined to `scan_lexicon` called on its own — through
  either composed gate, `active:raw-html` already caught it.

Both are recorded in `docs/LIMITATIONS.md` (still 29 items — these replace nothing).

### Method note

The defect was found by a composed-gate DoS test that stayed red after every
individual scanner had been made linear; the remaining 813× was the lexicon's six
`html-obfuscation` patterns. A per-scanner test alone would have shipped it. The
static shape analysis used to find candidates also missed `[\s\S]*?` in
`script-tag` — the sweep that matters is measurement, not a regex over regexes.

662 tests pass (was 642), and the suite is faster than before the fix.

## [0.3.1] — 2026-07-25

> **Regression fix. Upgrade from 0.3.0.** v0.3.0 made the high-untrust upload path
> unusable for ordinary documents — measured, not projected. `llm-ingestion-okf`
> projected the consequence from the 0.3.0 changelog text *before* the tag was cut;
> the release went out without the inbox being read. The v0.3.0 tag is not moved.

### Fixed — the upload path is usable again without losing EchoLeak detection

Measured on v0.3.0, both doors (`screen_output` under `PRESET_USER_UPLOAD` and
`okf.import_bundle` with `origin=EXTERNAL`): a document with **one ordinary remote
image** disposed `fail_secure`; one ordinary link, autolink or reference definition
disposed `quarantine_review`. Only documents with no external references persisted.
Two independent defects compounded, and both had to be fixed — either alone leaves
the path blocked:

- **Severity graded on construct type instead of URL shape.** `markdown-image` was
  HIGH for *any* external image, but the exfiltration primitive is not "an image" —
  it is a URL that moves bytes to a host the attacker controls.
  `![diagram](https://example.com/arch.png)` carries nothing. Severity now grades on
  shape: a URL that only *names* a remote document (http(s) or protocol-relative,
  no query, no userinfo, no percent-escapes, no opaque host label or path segment)
  is **LOW**; anything that can carry a value keeps the carrier's full severity.
  `raw-html` and `data:` URIs have no ordinary form and stay HIGH unconditionally.
  Opacity reuses `entropy`'s primitives — decodable base64 (≥20 chars), hex id
  (≥32), or Shannon entropy ≥4.4 at ≥24 chars — calibrated 2026-07-25 against real
  documentation URLs (worst legitimate token H=4.08; exfil payload segments
  4.36-4.54). New constants live in `calibration` with the rest.
- **The `quarantine_default` floor fired on *any* finding.** It rested on the premise
  that a finding is the exception; adding the active-content detector in 0.3.0 made
  every ordinary markdown link a finding, and the floor then held ordinary documents
  for review. The floor now fires at **MEDIUM+**. This is a no-op for every detector
  that shipped before 0.3.0 — the lexicon holds no LOW/INFO pattern and no other
  detector emits LOW (asserted in `tests/test_calibration.py`) — which is why this is
  a patch and not a minor.

**Unchanged, deliberately:** no new public API and no new preset (a middle tier is
0.4.0 work); the `allow_reserved=True` mode-b default stands — two independent
consumers document it as load-bearing; the gate still never rewrites content.

### Added

- **False-positive corpus covers ordinary markdown.** The 0.3.0 corpus had zero
  markdown links or images, asserted only under `PRESET_TRUSTED_SOURCE` (where every
  non-CRITICAL finding WARNs anyway), and drove the *input* path — so `scan_output`
  step 6, where active content actually lives, was never reached. That is how a
  regression this size passed 522 green tests. The corpus now carries realistic
  documents and asserts them on the **output gate under the upload preset**, plus a
  counter-corpus of exfil-shaped URLs (query, base64/hex path segment,
  percent-encoded payload, opaque subdomain, userinfo) that must still block.
- **Two new documented gaps** in `docs/LIMITATIONS.md`, both asserted by the coverage
  matrix: **pure beaconing** (a bare-path image on a hostile host still fetches, and
  the fetch is not graded) and **short opaque URL segments** (<24 chars, below what
  entropy can resolve). Percent-escapes counting as data-carrying is recorded there
  as a known false positive.

## [0.3.0] — 2026-07-25

> **A minor bump, not a patch — deliberately.** The changes under *Changed* alter what
> an existing caller observes with no code change on their side, so a `>=0.2,<0.3` pin
> stops here rather than absorbing them silently. Re-test that branch before widening
> the pin. Still alpha: the public API may change again before 1.0.

### Changed — observable gate behaviour (re-test before upgrading)

Three commits since v0.2.0 change dispositions for an unchanged caller. Two tighten
the gate; one loosens it.

- **`okf.import_bundle` no longer path-rejects reserved basenames.** At v0.2.0,
  `index.md` / `log.md` anywhere in a received bundle was an unconditional
  per-concept hard reject (FAIL_SECURE), and `import_bundle` took no keyword for it.
  The new `allow_reserved` keyword **defaults to `True`** on this mode-b
  *received-bundle* path, so those files are scanned — their body is the
  highest-priority injection surface — rather than refused, and may clear the floor
  and become mergeable. **This is the one loosening change:** content a v0.2.0
  consumer never saw can now reach it, so a consumer whose tests pin the v0.2.0
  reject must re-check, not just bump. A front-end materialising individual
  *uploads* must pass `allow_reserved=False` to keep the shadow-reject there;
  `validate_concept_path` still defaults to `False`.
- **Active content now reaches the disposition engine.** `scan_output` step 6 runs
  `scan_active_content`, so markdown images/links, reference definitions, autolinks,
  raw active HTML and `data:` URIs surface as `active:*` findings (OWASP LLM05 — the
  EchoLeak / CVE-2025-32711 class) instead of being admitted with `findings=[]`.
  These carry real severities (zero-click auto-fetch/execute HIGH, click-required
  MEDIUM), and two MEDIUM+ findings compound-escalate one tier, so a document that
  passed clean at v0.2.0 can now WARN, quarantine, or fail secure in both
  `screen_output` and `okf.import_bundle`.
- **Base64-wrapped secrets are now caught as egress.** The output gate's
  decode-and-rescan feeds decoded base64 plaintext through both the lexicon and the
  LLM02 secret-egress detector, so a base64-wrapped credential surfaces as
  `decoded:egress:*` instead of disappearing. Hex-wrapped remains a documented gap
  (`docs/LIMITATIONS.md`).

### Added — runnable threat-coverage matrix

A single declarative manifest (`llm_ingestion_guard.coverage`) that proves, in one
place, every vulnerability class the guard stops — and the documented gaps it does
not. Two consumers of the same source of truth:

- `python -m llm_ingestion_guard.coverage` — a narrated matrix
  (`class -> OWASP -> expected -> observed -> verdict`); exit 0 iff every caught
  class is caught and every documented gap holds. Stdlib-only, CI-usable.
- `tests/test_coverage_matrix.py` — asserts total recall over the core matrix
  (carriers, all 83 lexicon patterns, entropy/decode-rescan, active content, the
  contract asserters, the disposition engine, OKF T1–T7), asserts every documented
  gap still holds, and guards completeness (every lexicon pattern id, and every
  OWASP anchor claimed, has a case). Adds the full 25-pattern LLM02 secret-egress
  set and the container-layer front-end classes (CSV formula-injection, zip-slip,
  zip-bomb, symlink). +165 tests (357 → 522), no core dependency added.

This is the real-case validation gate ahead of a v1.0 freeze.

### Documentation — consumer adoption + README value proposition

- `docs/ADOPTION-BRIEF.md` — a self-contained brief a consumer repo (OKF
  second-brain / LLM wiki) can plan an inclusion from: the write-time trust-boundary
  argument, the two bookends + 8-step contract, the shipped OKF adapter
  (`import_bundle` mode-b), how to verify (coverage matrix), how to depend
  (stdlib-only core), and a checklist for *when/where* to wire it.
- README rewritten to lead with the write-time trust-boundary framing, add a
  first-class **OKF / LLM-wiki support (shipped)** section for `import_bundle`, a
  concrete **What it protects against** catalogue (attack classes grouped by OWASP
  anchor, driven by the coverage matrix), and correct the test badge (357 → 522).
  Every claim verified against the code.
- `docs/LIMITATIONS.md` — the full honest-limitations list (15 items + the four
  documented gaps + out-of-scope) moved out of the README, which now carries a
  high-impact summary + link, so protection and limits read in balance.

## [0.2.0] — 2026-07-06

### Added — OKF adapter (stream 1)

An OKF (Google Open Knowledge Format v0.1) adapter *on top of* the
format-agnostic core (`llm_ingestion_guard.okf`). The core stays `text ->
findings`; the adapter knows OKF structure and routes scannable regions into the
existing machinery. All TDD (failing test first), +61 tests. Verified against the
OKF `SPEC.md` (2026-07-06). See `docs/OKF-INGESTION-BRIEF.md` §8.

- `parse_frontmatter` — strict, reject-by-default frontmatter loader; refuses
  anchors, aliases, explicit tags, merge keys, block scalars and flow collections
  by construction, so YAML anchor/alias DoS and `!!python/object` coercion cannot
  occur (not a general YAML parser, by design). (T2)
- `scan_concept` — whole-concept scan surface: frontmatter values (incl.
  `description`, read first under progressive disclosure), `resource` and body all
  go through `scan_output`. (T1)
- `validate_concept_path` — path / reserved-name gate: rejects `..` traversal,
  absolute paths and `index.md` / `log.md` shadowing; returns the concept-ID. (T4)
- `validate_resource_url` — `resource` https allowlist: rejects non-https before
  commit (reject, not defang — the format imposes no scheme constraint itself). (T3)
- `stamp_concept` / `format_log_entry` — provenance stamping: origin × channel →
  trust × disposition per concept, emitted as `log.md` lines. Trust follows the
  origin, never the insertion channel. (T6)
- `import_bundle` — received-bundle iterator (mode b): validates each concept
  (path, frontmatter, resource, scan, stamp) independently; one bad concept is
  rejected fail-secure while the rest are still checked; the aggregate disposition
  is the most severe. (T7)
- `link_graph` / `resolve_link` / `extract_link_targets` — in-import cross-link
  graph: resolves `.md` links (bundle-absolute or relative) to concept-IDs, flags
  dangling links (the §7.2 dormant-injection signal) and rejects dangerous-scheme
  or bundle-escaping targets. (T5a)

### Deferred

- Cross-run persisted link graph (T5b) — catching a link planted in one run whose
  poisoned target is written in a *later* run (§7.2) needs durable graph state
  whose storage/ownership depends on the consuming pipeline. Deferred to the
  consumer-wiring stream; cross-run dormant links remain a documented residual
  risk (README honest-limitations).

## [0.1.0] — 2026-07-06 (alpha)

The stdlib-only core, built test-first (TDD) per `docs/PLAN.md`. Tagged `v0.1.0`.

### Added

- `report` — shared `Finding` / `Report` / `Severity` / `Source` types.
- `sanitize` — carrier stripping (zero-width, BIDI, Unicode-tag, HTML comment,
  `data:`); byte-identical on clean input.
- `entropy` — Shannon / base64-like / hex-blob detection; base64 decode-and-rescan.
- `lexicon` — JSON pattern data + loader; raw/normalized/homoglyph/rot13 variants;
  ReDoS-bounded, size-capped.
- `fence` — randomized per-call spotlight delimiter; attacker marker-strip.
- `neutralize` — opt-in defang of active-content output (byte-identical when clean).
- `output` — compose lexicon + entropy + decode-rescan over emitted text; secret
  egress patterns (OWASP LLM02); report-only, never mutates.
- `disposition` — WARN | QUARANTINE_REVIEW | FAIL_SECURE under a source-trust
  policy; compound-signal escalation; fail-**closed** when the scanner errors.
- `contract` — write-time asserters that raise: `assert_tool_less`,
  `assert_credential_allowlist`, `scoped_env`.
- `grounding` — the `SourceGroundingCheck` seam for semantic poisoning (interface
  only; `[judge]` implementation plugs in behind an extra).
- Top-level wiring — the `prepare_input` / `screen_output` §6 bookends plus the
  full public surface; end-to-end showcase and adversarial + false-positive corpora.

### Security

Pre-release hardening from an independent adversarial review (all TDD, failing
test first):

- `entropy` — decode-and-rescan now runs **before** false-positive suppression,
  so an injection blob prefixed with an SRI/media marker (to dodge the entropy
  finding) is still decoded and rescanned by the lexicon. Suppression gates only
  the entropy finding, never the decode.
- `output` — the invisible-carrier invariant now holds on the persist gate:
  `scan_output` flags zero-width / BIDI presence (`output:zero-width-present`,
  `output:bidi-present`) and `disposition` treats those plus
  `lexicon:unicode-tags-present` as any-tier carriers, so a carrier in model
  output fails secure even under a trusted policy.
- `contract` — `assert_credential_allowlist` catches a bare `<PROVIDER>_KEY`
  (e.g. `STRIPE_KEY`); the previous regex silently missed it (fail-open). The
  rule is deliberately broad (also flags `PARTITION_KEY`/`SORT_KEY` as loud,
  allowlistable false positives) — fail-loud beats fail-silent for isolation.
- `disposition` — `guard` runs `decide` inside its guarded block, so a malformed
  report can no longer escape the fail-closed guarantee.
- `output` — secret-egress placeholder suppression anchors word markers
  (`example`, `todo`, …) to a word boundary, so a real secret that merely
  *contains* such a word is no longer suppressed (fail-open egress miss closed).