1
0
Fork 0
llm-ingestion-pipeline-secu.../CHANGELOG.md
Kjell Tore Guttormsen 0eab4345bf
feat(output): decode hex-wrapped credentials at the persist gate
Gap 1 closed. The output gate finds runs of 32+ hex digits
(calibration.HEX_WRAP_MIN_LEN), decodes both byte alignments, keeps text
that is at least 80% printable (the base64 bar, now one shared helper in
entropy) and runs the credential patterns on it. A hex-wrapped AWS key
surfaces as decoded:egress:aws-access-key-id with "in hex run @ N" as
evidence; the value is never in it. Credentials only: a hex-wrapped
injection is not decoded.

Measured before and after on the same trees: all 3583 clean documents
(three FP populations and the 78 pinned OKF concepts) are identical line
for line under both presets (disposition, labels, severity, count,
offset); OKF import stays 47/31. They and 569 READMEs hold 2403 hex runs
of 32+ digits, none of which decodes to text. scan_output CPU 122.9 s
before, 123.4 s after; one solid hex run at the 1M cap 4.2 s -> 5.8 s.

14 new tests (10 red for the right reason first), 6 of 6 mutations red.
Matrix 146/146 + 5/5 gaps, suite 1274 passed, ReDoS sweep 169 patterns.
2026-11-01 17:06:44 +01:00

88 KiB
Raw Blame History

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Added — the output gate decodes hex-wrapped credentials (gap 1 closed)

A credential written out as hex used to pass the output gate: entropy decodes base64 blobs for the rescan, not hex. The gate now finds runs of 32 or more hex digits (16 bytes), decodes them, and runs the credential patterns on the text, so a hex-wrapped AWS key surfaces as decoded:egress:aws-access-key-id, the label a base64-wrapped one already gets. The evidence says in hex run @ <offset>, the offset is where the run starts, and the value is never in the evidence.

  • Both byte alignments of a run are decoded, so one stray hex digit before or after the credential (or an odd-length run) does not hide it. A 0x prefix needs nothing.
  • The decoded bytes must be at least 80% printable, the bar a base64 blob's plaintext already meets. The check is now one shared helper in entropy.
  • Credentials only. A hex-wrapped injection is not decoded (one wrapped in base64 is). Hex split by separators (41:4b, \x41), other encodings and nested wraps stay a documented boundary.
  • The floor is calibration.HEX_WRAP_MIN_LEN = 32.

Documented gap 1 is closed: the coverage matrix asserts it as caught (the hex-wrapped case, which checks the finding came from the hex run), and tests/test_gap_register.py records it.

If you gate on decoded:egress:*: a document with a hex-wrapped credential now gets that finding and the disposition that comes with it. Nothing else moved. Measured before and after on the same trees: every one of the 3,583 clean documents (the three false-positive populations and the 78 pinned OKF concepts) keeps its disposition under both presets and the same labels, severities, counts and offsets. Import of the pinned OKF corpus is unchanged at 47 fail-secure / 31 warn. Those documents and 569 package READMEs hold 2,403 hex runs of 32 or more digits (checksums and ids), and none of them decodes to text. scan_output over the 3,583 took 122.9 s of CPU before and 123.4 s after. At the 1,000,000-character cap, one solid hex run costs 5.8 s against 4.2 s before.

[1.9.0] — 2026-11-01

Changed — a lexicon finding counts every match (gap 4 closed)

scan_lexicon still gives one finding per pattern id, at the offset of the first match. Its count was always 1. It is now the number of matches, so a document that repeats "ignore all previous instructions" five times says so. Documented gap 4 (count=1) is closed: the coverage matrix asserts it as caught (lexicon:count), and tests/test_gap_register.py records it.

The lexicon matches several views of the text: raw, decoded, homoglyph-folded and rot13. One match shows in several views, so the views are not added up. count is the most matches any one view shows, which makes it a lower bound. A plain match plus a homoglyph one counts 2, and a plain match plus a rot13-hidden one counts 1. For unicode:zero-width-in-word, only the joiners the row reports are counted, not a required ZWNJ in Persian.

If you read count: lexicon findings can now carry count > 1. No label, severity, offset or disposition changed, because count never reaches the disposition. Measured before and after on the same trees: the three false-positive populations are identical line for line at 106/198 · 196/2918 · 108/389, and every one of their 3,505 documents and the 78 pinned OKF concepts keeps its disposition under both presets and the same labels and severities. Import of the pinned OKF corpus is unchanged at 47 fail-secure / 31 warn. 53 documents now carry a lexicon count above 1 (23 + 23 + 7 + 0), the number measured before building; the highest is 26. Scanning costs the same: scan_lexicon over the 3,505 documents of the three populations took 96.0 s before and 97.1 s after.

Documented — four gaps stand with a measured reason

Gaps 2, 3, 5 and 7 now each say, in docs/LIMITATIONS.md, which rule was tried and what it cost on 3,583 clean documents (the three populations and the 78 pinned OKF concepts), plus 569 package READMEs, measured with docs/gap-sweep.py on v1.8.0:

Gap Rule tried Cost
2 · factual poisoning a claim like "is made of" or "was completed in" fires on the true sentence exactly as on the false one; no word rule can see the class
3 · lone HIGH in trusted prose a lone HIGH escapes WARN under the trusted preset 48 new stops on the trusted gate, 67 of 569 READMEs
5 · pure beaconing every external image above LOW 23 new stops among 569 READMEs (the 3,583 hold no external image)
7 · task with no marker (A12) "gather it from private places" + "put it into this document" 0 new alarms, but catches 2 of 5 rewordings of the same request

Gaps 1 (hex-wrapped secret) and 6 (short opaque URL segment) are open and next.

Maintenance — the threat catalogs are watched

docs/threat-2026/watch-sources.json lists the 16 catalogs in the format of the edition watch from llm-security: 12 probed, 4 manual with a reason. tests/test_release_rule.py stops a release whose manual check or watch run is more than 30 days older than the release date, or whose watch output has a row that is behind, not verified or on watch. Run on 2026-11-01: 16 current. Nothing in the installed package changed for this.

[1.8.0] — 2026-10-31

Added — instructions in plain prose (attacks A08, A09, A16, A19)

Four attacks put their instruction in ordinary English, with no markup, no override phrase and no hidden carrier, so they gave 0 findings: a persistence order in a session summary (Unit 42, A08), a memory write on a later user turn (Rehberger, A09), a standing order in a calendar invite (Invitation Is All You Need, A16) and spoiler-hidden page text against a browsing agent (Brave on Comet, A19).

Finding Severity What it reports
indirect:prose-instruction MEDIUM a paragraph that combines at least two (calibration.PROSE_INSTRUCTION_MIN_CLASSES) of four classes: it speaks to the model (address), it is meant to act later (deferral), it writes to the model's memory (memory), it hides from the user (concealment)

Address is "Assistant, ...", "instructions for the assistant" (the plural, which the vendored commons row indirect:instruction-to-ai misses) or an all-caps pseudo-tag such as <INSTRUCTIONS>. Deferral is "from now on", "in all future sessions" or "if the user later says". Concealment is "never mention this", "do not tell the user" or a >!spoiler!<. One class alone is ordinary prose: vendor docs say "Block from now on" and "Instructions for Claude". The paragraph is the unit, so an instruction split over sentences is still one, and two features a blank line apart are not. The rule lives in output.py; the vendored lexicon stays byte-equal with commons.

MEDIUM sends the paragraph to review, not to a block. A document that quotes a system prompt or specifies a memory feature is itself an instruction to a model and is reported too. What it does not catch, measured: one class alone, a paraphrase outside the word lists, the two classes a blank line apart, and the same instruction in another language. All are in docs/LIMITATIONS.md.

If you route on labels: a document with such a paragraph now goes to QUARANTINE_REVIEW under PRESET_USER_UPLOAD, not WARN.

Documented — a task in plain prose with no marker (attack A12)

Invariant's toxic agent flow asks, in a public issue, for the README of every one of the author's repositories, "including private ones", to be added to this README. It addresses no one, defers nothing, writes no memory and hides nothing, and a maintainer could file the same request. A rule that caught it would be fitted to the payload's words. It is the seventh documented gap in the coverage matrix and in docs/LIMITATIONS.md, and the attack case asserts that the gap still stands.

Fixed — the ReDoS sweep skipped patterns kept in a dict

docs/redos-sweep.py walked lists, tuples and dataclasses but not dicts, so the four class patterns were collected by nothing. It now walks dicts too. Only the output table changed: 168 patterns are swept, 0 flagged. A positive control (a pattern quadratic on whitespace, placed in a dict) is flagged at x4.0.

Measured against the guard rule, before and after: the false-positive populations are identical line for line at 106/198 · 196/2918 · 108/389, and the new label fires on none of their documents. The pinned OKF corpus is identical on import at 47 fail-secure / 31 warn, with 0 firings. Before building, the rule was measured on 5 494 benign documents: the three false-positive populations, 411 npm READMEs, 166 PyPI descriptions and 1 412 skills, agents and commands from two vendor-published plugin collections, which are instructions to an agent by design. 0 fire. With the whole document as the unit instead of the paragraph, 1 of 5 494 fires (a skill file). The coverage matrix is at 144/144 classes and 7/7 documented gaps, 151 cases. Test suite: 1233 passed, 0 expected failures (was 1179 + 5). Cost at the 1 000 000-character cap, measured on a heavily loaded machine: about 0.7 s of CPU in a 9 s scan when the text holds the hint words, and 0.01 s when it holds none (a keyword prefilter skips the patterns). Attack row 3: 17 caught, 2 documented gaps, 0 open, so every in-scope row is caught or documented.

[1.7.0] — 2026-10-31

Added — a run of images whose order carries a value (attack A05, CamoLeak)

CamoLeak had the assistant render a secret as a run of images, one per character, from a pre-built URL alphabet. Each URL on its own names a remote image and carries nothing, so each was graded LOW and the gate WARNed. The value is in which image comes where.

Finding Severity What it reports
active:markdown-image-sequence MEDIUM at least 4 ordinary external images (calibration.IMAGE_SEQUENCE_MIN_RUN), inline or reference-style, on one URL template whose varying tokens are 2 characters or fewer (IMAGE_SEQUENCE_SYMBOL_MAX) or digit runs

A run whose only varying parts are numbers that count up in document order (step-1.png, step-2.png, or ch1/fig2.png before ch2/fig1.png) is a counter and is not reported. Images on another template are not part of the run, so decoys do not dilute it. Reference-style images (![a][r], ![r][], ![r]) resolve through their definitions. Each image keeps its own active:markdown-image grade; the new finding sits beside it.

MEDIUM sends the run to review rather than to a block. The rule reads the order from the shape of the URLs, and that shape has benign twins: a table of country flags (/w20/no.png, /w20/se.png, …) or figures named 1a, 1b, 2a are reported too. What it does not catch, measured: a dictionary of words, symbols of 3 characters or more, a counter whose increments carry the value, a run of 3, and a run that changes template for every symbol. All are in docs/LIMITATIONS.md.

If you route on labels: a document with such a run now goes to QUARANTINE_REVIEW under PRESET_USER_UPLOAD, not WARN.

Measured against the guard rule, before and after: the false-positive populations are identical at 106/198 · 196/2918 · 108/389. Those three hold no external markdown image, so the rule was also run over 411 npm READMEs and 166 PyPI descriptions (312 with external images, inline or by reference, badge rows included): 0 firings, and 0 at every run length from 3 to 8. The pinned OKF corpus is identical on import at 47 fail-secure / 31 warn, with 0 firings. The coverage matrix is at 140/140 classes. The ReDoS sweep covers 163 patterns, 0 flagged. Cost at the 1 000 000-character cap on an image-only document: the rule adds about 0.5 s of CPU to a 4.6 s scan. Attack row 3: A05 is caught, leaving 13 caught, 1 documented gap and 5 open.

[1.6.0] — 2026-10-31

Added — hidden markup at the persist gate (attacks A04, A14, A15)

Three new findings. Each is caught by scan_output, so it fires on both the input-side scan and the persist gate.

Finding Severity What it reports
html-obfuscation:tiny-font HIGH an element styled with a font below 4px (calibration.HIDDEN_FONT_MAX_PX). ShadowLeak's font-size:1px passed with 0 findings, because the vendored commons row zero-font only matches a size written with a leading 0
html-obfuscation:invisible-colour HIGH text coloured like the background declared in the same style attribute (within 8 per channel), or transparent
output:html-comment-present MEDIUM an HTML comment in the artifact outside fenced code and inline code. The input side already stripped comments; the output side never looked, so a CopyPasta comment the model carried along passed

White text on its own is deliberately not flagged, because HTML email puts white text on dark buttons. A comment inside code is not flagged, because a renderer shows it as code. Both residuals are in docs/LIMITATIONS.md.

If you route on labels: documents with hidden styling or a hidden comment now get findings they did not get before. Under PRESET_USER_UPLOAD that is QUARANTINE_REVIEW or FAIL_SECURE, not WARN.

Measured against the guard rule, before and after: the false-positive populations are identical at 106/198 · 196/2918 · 108/389. The new labels fire once in total, on a red-team document that was already non-WARN. The pinned OKF corpus is identical at 6/53 parsed and 47/31 on import. The coverage matrix is at 139/139 classes. The ReDoS sweep covers 161 patterns, 0 flagged. Attack row 3: A04 and A15 are caught, leaving 12 caught, 1 documented gap and 6 open.

Documentation — every cited catalog on its newest edition, dated and linked

No code behaviour changed; one code comment was corrected.

  • README maps the persisted store to OWASP ASI06 and MITRE ATLAS v2026.10. A new subsection pairs ASI06 (Memory & Context Poisoning, OWASP Top 10 for Agentic Applications for 2026) and seven ATLAS techniques with what the guard stops at the write and what it does not. Each technique was read in the publisher's v2026.10 data file. ATLAS released v2026.10 on 2026-10-28, after the register had recorded v2026.09. AML.T0057 (LLM Data Leakage) is gone from v2026.10, so it is not linked.
  • Catalog register 1.1.0 (docs/threat-2026/catalogs.json): every entry now records the edition the repo cites next to the newest one, re-checked at the publisher on 2026-10-31. CleanBase (arXiv 2605.00460) was cited but unregistered, so it is added: 15 → 16 entries. Sources that cannot be read at their publisher are listed as candidates and not counted. A new test, tests/test_catalog_register.py, fails when an entry lacks a check date or a link, or cites an older edition than the latest. It also fails when the README names an ATLAS or Agentic entry the register has not verified. It never fails on age.
  • Unicode 17.0 → 18.0 in the ZWJ measurement. Re-measured on 18.0: 1614 RGI sequences, 122/122 neighbouring codepoints covered. The sanitize.py comment's "458 unassigned" came from Python's Unicode 16.0 tables; the correct figure for 18.0 is 406.
  • BRIEF no longer dates "AI Agent Traps" to 2025: the publisher (SSRN) could not be read, so the year is not stated.

[1.5.0] — 2026-10-31

Changed — every OWASP number carries the 2026 edition (filters on the old value lose hits)

OWASP published the Top 10 for LLM Applications 2026 on 2026-08-04 with a new order. Only LLM01 and LLM02 kept their meaning. Every Finding.owasp up to 1.4.2 was written against 2025, without a year, so it now reads as a different risk. EchoLeak findings said LLM05, which in 2026 means Data and Model Poisoning. From 1.5.0 every number is written the way OWASP writes it, with the year:

Finding (examples) ≤ 1.4.2 1.5.0
injection lexicon, carriers, entropy, fence, decoded injection LLM01 LLM01:2026
egress:*, decoded:egress:* (secret egress) LLM02 LLM02:2026
active:*, neutralize (Improper Output Handling) LLM05 LLM10:2026
*:oversize-input, scan caps (Unbounded Consumption) LLM10 LLM06:2026

If you filter, route or alert on owasp, update the value. A filter on "LLM05" matches nothing from 1.5.0, and a filter on "LLM10" now means a different risk. Finding.label, severities and dispositions are unchanged, so no detection behaviour moved. Only the catalog field changed. The exported surface is unchanged: owasp was, and is, an optional string.

The number now comes from one table keyed by risk name. The vendored lexicon stays byte-equal with llm-security-commons and gets its year where the finding is made. A commons number whose meaning moved between editions raises instead of being relabelled by guesswork.

Coverage matrix. The rows map the same way. 14 rows whose class no edition describes now carry no number (Case.owasp is None, printed -):

  • the un-scannable artifact failing closed (the guard's own failure semantics);
  • four OKF frontmatter parser-surface refusals;
  • resource URL and cross-link scheme validation of received metadata;
  • the five OKF path rows, which carried LLM01 by default.

The factual-poisoning gap (PoisonedRAG) moves from LLM09 (Misinformation) to LLM05:2026, whose example 6 is RAG knowledge-base poisoning. 136/136 classes and 6/6 gaps are unchanged.

Held at zero. tests/test_catalog_numbers.py counts every number in the tracked files with docs/catalog-census.py. It fails on an older or wrong meaning, and on a missing year anywhere except the vendored lexicon. It also records every finding the coverage matrix drives and fails on one without the newest year. The newest edition is read from docs/threat-2026/catalogs.json. Row 2 of the 2026 plan went from 95 sites to 0 (frozen denominator at 6702d7a, unchanged).

[1.4.2] — 2026-10-29

Fixed — invisible-character smuggling passed with 0 findings

Two published 2025 techniques reached the model through every gate untouched. Variation-selector smuggling (Butler, 2025-02; Gao et al., 2025-10) hangs one selector per hidden byte off a visible character; Sneaky Bits (Rehberger, 2025-03) encodes bits in U+2062 and U+2064. On 1.4.1 both gave 0 findings on the input path and on the output gate, and every hidden codepoint stayed in the "sanitized" text. Measured more broadly in session 1: 17 of 19 invisible character classes gave 0 findings.

The fix starts from a principled denominator instead of a hand-picked list: Default_Ignorable_Code_Point | format characters (Cf) | private use (Co) in Unicode 18.0, 141 674 codepoints, frozen with the source files' sha256 in docs/threat-2026/codepoints.json. The guard knew 142 of them. Every one is now stripped by sanitize and flagged by the output gate unless a known legitimate use exempts it on its context. One predicate serves both surfaces, and a test checks all 141 674 codepoints, one by one, on both.

The exemptions are those the ZWJ fix established: judge by context, never by identity alone. A run is a carrier whatever it stands after.

  • one VS15/VS16 after an emoji base (371/371 of Unicode 18.0's emoji variation sequences): ⚠️ ⏱️ ❤️ 1️⃣ survive byte-identical;
  • one ideographic selector after a CJK ideograph (registered name variants, and the 1 002 CJK compatibility sequences);
  • ZWNJ/ZWJ next to an Arabic-script or Indic letter, with neither neighbour invisible. Persian, Urdu, Hindi and Malayalam orthography stops being an any-tier carrier. That closes the ZWNJ false-positive class 0.6.1 parked for want of a corpus to verify a rule against. Measured on 9 Wikipedia articles (fa ×3, hi ×2, ml, ur, he, ar), articles with a carrier finding went from 6/9 to 2/9, and the two left are real (a ZWSP where ZWNJ belongs; two ZWNJ in a row);
  • LRM/RLM/ALM in a document that carries right-to-left text;
  • an Arabic number sign (U+0600–0605 and kin) directly before a digit.

Private use is caught in full: an icon-font glyph is blocked rather than leaving a ~17-bit channel per visible gap. That decision, the number-sign rule and the one-selector-per-ideograph rule are the operator's (2026-10-28). What each exemption leaves open, and what is now blocked despite a legitimate use, is in LIMITATIONS.

The lexicon's word-split row (unicode:zero-width-in-word, \w[ZW]\w) matched every Persian word that carries its required ZWNJ. It now judges the joiner it matched by the same predicate. The regex itself is unchanged and stays byte-identical with the commons lexicon, and a Latin word split still fires.

Added — labels

sanitize:variation-selector, sanitize:private-use, sanitize:invisible-format and their output-gate counterparts output:variation-selector-present, output:private-use-present and output:invisible-format-present, plus output:unicode-tag-present. That last one is the complete tag check on the output side, because the lexicon's narrower one misses U+E0000. All seven are any-tier carriers (FAIL_SECURE in every tier), as the zero-width and BIDI labels were. A consumer filtering on the old labels sees new ones. Labels are detection behaviour, not the frozen surface.

Measured — the guard rule, before and after

  • Attack cases: A01 and A02 are green on both arrivals. Row 3 of the threat-2026 plan stands at 10 caught, 1 documented gap and 8 red rows (8 of 31 arrivals, down from 12).
  • Test suite: 1049 passed, 8 strict xfail (was 935 + 12).
  • Coverage matrix: 136/136 caught classes (was 130/130; six carrier cases added); 6/6 documented gaps hold.
  • Pinned OKF corpus (_okf-upstream @ 3fcbb9f): unmoved, 6/53 parse and 47 fail_secure / 31 warn on import, identical before and after.
  • False-positive populations (docs/fp-sweep.py, upload preset, documents disposed non-WARN): unmoved — vendor-harvest 106/198, generated-notes 196/2918, reference-corpus 108/389, identical line for line before and after. Positive control: doubling the VS16 that a reference-corpus document already carries moves it from quarantine_review to fail_secure.
  • Runtime: the classes are located with one regex pass and judged per hit. A 1 000 000-character document made entirely of carriers costs 1.4 s in sanitize, and a clean one 0.05 s. The RTL context is looked up once per document, so a document of directional marks stays linear (pinned). The ReDoS sweep collects 154 patterns (the two new character classes included) and flags neither.

[1.4.1] — 2026-09-23

Fixed — a flow sequence refused # and : where YAML reads them as text

1.4.0 refused a flow-sequence scalar element carrying # or : anywhere. okf writes references: [...] as a flow sequence of doc links, and an entry may carry a #fragment or a scheme://. A consumer measured the cost (claude-code-llm-wiki, 2026-09-17): okf.parse_frontmatter raised on 2 038 of 5 467 concepts of a bundle okf itself produced, so okf.import_bundle returned a parse error instead of a verdict for 37 % of it. Reproduced here on the same bundle before the fix (2 038 / 5 467); after it, 0 / 5 467 (and 0 / 4 466 and 0 / 5 530 on the consumer's two other builds).

The two characters are now refused by position, where YAML gives them a meaning, ground-truthed against PyYAML 6.0.3 on every test row: a # that opens the element or follows whitespace opens a comment, and a : that opens or ends the element, or precedes whitespace, opens a mapping. Both stay refused there. Elsewhere they are scalar characters, so a#b, x://y#z, claude-cli://open and https://e.com:8443/a parse as the strings YAML reads. The rest of the refused set ({ } [ ] , " '), the scalar-indicator rule and the no-mixing rule are unchanged.

The pinned upstream corpus is unmoved at 6 / 53 (_okf-upstream @ 3fcbb9f); its binding constraint is the top-level block mapping, not this rule. The predicate stays linear: CPU time doubles with each doubling of element count and element length (100k → 800k), about 10 % over 1.4.0.

Changed — documentation surfaces caught up with 1.3.0 and 1.4.0

README, ADOPTION-BRIEF and BRIEF named 1.3.0 after the 1.4.0 tag. README also said resource is not on the frontmatter allowlist and that a sources block list of mappings is refused (both false since 1.3.0), and described the resource https-allowlist without saying it covers the top-level resource only — sources[].resource is never URL-validated. LIMITATIONS listed tags: [a, b] among the routes that fail (false since 1.4.0). All corrected.

[1.4.0] — 2026-09-08

Added — tags: [a, b, c], the one flow-sequence form SPEC §4.1 writes out

okf.parse_frontmatter refused a flow sequence of plain scalars. That is the form SPEC.md §4.1's own frontmatter skeleton uses for tags, and tags is present in 53/53 upstream reference concepts, 9/53 of them in exactly this form. Admitted now, parsing to the same value its block-sequence sibling already did:

tags: [finance, revenue, headline-metric]

An element is a plain scalar only if it is non-empty and carries none of { } [ ] : , " ' #, and it then passes the unchanged scalar-indicator rule. The refusals that make that a rule rather than a preference: a quoted element (this parser retains quotes rather than stripping them, so reading one would hand back a value YAML does not), an element carrying a colon or a comma, a sequence inside a sequence, an empty element, an anchor or an alias. A sequence may not mix scalars and mappings — the same rule the block list already carries, for the same reason: a consumer iterating the value and reading entry.get("id") crashes on the first str. The sources flow-mapping carrier added in 1.3.0 is unchanged and pinned against regression.

Measured, with the denominator: this takes the pinned OKF corpus from 0/53 to 6/53 (_okf-upstream/okf @ 3fcbb9f, 53 non-reserved documents; the six are all in acme_retail). It does not close the corpus. 44/53 still stop on generated written as a top-level block mapping — a form refused on the no-nesting-past-depth-1 rule, which is a security decision and out of scope here. Two neighbouring predicates were measured and deliberately not built: a flush-left block sequence and a folded plain scalar release 0/53 each on their own and leave the corpus at 6/53 when stacked on this one. Detection behaviour is not frozen under semver; the exported surface is unchanged.

Changed

  • docs/LIMITATIONS.md's tags/description entry is rewritten against the measurement. Three of its claims were wrong: the parser does have a sequence-value type (since 1.3.0 — what it lacks is the indentation the corpus omits), the figure is 6/53 and not the 1.2.0-era 4/53, and tags/description are not the residual that blocks the corpus.
  • Self-safety: the new predicate is character-level with no regex, so docs/redos-sweep.py cannot see it (it collects compiled patterns). It is measured instead — linear in both element length (exponent 0.86-0.99) and element count (0.97-1.05) over four doublings to 800_000, and pinned by two CPU-clock bounds in tests/test_okf.py.

[1.3.0] — 2026-09-02

Added — the sources provenance family becomes expressible in both spec carriers

okf.parse_frontmatter rejected sources in every form the OKF spec and its producers actually use. Measured 2026-09-02 by two consumers independently: sources: [{ id: a, resource: x }] raised on the [ indicator (with one entry as with two), and the block sequence of block mappings — SPEC.md §5.1's own example — raised "nested mappings are not supported". resource is REQUIRED within a sources entry (§5.1), so the whole provenance family was unwritable, and a bundle written the way the spec documents it was refused.

Admitted now, both parsing to the same value (a list of dicts):

sources: [{ id: a, resource: https://e.com/a }]
sources:
  - id: a
    resource: https://e.com/a

resource is allowlisted inside a sources entry and nowhere else. 1.2.0 left it off the allowlist on the argument that the parser could not tell sources[].resource (§5.1, a citation) from executor.resource / attester.resource (§10, a pointer to code to be run — the door-C route closed in 1.1.0). That premise was false: the owning key is in scope at every call site and was simply never threaded through. It is threaded now, so the discrimination is structural rather than a judgement about the value, and executor: [{ resource: skills/run.md }] and attester:\n - resource: … are refused on the allowlist through every carrier, including the two this adds.

Refusal stays the default everywhere else. A flow sequence of plain scalars (tags: [a, b]) still raises — the sequence carrier is opened for the flow mapping element and nothing else. A sources entry still admits scalar leaves only, so SPEC §5.1's optional per-entry usage_window (a mapping inside a mapping) is refused: no nesting past depth 1 is a security property, and it was not spent here. Registered as a conformance gap in docs/LIMITATIONS.md. A block list may not mix scalar items and mappings. Off-allowlist keys, anchors, aliases, tags, duplicate keys and unclosed collections raise as before, and a refused mapping still raises rather than degrading into a string.

New residual, registered: sources[].resource is scanned as text (T1) but never validated as a URL. T3's https allowlist cannot be extended to it without over-blocking conformant bundles — §5.1 permits a bundle-relative path or a scope descriptor a consumer cannot follow at all. A consumer that dereferences it must call okf.validate_resource_url itself.

No exported surface changed and no detector behaviour or calibration changed. Suite 834 → 868 (34 new rows, plus four pre-existing rows updated where this release changed the behaviour they pinned); 130/130 classes, 6/6 documented gaps hold, 45 limitations, ReDoS sweep 0/152 candidates flagged.

[1.2.0] — 2026-08-23

Added — OKF frontmatter can express one mapping form: typed and allowlisted

okf.parse_frontmatter gave the mapping class no expressible form at all. OKF v0.2 writes its whole trust and provenance layer as mappings — SPEC.md @ 62432a09 uses flow form in its own §5.1/§5.2 examples, and §11 carries a hard MUST for consumers ("MUST treat a bare verified mapping as a one-element list") that presupposes they parse. A consumer measured 0 of 53 upstream concepts through the gate on 0.3.4, 1.0.0 and 1.1.0. That was a contract collision, not a calibration setting: no threshold would have moved it.

Admitted now: a flow mapping (generated: { by: x, at: y }), as a value or as a block-list item, whose every key is on a nine-name allowlist (by, at, from, to, id, title, author, usage_count, last_modified) and whose every leaf is a plain scalar run through the unchanged dangerous-value and mapping-construct predicates.

The form is additive and refusal stays the default. A key off the allowlist, a nested collection, a quoted leaf, a duplicate key, an empty or unclosed mapping, and {a:b} (which PyYAML 6.0.3 reads as the key a:b) all raise, and a refused mapping still raises rather than degrading into a string — the 1.1.0 defect is not reopened. Nested-block (k:\n sub: v), dotted (k.sub: v) and inline-second-colon (k: sub: v) routes to a mapping still raise, each on its own rule.

resource is deliberately off the allowlist, though SPEC.md §5.1 names it inside a sources entry. It is a pointer rather than a label and the only key T3 exists for: admitting it would let executor: { resource: skills/run.md } carry an executable-code pointer through in typed clothes, which is the door-C route closed in 1.1.0. It costs nothing today — the conformant carrier for sources[].resource is a block sequence of block mappings, which this form does not admit either way.

Mapping leaves are scanned like every other frontmatter value (T1), so an injection parked in generated: { by: ... } reaches scan_output. Coverage matrix: 130 classes, up from 129 (the new row is the off-allowlist key).

No exported surface changed; no detector behaviour and no calibration changed.

Changed — the ReDoS sweep now measures on the same clock as the bounds it justifies

docs/redos-sweep.py timed on time.monotonic() while every ReDoS bound in the suite moved to process CPU time (tests/redos_clock.py), so the 1.5 ms sensitivity floor and the "~23 s at the cap" figure published in docs/LIMITATIONS.md came from a different instrument than the bounds they support. The script now imports scan_seconds rather than timing itself.

The floor was re-derived on that instrument and stayed at 1.5 ms: over twelve full runs of all 2585 arms the median ratio is 1.95-2.03 in every size bucket above 50 µs, but two-point excursions past the 2.6 flag threshold persist at every magnitude (p99 ratio 2.9-3.3 even above 1 ms) — 6.9 flagged arms per run at a 0.5 ms floor, 1.1 at 1.0 ms, 0.33 at 1.5 ms. Descheduling was never what made this sweep noisy; a ratio computed from two points is. Four arms flagged across those twelve runs, each in exactly one of them, and six arms that have ever flagged re-measure at exponent 0.97-1.09 over six doublings — at most 1.2 s at the 1 000 000-char cap. The pattern count the script prints is 152, not the 150 of the 0.3.4 entry below; docs/LIMITATIONS.md now carries the current number.

No exported surface, no detector behaviour and no calibration changed.

[1.1.0] — 2026-08-13

Fixed — a mapping construct in OKF frontmatter no longer degrades into a string

okf.parse_frontmatter gives the mapping class no expressible form by design (T2). Two routes escaped that: they parsed "successfully" into the wrong type instead of raising. Both are closed, and both now FAIL_SECURE through okf.import_bundle (door C).

route was now
sources:
- uri: https://e.com/a
string 'uri: https://e.com/a' — WARN OKFFrontmatterError — FAIL_SECURE
sources:
- uri:
string 'uri:' — WARN OKFFrontmatterError — FAIL_SECURE
attester: resource: attesters/x.py string 'resource: attesters/x.py' — WARN OKFFrontmatterError — FAIL_SECURE

The security consequence was the same in each: a pointer parked in a degraded mapping rides through in a key the resource allowlist never inspects, and mode-b import_bundle writes the merged concept verbatim. The first route was documented at docs/LIMITATIONS.md:43; the inline second colon was found by measurement while closing it, and is the reason this release names two routes rather than one. Neither shape is conformant OKF — a well-formed bundle does not produce them; a malformed or hostile one can.

What closed is the type confusion, not pointer-smuggling as a class. T3 still inspects resource and nothing else, so an honest string under another key rides through exactly as before: attester: attesters/sql_equality.py is WARN, while the same path under resource: FAIL_SECUREs. The string is still scanned like any other frontmatter value under T1. Nothing about that changed here.

The boundary is where YAML puts it, ground-truthed against PyYAML 6.0.3 rather than reasoned: ": " and a trailing ":" are exactly the two shapes where a plain scalar becomes a mapping, and they are refused. A colon carrying neither a space nor a line end opens no mapping — domain:security and https://e.com:8443/a still parse — and a quoted scalar (- "uri: x") is still a scalar. Quotes are retained rather than stripped; that divergence from YAML is unchanged and now pinned.

This is a behaviour change inside the freeze, not a break of it. No exported name moved. A document that disposed WARN on 1.0.0 may dispose FAIL_SECURE here — the 1.0.0 entry says exactly this is a fix, not a break. A consumer whose bundles carry an unquoted ": " in a frontmatter value will now see those concepts refused at import; quote the value, and it parses.

Suite 792 → 802: 13 rows added (4 rejected shapes, 7 admitted ones, 2 through import_bundle), 3 retired (the two that pinned the defect, and the one-key row in the block-list table). 129/129 classes, 6/6 documented gaps, 35 limitations — all unchanged.

[1.0.0] — 2026-08-13

Changed — the exported Python surface is frozen under semver

No code changed in this release. 1.0.0 is a governance promise, not a claim that the library is finished: no name exported from llm_ingestion_guard is removed, renamed or given a different meaning without a 2.0.0. Measured before the tag, the surface has been stable in form since 0.3.4 — four names added, none removed or renamed — while behaviour moved across five releases (0.4.0 … 0.7.0).

Detection behaviour is deliberately outside the freeze. Severities, thresholds, lexicon entries and the dispositions they produce are calibration, and calibration moves in minor and patch releases. A payload that disposes WARN here may dispose FAIL_SECURE in a later 1.x; that is a fix, not a break. Assert on the disposition your policy requires, not on a severity you observed.

The behaviour changes this freeze rests on are not repeated here — see [0.3.0] for the active-content gate and the OKF adapter, and [0.3.1] for the ordinary-link/image calibration that the two consumer promises pin.

Changed — three limitations are conceded for 1.x rather than deferred

docs/LIMITATIONS.md no longer says "deferred" or "pending" about any of them:

  • Severity still carries disposition intent on the detection side. Separating what was seen from how bad it is changes Finding and Severity, so it is a 2.0.0 change. Read a finding's id for the capability.
  • The input-cap asymmetry at MAX_INPUT_CHARS is permanent in 1.x: surfaces that return content raise OversizeInputError, surfaces that return findings truncate and emit active:oversize-input.
  • The multilingual homoglyph false positive is conceded more narrowly — no fix is promised, but it is calibration, so one may land in any 1.x release.

SECURITY.md carries all three as documented boundaries and states the support window for a 1.x line.

Known at the freeze, deliberately not blocking it

docs/LIMITATIONS.md §:43 — an OKF block sequence with exactly one key per element misparses silently in okf.import_bundle, so a pointer can ride through in a key the resource allowlist never inspects. Closing it tightens what the adapter admits: behaviour, not form, and shippable in a 1.x minor. It is recorded here because "we knew, and froze first" is a defensible position and "we forgot" is not.

Runtime coverage at the freeze: llm-ingestion-okf has measured 0.3.4 and run a 0.3.4→0.6.1 differential on its own door across two Python versions; llm-security-commons differentially tested its independent reconstruction of the raw-HTML classifier against ours over 42 probe tags with 0 disagreements. No external consumer has run the 0.7.0 runtime; the four symbols added since 0.3.4 are additive, so a caller that does not invoke them is unaffected.

[0.7.0] — 2026-08-13

Raw HTML graded on activity alone: every active tag was HIGH. So the same URL was LOW as [t](https://example.com/guide) and HIGH as <a href="https://example.com/guide"> — an asymmetry produced by syntax, not by affordance. Following an anchor needs a human, exactly like the markdown inline link that has been MEDIUM since 0.3.1.

<a> and <area> now report as active:raw-html-link at MEDIUM. Everything a renderer fetches or executes unattended keeps active:raw-html at HIGH, and the event-handler test runs first, so <a onclick=...> is graded as the execute-class carrier it is rather than downgraded with the anchors.

The URL-attribute branch deliberately stays on the HIGH side: a name outside the active set has unknown rendering, and href is not the only URL attribute it may carry. Grading <Card src="..."> as a link would be reasoning, not measurement.

This is a new label, and labels are a contract surface consumers pin against. A document that previously produced one active:raw-html finding may now produce two findings, one per carrier class.

Changed — a tag whose whole affordance is a URL it does not carry is inert

</a>, <Frame>, <video /> and <img alt="..."> without src were active by name while naming no target at all. This is <base />'s argument from 0.6.0 — "attribute-less, therefore no affordance in any renderer" — applied to the rest of the name branch. The test is for the URL attribute's presence, not for a readable value: a value the parser cannot resolve keeps the tag active, mirroring the fail-secure gap _url_attr_is_external already leaves open.

Every other member of the active name set does something a URL cannot describe — <script> executes its body, <style> restyles, <form> submits — and stays active with no attributes at all.

Changed — active_tag_class is the classification point; is_active_tag wraps it

docs/rawhtml-census.py measures candidates by patching this symbol, and a boolean could only express a narrowing, never a regrade. Left as a boolean, every carrier candidate would have measured equal to PRODUCTION — silently, and in the direction that reads as "no change helps".

Measured

docs/rawhtml-census.py, three populations, each at one corpus state and each against its own denominator — the two wiki corpora share content and are never summed. Documents that stop being fail_secure under PRESET_USER_UPLOAD, from 0.6.0 as shipped to 0.7.0, with the ceiling being the raw-HTML detector switched off entirely:

population documents 0.6.0 → 0.7.0 ceiling share of achievable
reference-corpus 389 54 → 53 53 1 of 1
vendor-harvest 187 62 → 20 18 42 of 44 (95%)
generated-notes 552 59 → 15 13 44 of 46 (96%)

Neither change alone is worth shipping, and the census is why they went out together. Alone, the split frees 8 documents in each wiki corpus and the narrowing 21 and 23 — but 8+21 measures 42 and 8+23 measures 44. The residual is 13 documents in both corpora: the narrowing strips a document's </a> and <Frame>, and what is left is the <a href=...> the split grades down, so each change alone leaves the document blocked by the other's residue.

Tightening, measured: 0 documents on both trust tiers, in all three populations. That zero is empirical and thinner than it looks — the split alone tightens 13 documents on the trusted tier in vendor-harvest and 14 in generated-notes, and the narrowing cancels each one. See docs/LIMITATIONS.md for why it must not be read as "cannot happen".

The PRODUCTION (as shipped) row matched C1 + D (0.7.0) field for field in every population, which is the check that the census and the shipped predicate have not drifted apart.

Known behaviour change

count drops on documents containing </a>. Through 0.6.1 an end tag was active by name, so count ran roughly 1.6× the opening-tag total and a start/end pair counted 2. It is now the opening-tag total. The field's meaning did not change and the finding count is unaffected — the class still collapses to one finding per class per document.

[0.6.1] — 2026-08-11

Fixed — the zero-width check tested identity, so emoji-composed documents were hard-blocked

_ZERO_WIDTH (sanitize, input) and _ZERO_WIDTH_CPS (output, _scan_invisible_carriers) tested U+200D on codepoint membership alone. disposition._CARRIER_LABELS grades a carrier as any-tier fail_secure with no appeal, so any first-party document containing a ZWJ-composed emoji — professions, families, skin tones, flag variants — was blocked permanently, with no preset able to release it. Reported by ms-ai-architect, confirmed here against the code.

The worse half was not in the report: sanitize removed the joiner, silently decomposing one emoji into two unrelated ones. A module whose published contract is "only ever removes carriers" was corrupting content.

The fix is the shape our own lexicon row unicode:zero-width-in-word (\w[ZW]\w) already used — judge the joiner by context, not identity. A ZWJ is exempt only when both neighbours are emoji-context codepoints. Half-context is not context, so a<ZWJ>😀 stays a carrier and an attacker cannot buy exemption with a single trailing emoji.

Blocks, not an emoji table. Measured against Unicode 17.0's emoji-zwj-sequences.txt: 1614 RGI sequences use 122 distinct codepoints adjacent to a ZWJ, and five block ranges cover 122/122. The measurement earned its keep — the hand-reasoned candidate table missed U+2194, U+2195 and U+2B1B. Shipping the RGI list itself would be exact the day it landed and stale at the next Unicode release, reopening this false positive for every new emoji; whole blocks carry the unassigned headroom (458 Cn codepoints) that future emoji are allocated into, so the table does not age. The predicate is defined once in sanitize and imported by output; a cross-surface test asserts the two halves agree on six inputs.

Known behaviour change

Documents whose only finding was an emoji-context ZWJ now persist unattended. On PRESET_USER_UPLOAD they move from fail_secure to WARN. Measured across the three false-positive populations at one corpus state: 1 document of 1126 (reference-corpus 1/389, vendor-harvest 0/187, generated-notes 0/550). This is a loosening of the upload door, not of detection — recall is unchanged at 128/128 demonstrated classes with 6/6 documented gaps holding, and a ZWJ anywhere else, including between a word character and an emoji, is graded exactly as before.

Patch, not minor. 0.6.0 called itself minor for loosening the same door, but that was a policy choice — <base> left the active name set by decision. This one restores a contract the module already published, against a class that was never meant to be blocked. A fix whose observable effect is the point of the fix is what the patch level is for.

Residuals — both documented (docs/LIMITATIONS.md, 33 → 34 items)

  • A ZWJ between two emoji is now exempt, so it can carry a narrowband covert channel: one emoji per bit, and it cannot split a word. A deliberate narrowing, stated rather than hidden.
  • U+200C (ZWNJ) still has no context test. It is orthographically required in Persian, Arabic and Devanagari, so those documents stay hard-blocked. The criterion has to be script-based rather than pictographic, and no corpus is here to verify one against — parked as a known false-positive class rather than guessed at.

736 tests pass (was 727).

[0.6.0] — 2026-08-11

Changed — active:raw-html stops firing on two things that carry no affordance

is_active_tag had two over-reaching branches, both measured on consumer corpora rather than argued from the code:

  • The URL-attribute branch was a presence test. Any element carrying href=, src=, action= … graded HIGH regardless of where the URL pointed. An MDX <Card href="/en/agent-sdk/quickstart"> — an internal doc route — reaches no attacker-controlled host, and neither does Azure APIM policy XML's <set-header>. The branch now requires an external target (absolute scheme or protocol-relative), the rule the markdown paths have applied since 0.3.1.
  • <base> left the active name set. HTML's <base> has its entire affordance in its href, which the URL-attribute branch still catches. APIM's attribute-less <base /> means "run the inherited policy" and is inert in every renderer.

Measured before and after in one session, against one corpus state — the two wiki corpora are living, so a before/after split across sessions would mix this change with re-harvest drift:

population n before after ceiling (raw-HTML off)
reference-corpus 389 133 108 107
vendor-harvest 187 100 98 62
generated-notes 550 90 88 49

96% of the achievable reduction in reference-corpus, 5% in the two wiki corpora: the over-reach was nearly the whole raw-html cost in APIM policy XML and nearly none of it in vendor documentation, where what remains is real HTML — <a> 298, <frame> 94, <img> 63 — caught correctly by the name branch.

The two classes had to be measured together. Alone they free 3 and 13 documents in reference-corpus; together, 25. A document carrying one usually carries the other, so closing either alone leaves it blocked by its twin. docs/rawhtml-census.py now carries a PRODUCTION row that re-measures the shipped predicate instead of a hypothetical, so a doc number and the code cannot drift apart unnoticed.

Known behaviour change

Documents whose only finding was one of these two classes now persist unattended. On PRESET_USER_UPLOAD they move from fail_secure / quarantine_review to WARN — 25 documents in the reference corpus, 2 in each wiki corpus. This is a deliberate loosening of the upload door, not of detection: recall is unchanged at 128/128 demonstrated classes with 6/6 documented gaps holding, and a tag that is active by name, carries an on*= handler, or points anywhere external is graded exactly as before. An element outside the active name set whose only URL attribute is doc-relative is the whole of what changed.

Fixed — the scanner and the mutator no longer share one predicate

neutralize imported is_active_tag from active_content by name, so narrowing the scanner would have silently narrowed the opt-in mutator as well — and no test in the suite discriminated the two halves: every neutralize:raw-html payload stayed active under each narrowing considered. The predicates are now separate symbols, is_active_tag (scanner, external-target rule) and is_defangable_tag (mutator, unchanged broad behaviour), and the mutator half is pinned by its own test. Over- defanging costs nothing there — neutralize is opt-in and blocks no disposition — while under-defanging would hand a human a live construct.

Self-safety (OWASP LLM10)

Reading a URL attribute's value needs a pattern the presence test does not provide. It reuses the same literal alternation with the value attached, so no new run shape enters the table, and _REDOS_PAYLOADS gains a row (active-url-attr-value) whose unit denies the = the pattern requires — a unit supplying it matches at once and never exercises the run. Measured at 100_000 chars: 0.031–0.046s across five attack shapes, against the suite's 2.0s bound. A gap between the two patterns fails secure: an attribute seen by the presence test but unreadable by the value parser counts as external, so it over-blocks rather than under-blocks.

727 tests pass (was 717).

[0.5.0] — 2026-08-11

Added — the axis separation: assessment (Risk) vs action (Disposition)

Additive, and measured to be so. Every disposition 0.4.0 rendered is rendered identically: the full suite went 703 → 715 with no test changed, the coverage matrix holds at 128/128 recall with 6/6 documented gaps, and the PRESET_USER_UPLOAD grading table locked in 0.3.1 was re-measured row by row and is unchanged. A caller that never reads the new field sees no difference.

decide and guard returned a Disposition — WARN / QUARANTINE_REVIEW / FAIL_SECURE — which names an action. But BRIEF design principle 4 says the library reports and the pipeline decides, and disposition.py admitted the gap in its own docstring: "It imposes no blocking of its own." So the library returned an action it cannot enforce, while discarding the judgement that produced it. A consumer wanting different behaviour had to reinterpret the action itself, which is why a consumer ends up pinning our grading — the action was all they got.

  • Risk — the new assessment axis: NONE / LOW / ELEVATED / SEVERE. It answers how dangerous is this artifact given its source context, and is trust-aware exactly as BRIEF §4.7 describes the domain: the same finding genuinely is a different judgement in authored prose than in a code fence.
  • DispositionResult.assessment — carries that judgement alongside the action. The field is required, with no default: Risk.NONE would be the natural-looking default and is the wrong one, since a construction site that forgot it would report clean and the axis would fail open.
  • Policy.action_map — an optional Risk -> Disposition override, so "hold for review where you would block" is a policy statement rather than a reason to pin our grading. Defaults to None, which means DEFAULT_ACTION_MAP and keeps an untouched Policy hashable as before. A partial map falls back per-level instead of raising.
  • Both overlays — compound escalation and the quarantine floor — now move the assessment, so a custom action map cannot silently drop them.
  • The fail-closed path in guard pins both axes to their most severe value and deliberately does not route through the action map: a policy that downgrades SEVERE means "I accept this class of finding", never "I accept a scanner that crashed on crafted input" (§4.6).

NONE and LOW both map to WARN, which is the point rather than an oversight: a clean document and one carrying only low-severity findings were a single indistinguishable value through 0.4.0.

Known limitation recorded (32, was 31)

Severity still carries disposition intent on the detection side — the separation above is caller-side only. Two places say so outright: ACTIVE_CONTENT_ORDINARY_SEVERITY = LOW exists because grading an ordinary external image HIGH fail-secured ordinary uploads, and the quarantine floor was raised to MEDIUM+ to repair the same regression from the other end. Closing it changes the grading and so fires a consumer-notification promise; deferred deliberately. See docs/LIMITATIONS.md.

Measured — what the upload door costs on benign documents (33rd limitation)

Every field measurement this project had published was per URL. None of them answered the question a consumer actually asks: how often does an ordinary document cost me a human? Three benign populations were run through screen_output under PRESET_USER_UPLOAD and counted at document granularity — 98 of 185 vendor-published doc pages (53.0%), 88 of 547 model-written notes (16.1%), and 133 of 389 first-party reference documents (34.2%) disposed to something other than WARN.

The number is bad and is published as measured. docs/PLAN-v1.md committed to that in advance — "et rødt FP-resultat er like verdifullt" — and the response here is a documented limitation, not a recalibration: moving the grading would fire a locked consumer-notification promise, and the drivers are residuals this document already concedes rather than anything newly discovered.

  • docs/fp-sweep.py — the method, re-runnable, corpus roots as arguments. It refuses to print a pooled total (the populations have different provenance and different denominators) and it aborts if the default action map stops sending exactly NONE and LOW to WARN, since the published count is a statement about assessed risk and only equals one while that holds.
  • tests/test_corpus.py::test_the_published_fp_metric_is_a_risk_statement — the same equivalence, pinned in the suite. An action_map override is a supported feature as of the axis separation above, so without this pin a consumer-facing number could change meaning with nothing failing.

Corrected — 0.3.3 listed two behaviour-change classes and there were three

Sweeping the same population against the v0.3.1 tag a consumer actually pins (scratch venv, git+file://…@v0.3.1, resolved version asserted) returned 99 of 185 (53.5%) where the current tree returns 98 (53.0%). One document moved, and chasing it corrects a claim rather than confirming one.

docs-en-fullscreen.txt disposed QUARANTINE_REVIEW at 0.3.1 and WARN now, because markdown:link-anchor-injection no longer fires on it. Under 0.3.1 that pattern matched 300 characters of ordinary prose: it opened at a [, ran across intervening text containing the word execute, and closed at a distant ](…) belonging to a different construct. The 0.3.3 ReDoS fix excluded [ from the anchor class and ( from the target class, which telescopes the runaway — and, as a side effect nobody measured at the time, deletes this false-positive class too.

0.3.3's Known behaviour changes said "None measured" and then named two exceptions: URLs with a literal ( in the target, and comment bodies with a literal ( before the keyword. It missed the third: an anchor can no longer span a [, so a match that used to bridge two separate markdown constructs no longer forms. The correction is in our favour — one fewer false positive per 185 documents of vendor documentation — but it was a behaviour change presented as none, and it took a field sweep to find it.

Fixed

  • A retracted number was still living in a test comment. tests/test_wiring.py credited a consumer's capture store with 35 of 35 query-carrying URLs. That consumer retracted it the next day and re-measured 28 of 28 on the same 81-URL corpus; docs/LIMITATIONS.md was corrected then and the comment was not. Corrected, with the retraction written into the comment so it cannot read as a second, disagreeing measurement.

  • Five current-state version claims had never been updated by any release. The 0.4.0 release commit touched three files — CHANGELOG.md, pyproject.toml, src/llm_ingestion_guard/__init__.py — and deferred the README deliberately, so that the install block would not point at a tag before a clean-venv install had proven it resolved. That proof step never ran, so tag v0.4.0 permanently carries a README advertising v0.3.4. The tag is not moved; the ordering is.

    Sweeping every tracked file for a version claim, rather than the four surfaces the release checklist named, found four more that no release had ever touched — plus a stale test count:

    • SECURITY.md — "The project is pre-1.0 (0.2.x, alpha). Only the latest published version receives fixes." The only one with a consequence for an outsider: it named a support window two minor lines behind the code.
    • README.md — **Status:** v0.3, stale since 0.4.0.
    • docs/BRIEF.md and CLAUDE.md — "v0.2 (alpha)", stale since 0.3.0. The latter also claimed 12 modules where src/ has 15.
    • docs/ADOPTION-BRIEF.md — "703 passing", where the suite is at 717.

    Every one of them is a current-state claim. Measurement provenance — "New in v0.4.0", "verified identical on 0.2.0 and 0.3.1", "measured against the v0.3.1 tag" — is left exactly as written, because bumping those would falsify the record rather than update it. From here all current-state surfaces move in the release commit itself and are verified by git show <sha> before the tag exists, since that is the only check the previous ordering could not perform.

    Found because llm-ingestion-okf took our report of this defect class as a hypothesis about their own repo, measured it, found a worse instance, and sent back the generalization: writing down a trap is not the same as applying it.

[0.4.0] — 2026-08-10

Behaviour change, not a pure fix — and that is why this is 0.4.0 and not 0.3.5. The three transform surfaces gain a refusal path they did not have. A caller that passes a document larger than 1 000 000 characters now gets an exception where it previously got a result. Adding a raise to a function that was previously total is breaking under SemVer whatever the measured blast radius turns out to be, so the number follows the change, not the survey.

The measured blast radius, for the record: zero. linkedin-studio pins an exact tag, so nothing reaches it until it re-pins. llm-ingestion-okf moved to the range >=0.3,<0.4 (their f536e13), so a 0.3.5 would have landed on them at their next resolve without an action on their part — but they answered our query (20260802T193351Z) with a measured no: zero call sites for sanitize / fence / neutralize / prepare_input anywhere in their src/. Their screen_output path reaches only scan_output, which truncates and does not raise. Releasing as 0.4.0 puts this outside their ceiling regardless, so they cross it deliberately rather than by resolving.

Added — input-size cap on the transform surfaces (OWASP LLM10)

sanitize, fence and neutralize now raise OversizeInputError above MAX_INPUT_CHARS (1 000 000) instead of accepting text of any length. Since sanitize is step 1 of prepare_input and only ever removes, that single refusal bounds the whole input path.

They reject where the scanners truncate, and the asymmetry is the point:

  • scan_lexicon / scan_output return findings. Reading a prefix costs detection in the tail and nothing else — a lossy answer, but an answer.
  • sanitize / fence / neutralize return content. Truncating would return a shortened document (silent data loss for anything that persists the result) or a transformed prefix followed by an untransformed tail — a bypass, since an attacker chooses where in the document the payload sits.

The invariant the three now keep: returned text is always fully transformed, or not returned at all.

OversizeInputError subclasses ContractViolation, so a pipeline already bracketing its quarantined stage in except ContractViolation keeps failing closed. Like its parent it is alert-routable: the message carries the size and the cap, details names the refusing surface, and neither carries input. max_input_chars is a per-call parameter, defaulting to the single calibrated constant.

Added — the last two detection surfaces bound their input too

scan_active_content called directly and okf.link_graph were the two surfaces still reading attacker-supplied text with no cap. Both truncate and record, the way the other scanners do:

  • scan_active_content(text, source, max_scan_chars=MAX_SCAN_CHARS) emits one active:oversize-input finding (MEDIUM, LLM10) and scans the prefix. Reached through scan_output the text is already under that surface's cap, so the flag is raised once, there — max_scan_chars is now passed down.
  • link_graph(bundle, max_scan_chars=MAX_SCAN_CHARS) caps each body and records (from_id, body_length) in the new LinkGraphResult.truncated field. The field is additive with a default, so existing positional construction and attribute access are unaffected.

What truncation costs is named rather than implied: past the cap, "no finding" means "not looked at". That is precisely what a silent truncation would hide, and why truncated exists as a field instead of a log line — it is what separates "no links past here" from "no links read past here".

Recorded in docs/LIMITATIONS.md.

[0.3.4] — 2026-08-01

Denial-of-service fix on the INPUT path. Upgrade from 0.3.3. 0.3.3 swept the 83 lexicon patterns arm by arm and left every other table on 0.3.2's hand-written rows. Generalising the sweep over all eleven regex-bearing modules found three more quadratic patterns — two of them on the input path, one in sanitize, the first thing every ingested document touches. No disposition changes: recall was measured case by case and nothing was lost. Earlier tags are not moved.

Fixed — three quadratic patterns, two on the input path

Same class as everything 0.3.2 and 0.3.3 fixed: a run in front of a required literal, so crafted input that never supplies the literal makes every start position rescan the tail. Each exponent is read across four doublings, not from a two-point ratio.

Pattern Crafted payload Measured @ 100 000 Exponent
sanitize._HTML_COMMENT_RE <!-- × N 20.1 s 1.96–2.14
active_content.URL_IN_TEXT_RE <a + A × N + > 12.99 s / 14.9 s 1.87–2.22
okf._MD_LINK_RE [ × N 7.1 s 1.99–2.05

These are worse than the 0.3.3 findings, and the reason is a separate finding of its own: MAX_SCAN_CHARS is applied in scan_lexicon and scan_output only. sanitize, neutralize, scan_active_content and the okf link graph accept input of any size, so there is no cap to extrapolate to. Now documented as a residual in docs/LIMITATIONS.md; extending the cap into the input path changes the contract for existing callers and is deliberately not done in a ReDoS patch.

Each fix is the one the pattern's own shape allows — the 0.3.3 lesson that a fix choice must not be copied blindly from a neighbouring table:

  • sanitize drops the regex for a str.find scan, semantically identical to the lazy <!--.*?--> it replaces. Excluding < from the run would lose every comment containing markup (<!-- <b>x</b> --> is the ordinary case); bounding the run would be a one-line carrier bypass of the exact construct the stripper exists to remove. The module's own "no catastrophic backtracking" comment was wrong in the same way output's was before 0.3.2, and is corrected in place.
  • URL_IN_TEXT_RE bounds its scheme run to an RFC 3986 scheme ({0,63}). Bounding is safe here only because this is a defanger applied inside a tag already flagged active:raw-html, so padding shifts where the match starts rather than evading detection. A lookbehind killing interior start positions was measured too and rejected: it drops -http://evil.com and .http://x.com, a one-character evasion of the defanger. Bounded, the pattern runs in 0.185 s at the full 1 000 000-char cap.
  • okf._MD_LINK_RE excludes [, matching active_content.MD_LINK_RE exactly, including the nested-label trade already documented there.

Changed — the sweep covers every regex surface, not one table

docs/redos-sweep.py now sweeps 150 patterns across 11 tables (0.3.3 covered 83 in one). The collector is mechanical on both axes so no one has to remember to list anything: it walks each module's namespace for compiled patterns, and it derives each pattern's call mode from the module source, because .sub() and .finditer() visit every start position where .match() cannot. A pattern reachable only through a helper parameter gets the worst mode, marked * — the fallback can over-measure but never miss.

Two arm shapes the generator cannot express are pinned by hand as a result: a tag that closes around a long body (repeating-unit payloads never close it), and a run of plain characters carrying no anchor at all. The okf destination run gets no row on purpose: [^)\s]+ cannot fail, so a pin for it could never go red.

A lexicon candidate flagged at ×2.8 measured linear across four doublings (exponent 0.96–1.03) — the near-noise-floor false flag the script's own docstring warns about, confirmed a second time.

676 tests (+10). Coverage matrix unchanged at 128/128 caught, 6/6 gaps holding.

[0.3.3] — 2026-07-31

Denial-of-service fix, and a correction to 0.3.2. Upgrade from 0.3.2. The sweep 0.3.2 shipped was incomplete, and it said otherwise. Two lexicon patterns were still quadratic — reachable through scan_output, not only on the input path. No disposition changes: recall was measured case by case and nothing was lost. The v0.3.2 tag is not moved.

Fixed — two quadratic patterns in the lexicon table

8deca93 scoped the remaining ReDoS duty to the lexicon path, and this is that work: all 83 patterns measured, arm by arm. Two are quadratic, same shape as everything 0.3.2 fixed — a run in front of a required literal, where the run may cross the pattern's own opening anchor.

Pattern (arm) Crafted unit Measured At the 1 000 000-char cap
markdown:link-anchor-injection (anchor text) [ 1.91 s @ 8 000 ~8.3 hours
markdown:link-anchor-injection (URL run) [system]( 0.006 s @ 8 000 ~89 seconds
markdown:link-ref-comment (.* run) [//]: # ( 0.22 s @ 8 000 ~1.0 hour

Exponent measured over five points (1 000 → 16 000): 1.98 — quadratic, not exponential. Legitimate content of the same size is unaffected: 0.316 s at N=100 000 (prose 0.316 / html 0.315 / markdown 0.297 / connection-string 0.296).

These were not input-path-only, and that is the correction. scan_lexicon runs on the output path too, so 0.3.2's "the last quadratic-backtracking site on the output path" was false when written. Measured through the public gate before this fix: scan_output("[" * 100_000) took 334.7 s. The claim was too broad because the sweep behind it drove the [ payload only through scan_active_content — no row ever drove it through the lexicon. The statement is corrected in docs/LIMITATIONS.md.

The fix is anchor exclusion, per the rule active_content already documents — bounding attacker-controlled content would be a one-line detection bypass. The excluded character is (, not [:

markdown:link-anchor-injection
  \[[^\]\[]*(?:system|…)[^\]\[]*\]\([^)(]+\)
markdown:link-ref-comment
  \[//\]:\s*#\s*\([^(\n]*(?:ignore|…)

[ was the obvious choice and it was measurably worse. Excluding [ from the URL run drops [override your rules](https://[::1]/x) — still covered, three other patterns fire on it — but excluding [ from the link-ref comment run drops [//]: # (see [x] then ignore this), which no other pattern catches. The anchors contain ( as well, so excluding ( telescopes just as effectively at zero measured recall cost. Both forms verified linear (×1.99–2.02 on doubling).

Known behaviour changes

  • None measured. Every case that matched before still matches, except URLs containing a literal ( inside a markdown link target and comment bodies containing a literal ( before the keyword. No corpus, showcase, or coverage row moved; 666 tests pass.

Tests

Four rows added. Three name the guilty pattern per arm (test_crafted_redos_payload_stays_bounded_in_the_lexicon), one covers the composed gate (test_gate_is_bounded_on_the_payload_the_first_sweep_missed). Pre-fix they failed at 297 s, 8.1 s, 55 s and 334.7 s.

N is per row deliberately. The URL arm is quadratic with a small constant and ran 0.9 s unfixed at N=100 000 — under the 2.0 s bound, so that row would have passed whether or not the pattern was fixed. It is measured at N=300 000 instead, where crafted (8.10 s) and legitimate (0.926 s) separate 8.8×.

Residual

The sweep flags on timing and ignores measurements below a 1.5 ms noise floor at N=8 000. An arm hiding just under it could still cost ~23 s at the cap, so what this supports is "no arm worse than ~23 s", not "no quadratic arm remains". The blind spot is not hypothetical: a generic-payload pass found only one of the two patterns. The second appeared only once payloads were synthesised per run from each pattern's own skeleton. Recorded in docs/LIMITATIONS.md.

[0.3.2] — 2026-07-31

Denial-of-service fix. Upgrade from 0.3.1. The output gate could be made to spend hours on a single call by crafted input it accepts by design. No disposition changes for ordinary documents — the one measured exception is listed under Known behaviour changes below. The v0.3.1 tag is not moved.

Fixed — 19 quadratic regex runs on the output path

scan_output claimed LLM10 self-safety on the grounds that its patterns contain no nested quantifiers. That is true and it is not the property that matters. A run in front of a required literal, reachable from a short anchor, is enough: crafted input repeats the anchor and never supplies the literal, so every start position rescans the tail. Quadratic, not exponential — and quadratic is sufficient here.

Measured, not argued (Python 3.14, this machine):

Input Time through scan_output
<a: × 100 000 (300 KB) 458.7 s
size-matched ordinary prose 0.31 s
same payload extrapolated to the 1 000 000-char input the gate itself accepts ~5.7 hours for one call

max_scan_chars does not mitigate this. It bounds the input; quadratic work on a bounded input is still hours. That claim was stated in both output.py and calibration.py and is corrected in both.

The fix is per pattern, not uniform:

  • active_content + the lexicon table (15 runs) — exclude the character that opens the pattern's own anchor ([ for markdown, < for tags), so a run cannot reach past the next start position and the per-start costs telescope. Verified to cost no recall: long URLs, long alt text, and < inside a quoted attribute all still match. Bounding instead would have been linear too, but wrong here — the content is attacker-controlled, so padding past a bound would be a one-line bypass of the EchoLeak class this table exists to catch.
  • *-connstr secret egress (4 runs) — bound the password at the new MAX_CONNSTR_VALUE (256) in calibration. The exclusion fix is unavailable: the anchor character is /, and passwords containing / are the common case (measured — they match today).
  • hybrid-xss:script-tag — had neither option, since its run is the script body, which may legitimately contain <. It now matches the opening tag and no longer requires </script>.

Known behaviour changes

Two, both measured against the v0.3.1 tag rather than reasoned about:

  • A JWT used as a DB password, over 256 chars, is no longer CRITICAL. The remaining detections (entropy:base64-blob HIGH, egress:jwt-token MEDIUM) top out below CRITICAL, so the any-tier block is lost: under PRESET_TRUSTED_SOURCE such a document moves from fail_secure to quarantine_review. Under PRESET_USER_UPLOAD it still fail_secures, and a generic long password still trips entropy:base64-blob at CRITICAL with no change at all. The credential is never silently missed; on one preset it is held for review instead of halted.
  • hybrid-xss:script-tag now fires on prose that merely mentions <script> — and this costs no consumer a disposition. Any text containing a literal <script> already produced active:raw-html at HIGH on 0.3.1, so the same document disposed identically before and after. The label is new; the outcome is not. By the same measurement, the fail-open this closed (an unclosed <script>alert(1)) was confined to scan_lexicon called on its own — through either composed gate, active:raw-html already caught it.

Both are recorded in docs/LIMITATIONS.md (still 29 items — these replace nothing).

Method note

The defect was found by a composed-gate DoS test that stayed red after every individual scanner had been made linear; the remaining 813× was the lexicon's six html-obfuscation patterns. A per-scanner test alone would have shipped it. The static shape analysis used to find candidates also missed [\s\S]*? in script-tag — the sweep that matters is measurement, not a regex over regexes.

662 tests pass (was 642), and the suite is faster than before the fix.

[0.3.1] — 2026-07-25

Regression fix. Upgrade from 0.3.0. v0.3.0 made the high-untrust upload path unusable for ordinary documents — measured, not projected. llm-ingestion-okf projected the consequence from the 0.3.0 changelog text before the tag was cut; the release went out without the inbox being read. The v0.3.0 tag is not moved.

Fixed — the upload path is usable again without losing EchoLeak detection

Measured on v0.3.0, both doors (screen_output under PRESET_USER_UPLOAD and okf.import_bundle with origin=EXTERNAL): a document with one ordinary remote image disposed fail_secure; one ordinary link, autolink or reference definition disposed quarantine_review. Only documents with no external references persisted. Two independent defects compounded, and both had to be fixed — either alone leaves the path blocked:

  • Severity graded on construct type instead of URL shape. markdown-image was HIGH for any external image, but the exfiltration primitive is not "an image" — it is a URL that moves bytes to a host the attacker controls. ![diagram](https://example.com/arch.png) carries nothing. Severity now grades on shape: a URL that only names a remote document (http(s) or protocol-relative, no query, no userinfo, no percent-escapes, no opaque host label or path segment) is LOW; anything that can carry a value keeps the carrier's full severity. raw-html and data: URIs have no ordinary form and stay HIGH unconditionally. Opacity reuses entropy's primitives — decodable base64 (≥20 chars), hex id (≥32), or Shannon entropy ≥4.4 at ≥24 chars — calibrated 2026-07-25 against real documentation URLs (worst legitimate token H=4.08; exfil payload segments 4.36-4.54). New constants live in calibration with the rest.
  • The quarantine_default floor fired on any finding. It rested on the premise that a finding is the exception; adding the active-content detector in 0.3.0 made every ordinary markdown link a finding, and the floor then held ordinary documents for review. The floor now fires at MEDIUM+. This is a no-op for every detector that shipped before 0.3.0 — the lexicon holds no LOW/INFO pattern and no other detector emits LOW (asserted in tests/test_calibration.py) — which is why this is a patch and not a minor.

Unchanged, deliberately: no new public API and no new preset (a middle tier is 0.4.0 work); the allow_reserved=True mode-b default stands — two independent consumers document it as load-bearing; the gate still never rewrites content.

Added

  • False-positive corpus covers ordinary markdown. The 0.3.0 corpus had zero markdown links or images, asserted only under PRESET_TRUSTED_SOURCE (where every non-CRITICAL finding WARNs anyway), and drove the input path — so scan_output step 6, where active content actually lives, was never reached. That is how a regression this size passed 522 green tests. The corpus now carries realistic documents and asserts them on the output gate under the upload preset, plus a counter-corpus of exfil-shaped URLs (query, base64/hex path segment, percent-encoded payload, opaque subdomain, userinfo) that must still block.
  • Two new documented gaps in docs/LIMITATIONS.md, both asserted by the coverage matrix: pure beaconing (a bare-path image on a hostile host still fetches, and the fetch is not graded) and short opaque URL segments (<24 chars, below what entropy can resolve). Percent-escapes counting as data-carrying is recorded there as a known false positive.

[0.3.0] — 2026-07-25

A minor bump, not a patch — deliberately. The changes under Changed alter what an existing caller observes with no code change on their side, so a >=0.2,<0.3 pin stops here rather than absorbing them silently. Re-test that branch before widening the pin. Still alpha: the public API may change again before 1.0.

Changed — observable gate behaviour (re-test before upgrading)

Three commits since v0.2.0 change dispositions for an unchanged caller. Two tighten the gate; one loosens it.

  • okf.import_bundle no longer path-rejects reserved basenames. At v0.2.0, index.md / log.md anywhere in a received bundle was an unconditional per-concept hard reject (FAIL_SECURE), and import_bundle took no keyword for it. The new allow_reserved keyword defaults to True on this mode-b received-bundle path, so those files are scanned — their body is the highest-priority injection surface — rather than refused, and may clear the floor and become mergeable. This is the one loosening change: content a v0.2.0 consumer never saw can now reach it, so a consumer whose tests pin the v0.2.0 reject must re-check, not just bump. A front-end materialising individual uploads must pass allow_reserved=False to keep the shadow-reject there; validate_concept_path still defaults to False.
  • Active content now reaches the disposition engine. scan_output step 6 runs scan_active_content, so markdown images/links, reference definitions, autolinks, raw active HTML and data: URIs surface as active:* findings (OWASP LLM05 — the EchoLeak / CVE-2025-32711 class) instead of being admitted with findings=[]. These carry real severities (zero-click auto-fetch/execute HIGH, click-required MEDIUM), and two MEDIUM+ findings compound-escalate one tier, so a document that passed clean at v0.2.0 can now WARN, quarantine, or fail secure in both screen_output and okf.import_bundle.
  • Base64-wrapped secrets are now caught as egress. The output gate's decode-and-rescan feeds decoded base64 plaintext through both the lexicon and the LLM02 secret-egress detector, so a base64-wrapped credential surfaces as decoded:egress:* instead of disappearing. Hex-wrapped remains a documented gap (docs/LIMITATIONS.md).

Added — runnable threat-coverage matrix

A single declarative manifest (llm_ingestion_guard.coverage) that proves, in one place, every vulnerability class the guard stops — and the documented gaps it does not. Two consumers of the same source of truth:

  • python -m llm_ingestion_guard.coverage — a narrated matrix (class -> OWASP -> expected -> observed -> verdict); exit 0 iff every caught class is caught and every documented gap holds. Stdlib-only, CI-usable.
  • tests/test_coverage_matrix.py — asserts total recall over the core matrix (carriers, all 83 lexicon patterns, entropy/decode-rescan, active content, the contract asserters, the disposition engine, OKF T1–T7), asserts every documented gap still holds, and guards completeness (every lexicon pattern id, and every OWASP anchor claimed, has a case). Adds the full 25-pattern LLM02 secret-egress set and the container-layer front-end classes (CSV formula-injection, zip-slip, zip-bomb, symlink). +165 tests (357 → 522), no core dependency added.

This is the real-case validation gate ahead of a v1.0 freeze.

Documentation — consumer adoption + README value proposition

  • docs/ADOPTION-BRIEF.md — a self-contained brief a consumer repo (OKF second-brain / LLM wiki) can plan an inclusion from: the write-time trust-boundary argument, the two bookends + 8-step contract, the shipped OKF adapter (import_bundle mode-b), how to verify (coverage matrix), how to depend (stdlib-only core), and a checklist for when/where to wire it.
  • README rewritten to lead with the write-time trust-boundary framing, add a first-class OKF / LLM-wiki support (shipped) section for import_bundle, a concrete What it protects against catalogue (attack classes grouped by OWASP anchor, driven by the coverage matrix), and correct the test badge (357 → 522). Every claim verified against the code.
  • docs/LIMITATIONS.md — the full honest-limitations list (15 items + the four documented gaps + out-of-scope) moved out of the README, which now carries a high-impact summary + link, so protection and limits read in balance.

[0.2.0] — 2026-07-06

Added — OKF adapter (stream 1)

An OKF (Google Open Knowledge Format v0.1) adapter on top of the format-agnostic core (llm_ingestion_guard.okf). The core stays text -> findings; the adapter knows OKF structure and routes scannable regions into the existing machinery. All TDD (failing test first), +61 tests. Verified against the OKF SPEC.md (2026-07-06). See docs/OKF-INGESTION-BRIEF.md §8.

  • parse_frontmatter — strict, reject-by-default frontmatter loader; refuses anchors, aliases, explicit tags, merge keys, block scalars and flow collections by construction, so YAML anchor/alias DoS and !!python/object coercion cannot occur (not a general YAML parser, by design). (T2)
  • scan_concept — whole-concept scan surface: frontmatter values (incl. description, read first under progressive disclosure), resource and body all go through scan_output. (T1)
  • validate_concept_path — path / reserved-name gate: rejects .. traversal, absolute paths and index.md / log.md shadowing; returns the concept-ID. (T4)
  • validate_resource_url — resource https allowlist: rejects non-https before commit (reject, not defang — the format imposes no scheme constraint itself). (T3)
  • stamp_concept / format_log_entry — provenance stamping: origin × channel → trust × disposition per concept, emitted as log.md lines. Trust follows the origin, never the insertion channel. (T6)
  • import_bundle — received-bundle iterator (mode b): validates each concept (path, frontmatter, resource, scan, stamp) independently; one bad concept is rejected fail-secure while the rest are still checked; the aggregate disposition is the most severe. (T7)
  • link_graph / resolve_link / extract_link_targets — in-import cross-link graph: resolves .md links (bundle-absolute or relative) to concept-IDs, flags dangling links (the §7.2 dormant-injection signal) and rejects dangerous-scheme or bundle-escaping targets. (T5a)

Deferred

  • Cross-run persisted link graph (T5b) — catching a link planted in one run whose poisoned target is written in a later run (§7.2) needs durable graph state whose storage/ownership depends on the consuming pipeline. Deferred to the consumer-wiring stream; cross-run dormant links remain a documented residual risk (README honest-limitations).

[0.1.0] — 2026-07-06 (alpha)

The stdlib-only core, built test-first (TDD) per docs/PLAN.md. Tagged v0.1.0.

Added

  • report — shared Finding / Report / Severity / Source types.
  • sanitize — carrier stripping (zero-width, BIDI, Unicode-tag, HTML comment, data:); byte-identical on clean input.
  • entropy — Shannon / base64-like / hex-blob detection; base64 decode-and-rescan.
  • lexicon — JSON pattern data + loader; raw/normalized/homoglyph/rot13 variants; ReDoS-bounded, size-capped.
  • fence — randomized per-call spotlight delimiter; attacker marker-strip.
  • neutralize — opt-in defang of active-content output (byte-identical when clean).
  • output — compose lexicon + entropy + decode-rescan over emitted text; secret egress patterns (OWASP LLM02); report-only, never mutates.
  • disposition — WARN | QUARANTINE_REVIEW | FAIL_SECURE under a source-trust policy; compound-signal escalation; fail-closed when the scanner errors.
  • contract — write-time asserters that raise: assert_tool_less, assert_credential_allowlist, scoped_env.
  • grounding — the SourceGroundingCheck seam for semantic poisoning (interface only; [judge] implementation plugs in behind an extra).
  • Top-level wiring — the prepare_input / screen_output §6 bookends plus the full public surface; end-to-end showcase and adversarial + false-positive corpora.

Security

Pre-release hardening from an independent adversarial review (all TDD, failing test first):

  • entropy — decode-and-rescan now runs before false-positive suppression, so an injection blob prefixed with an SRI/media marker (to dodge the entropy finding) is still decoded and rescanned by the lexicon. Suppression gates only the entropy finding, never the decode.
  • output — the invisible-carrier invariant now holds on the persist gate: scan_output flags zero-width / BIDI presence (output:zero-width-present, output:bidi-present) and disposition treats those plus lexicon:unicode-tags-present as any-tier carriers, so a carrier in model output fails secure even under a trusted policy.
  • contract — assert_credential_allowlist catches a bare <PROVIDER>_KEY (e.g. STRIPE_KEY); the previous regex silently missed it (fail-open). The rule is deliberately broad (also flags PARTITION_KEY/SORT_KEY as loud, allowlistable false positives) — fail-loud beats fail-silent for isolation.
  • disposition — guard runs decide inside its guarded block, so a malformed report can no longer escape the fail-closed guarantee.
  • output — secret-egress placeholder suppression anchors word markers (example, todo, …) to a word boundary, so a real secret that merely contains such a word is no longer suppressed (fail-open egress miss closed).