open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Releases 4 Activity
4 releases 13 tags
  • v1.1.0 ca4f97c8c9

    v1.1.0 Stable

    ktg released this 2026-08-13 20:58:49 +00:00 | 11 commits to main since this release

    No known key found for this signature in database

    [1.1.0] — 2026-08-13

    Fixed — a mapping construct in OKF frontmatter no longer degrades into a string

    okf.parse_frontmatter gives the mapping class no expressible form by design
    (T2). Two routes escaped that: they parsed "successfully" into the wrong type
    instead of raising. Both are closed, and both now FAIL_SECURE through
    okf.import_bundle (door C).

    route was now
    sources:
    - uri: https://e.com/a    		 string 'uri: https://e.com/a' — WARN OKFFrontmatterError — FAIL_SECURE
    sources:
    - uri:    		 string 'uri:' — WARN OKFFrontmatterError — FAIL_SECURE
    attester: resource: attesters/x.py string 'resource: attesters/x.py' — WARN OKFFrontmatterError — FAIL_SECURE

    The security consequence was the same in each: a pointer parked in a degraded
    mapping rides through in a key the resource allowlist never inspects, and mode-b
    import_bundle writes the merged concept verbatim. The first route was documented
    at docs/LIMITATIONS.md:43; the inline second colon was found by measurement
    while closing it    , and is the reason this release names two routes rather than one.
    Neither shape is conformant OKF — a well-formed bundle does not produce them; a
    malformed or hostile one can.

    What closed is the type confusion, not pointer-smuggling as a class. T3 still
    inspects resource and nothing else, so an honest string under another key rides
    through exactly as before: attester: attesters/sql_equality.py is WARN, while
    the same path under resource: FAIL_SECUREs. The string is still scanned like any
    other frontmatter value under T1. Nothing about that changed here.

    The boundary is where YAML puts it, ground-truthed against PyYAML 6.0.3 rather
    than reasoned: ": " and a trailing ":" are exactly the two shapes where a plain
    scalar becomes a mapping, and they are refused. A colon carrying neither a space nor
    a line end opens no mapping — domain:security and https://e.com:8443/a still
    parse — and a quoted scalar (- "uri: x") is still a scalar. Quotes are retained
    rather than stripped; that divergence from YAML is unchanged and now pinned.

    This is a behaviour change inside the freeze, not a break of it. No exported
    name moved. A document that disposed WARN on 1.0.0 may dispose FAIL_SECURE
    here — the 1.0.0 entry says exactly this is a fix, not a break. A consumer whose
    bundles carry an unquoted ": " in a frontmatter value will now see those concepts
    refused at import; quote the value, and it parses.

    Suite 792 → 802: 13 rows added (4 rejected shapes, 7 admitted ones, 2 through
    import_bundle), 3 retired (the two that pinned the defect, and the one-key row
    in the block-list table). 129/129 classes, 6/6 documented gaps, 35 limitations —
    all unchanged.

    Downloads
  • v0.7.0 be9759b4b3

    v0.7.0 Stable

    ktg released this 2026-08-13 18:22:29 +00:00 | 21 commits to main since this release

    No known key found for this signature in database

    Added — active:raw-html-link, a click-required carrier class for raw HTML

    Raw HTML graded on activity alone: every active tag was HIGH. So the same URL
    was LOW as [t](https://example.com/guide) and HIGH as
    <a href="https://example.com/guide"> — an asymmetry produced by syntax, not by
    affordance. Following an anchor needs a human, exactly like the markdown inline
    link that has been MEDIUM since 0.3.1.

    <a> and <area> now report as active:raw-html-link at MEDIUM. Everything
    a renderer fetches or executes unattended keeps active:raw-html at HIGH, and the
    event-handler test runs first, so <a onclick=...> is graded as the
    execute-class carrier it is rather than downgraded with the anchors.

    The URL-attribute branch deliberately stays on the HIGH side: a name outside the
    active set has unknown rendering, and href is not the only URL attribute it may
    carry. Grading <Card src="..."> as a link would be reasoning, not measurement.

    This is a new label, and labels are a contract surface consumers pin against.
    A document that previously produced one active:raw-html finding may now produce
    two findings, one per carrier class.

    Changed — a tag whose whole affordance is a URL it does not carry is inert

    </a>, <Frame>, <video /> and <img alt="..."> without src were active by
    name while naming no target at all. This is <base />'s argument from 0.6.0 —
    "attribute-less, therefore no affordance in any renderer" — applied to the rest of
    the name branch. The test is for the URL attribute's presence, not for a
    readable value: a value the parser cannot resolve keeps the tag active, mirroring
    the fail-secure gap _url_attr_is_external already leaves open.

    Every other member of the active name set does something a URL cannot describe —
    <script> executes its body, <style> restyles, <form> submits — and stays
    active with no attributes at all.

    Changed — active_tag_class is the classification point; is_active_tag wraps it

    docs/rawhtml-census.py measures candidates by patching this symbol, and a
    boolean could only express a narrowing, never a regrade. Left as a boolean, every
    carrier candidate would have measured equal to PRODUCTION — silently, and in the
    direction that reads as "no change helps".

    Measured

    docs/rawhtml-census.py, three populations, each at one corpus state and each
    against its own denominator — the two wiki corpora share content and are never
    summed. Documents that stop being fail_secure under PRESET_USER_UPLOAD, from
    0.6.0 as shipped to 0.7.0, with the ceiling being the raw-HTML detector switched
    off entirely:

    population documents 0.6.0 → 0.7.0 ceiling share of achievable
    reference-corpus 389 54 → 53 53 1 of 1
    vendor-harvest 187 62 → 20 18 42 of 44 (95%)
    generated-notes 552 59 → 15 13 44 of 46 (96%)

    Neither change alone is worth shipping, and the census is why they went out
    together.     Alone, the split frees 8 documents in each wiki corpus and the
    narrowing 21 and 23 — but 8+21 measures 42 and 8+23 measures 44. The residual is
    13 documents in both corpora: the narrowing strips a document's </a> and
    <Frame>, and what is left is the <a href=...> the split grades down, so each
    change alone leaves the document blocked by the other's residue.

    Tightening, measured: 0 documents on both trust tiers, in all three
    populations.     That zero is empirical and thinner than it looks — the split
    alone tightens 13 documents on the trusted tier in vendor-harvest and 14 in
    generated-notes, and the narrowing cancels each one. See docs/LIMITATIONS.md
    for why it must not be read as "cannot happen".

    The PRODUCTION (as shipped) row matched C1 + D (0.7.0) field for field in
    every population, which is the check that the census and the shipped predicate
    have not drifted apart.

    Known behaviour change

    count drops on documents containing </a>. Through 0.6.1 an end tag was
    active by name, so count ran roughly 1.6× the opening-tag total and a start/end
    pair counted 2. It is now the opening-tag total. The field's meaning did not
    change and the finding count is unaffected — the class still collapses to one
    finding per class per document.

    Downloads
  • v0.6.1 0903785187

    v0.6.1 Stable

    ktg released this 2026-08-11 20:30:51 +00:00 | 25 commits to main since this release

    Fixed — the zero-width check tested identity, so emoji-composed documents were hard-blocked

    _ZERO_WIDTH (sanitize, input) and _ZERO_WIDTH_CPS (output,
    _scan_invisible_carriers) tested U+200D on codepoint membership alone.
    disposition._CARRIER_LABELS grades a carrier as any-tier fail_secure with no
    appeal, so any first-party document containing a ZWJ-composed emoji —
    professions, families, skin tones, flag variants — was blocked permanently, with
    no preset able to release it. Reported by ms-ai-architect, confirmed here
    against the code.

    The worse half was not in the report: sanitize removed the joiner, silently
    decomposing one emoji into two unrelated ones. A module whose published contract
    is "only ever removes carriers" was corrupting content.

    The fix is the shape our own lexicon row unicode:zero-width-in-word
    (\w[ZW]\w) already used — judge the joiner by context, not identity. A ZWJ
    is exempt only when both neighbours are emoji-context codepoints. Half-context
    is not context, so a<ZWJ>😀 stays a carrier and an attacker cannot buy
    exemption with a single trailing emoji.

    Blocks, not an emoji table. Measured against Unicode 17.0's
    emoji-zwj-sequences.txt: 1614 RGI sequences use 122 distinct codepoints
    adjacent to a ZWJ, and five block ranges cover 122/122. The measurement earned
    its keep — the hand-reasoned candidate table missed U+2194, U+2195 and U+2B1B.
    Shipping the RGI list itself would be exact the day it landed and stale at the
    next Unicode release, reopening this false positive for every new emoji; whole
    blocks carry the unassigned headroom (458 Cn codepoints) that future emoji are
    allocated into, so the table does not age. The predicate is defined once in
    sanitize and imported by output; a cross-surface test asserts the two halves
    agree on six inputs.

    Known behaviour change

    Documents whose only finding was an emoji-context ZWJ now persist unattended.
    On PRESET_USER_UPLOAD they move from fail_secure to WARN. Measured across
    the three false-positive populations at one corpus state: 1 document of 1126
    (reference-corpus 1/389, vendor-harvest 0/187, generated-notes 0/550). This is a
    loosening of the upload door, not of detection — recall is unchanged at 128/128
    demonstrated classes with 6/6 documented gaps holding, and a ZWJ anywhere else,
    including between a word character and an emoji, is graded exactly as before.

    Patch, not minor. 0.6.0 called itself minor for loosening the same door, but
    that was a policy choice — <base> left the active name set by decision. This
    one restores a contract the module already published, against a class that was
    never meant to be blocked. A fix whose observable effect is the point of the fix
    is what the patch level is for.

    Residuals — both documented (docs/LIMITATIONS.md, 33 → 34 items)

    • A ZWJ between two emoji is now exempt, so it can carry a narrowband covert
      channel: one emoji per bit, and it cannot split a word. A deliberate narrowing,
      stated rather than hidden.
    • U+200C (ZWNJ) still has no context test. It is orthographically required
      in Persian, Arabic and Devanagari, so those documents stay hard-blocked. The
      criterion has to be script-based rather than pictographic, and no corpus is
      here to verify one against — parked as a known false-positive class rather than
      guessed at.

    736 tests pass (was 727).

    Downloads
  • v0.3.4 adf93e47fb

    v0.3.4 Stable

    ktg released this 2026-08-01 18:09:29 +00:00 | 46 commits to main since this release

    Denial-of-service fix on the INPUT path. Upgrade from 0.3.3. 0.3.3 swept
    the 83 lexicon patterns arm by arm and left every other table on 0.3.2's
    hand-written rows. Generalising the sweep over all eleven regex-bearing modules
    found three more quadratic patterns — two of them on the input path, one in
    sanitize, the first thing every ingested document touches. No disposition
    changes: recall was measured case by case and nothing was lost. Earlier tags
    are not moved.

    Fixed — three quadratic patterns, two on the input path

    Same class as everything 0.3.2 and 0.3.3 fixed: a run in front of a required
    literal, so crafted input that never supplies the literal makes every start
    position rescan the tail. Each exponent is read across four doublings, not from a
    two-point ratio.

    Pattern Crafted payload Measured @ 100 000 Exponent
    sanitize._HTML_COMMENT_RE <!-- × N 20.1 s 1.96–2.14
    active_content.URL_IN_TEXT_RE <a + A × N + > 12.99 s / 14.9 s 1.87–2.22
    okf._MD_LINK_RE [ × N 7.1 s 1.99–2.05

    These are worse than the 0.3.3 findings, and the reason is a separate finding of
    its own: MAX_SCAN_CHARS is applied in scan_lexicon and scan_output only.
    sanitize, neutralize, scan_active_content and the okf link graph accept
    input of any size, so there is no cap to extrapolate to. Now documented as a
    residual in docs/LIMITATIONS.md; extending the cap into the input path changes
    the contract for existing callers and is deliberately not done in a ReDoS patch.

    Each fix is the one the pattern's own shape allows — the 0.3.3 lesson that a fix
    choice must not be copied blindly from a neighbouring table:

    • sanitize drops the regex for a str.find scan, semantically identical to
      the lazy <!--.*?--> it replaces. Excluding < from the run would lose every
      comment containing markup (<!-- <b>x</b> --> is the ordinary case); bounding
      the run would be a one-line carrier bypass of the exact construct the stripper
      exists to remove. The module's own "no catastrophic backtracking" comment was
      wrong in the same way output's was before 0.3.2, and is corrected in place.
    • URL_IN_TEXT_RE bounds its scheme run to an RFC 3986 scheme ({0,63}).
      Bounding is safe here only because this is a defanger applied inside a tag
      already flagged active:raw-html, so padding shifts where the match starts
      rather than evading detection. A lookbehind killing interior start positions
      was measured too and rejected: it drops -http://evil.com and
      .http://x.com, a one-character evasion of the defanger. Bounded, the pattern
      runs in 0.185 s at the full 1 000 000-char cap.
    • okf._MD_LINK_RE excludes [, matching active_content.MD_LINK_RE
      exactly, including the nested-label trade already documented there.

    Changed — the sweep covers every regex surface, not one table

    docs/redos-sweep.py now sweeps 150 patterns across 11 tables (0.3.3 covered
    83 in one). The collector is mechanical on both axes so no one has to remember to
    list anything: it walks each module's namespace for compiled patterns, and it
    derives each pattern's call mode from the module source, because .sub() and
    .finditer() visit every start position where .match() cannot. A pattern
    reachable only through a helper parameter gets the worst mode, marked * — the
    fallback can over-measure but never miss.

    Two arm shapes the generator cannot express are pinned by hand as a result: a tag
    that closes around a long body (repeating-unit payloads never close it), and a
    run of plain characters carrying no anchor at all. The okf destination run gets
    no row on purpose: [^)\s]+ cannot fail, so a pin for it could never go red.

    A lexicon candidate flagged at ×2.8 measured linear across four doublings
    (exponent 0.96–1.03) — the near-noise-floor false flag the script's own docstring
    warns about, confirmed a second time.

    676 tests (+10). Coverage matrix unchanged at 128/128 caught, 6/6 gaps holding.

    Downloads