# Honest limitations Conceding these plainly is itself a control — it prevents the false assurance that a green scan means safe content. The README carries a summary of the highest-impact items; this is the full list, each with the mechanism. - **Structural unsolvability at the text layer.** Pattern/lexicon detection is bypassable in isolation; character-injection and novel phrasings evade it. The *contract* (tool-less transform, capability isolation, fail-secure) carries the security — the lexicon is defense-in-depth, not a wall. - **A lone HIGH finding in trusted prose disposes to WARN, not quarantine.** Under `PRESET_TRUSTED_SOURCE`, trust-scaling downgrades a single HIGH to WARN, and one HIGH is not "compound" (escalation needs ≥2 findings at MEDIUM+). So a HIGH injection reproduced verbatim under a *trusted* policy persists with a WARN. This is by design: if your "trusted" sources can carry attacker-influenced text, run them as untrusted (or add a quarantine floor). - **The quarantine floor fires at MEDIUM+, and is a no-op under the shipped upload preset.** Through 0.3.0 `quarantine_default` floored *any* finding to QUARANTINE_REVIEW. That premise ("a finding is the exception") broke when the active-content detector made every ordinary markdown link a finding, so 0.3.1 raised the floor to MEDIUM+. Under `PRESET_USER_UPLOAD` (untrusted) a MEDIUM already escalates on trust alone, so the floor still changes no outcome there; it is live only for a caller-defined *trusted* policy that opts into `quarantine_default`. Documented so the preset is not over-read: a LOW finding on an upload now disposes WARN. - **Semantic / factual poisoning is invisible** to lexicon + entropy: a false claim in clean prose carries no suspicious token. **Highest impact for a wiki.** The `grounding` module ships only a `SourceGroundingCheck` *seam* — the deterministic core does not judge semantics; a `[judge]` implementation must be plugged in. - **Adversarial-ML evasion** can survive normalization; **tokenizer mismatch** between scanner and model leaves gaps. **Latent / dormant memory poisoning** is not judgeable at write time. - **Dormant / broken-link injection** in a linked corpus (e.g. an OKF bundle): a link to a not-yet-existing target passes a per-concept write-time scan clean — the payload is planted later, when that target is written. `link_graph` surfaces the *dangling* edge as the signal, but catching the payload needs cross-write re-scan over time (the caller's disposition call). - **OKF reserved files (`index.md` / `log.md`).** In a *received* bundle these are legitimate structure, so mode-b `import_bundle` scans their body and frontmatter (an injection in a directory listing is caught) rather than path-rejecting the conformant bundle. A front-end materialising individual uploads keeps the opposite rule (`allow_reserved=False`): a reserved basename is a listing-shadow and refused. - **OKF frontmatter is a restricted grammar: the mapping class has no expressible form.** Gate T2 accepts a line-oriented subset deliberately — full YAML is a larger parse-attack surface than a write-time gate needs. Nested mappings and flow collections (`[a, b]`, `{k: v}`) are *rejected outright*, which fails secure. **All four routes to a mapping fail, each on a different rule** — flow (`{k: v}`) on the disallowed value-start indicator, block (`k:\n sub: v`) on the nested-mapping check, dotted keys (`k.sub: v`) on the key pattern, and the inline second colon (`k: sub: v`) on the mapping-construct check — so the mapping *class* has no expressible form, rather than one form being preferable to another. What survives is scalars and flat lists of strings. **Two of those routes used to degrade into a string instead of failing, and that defect is closed in `1.1.0`**: a block-sequence item carrying exactly one key (`sources:\n - uri: https://e.com/a` yielded the *string* `'uri: https://e.com/a'`) and the inline second colon (`attester: resource: attesters/sql_equality.py`, which a real YAML parser refuses outright). Both parsed "successfully" into the wrong *type*, and a pointer parked in one rode through in a key the `resource` allowlist never inspects — mode-b `import_bundle` returned WARN and wrote the merged concept verbatim. Both now FAIL_SECURE at T2, before the allowlist is reached. **What closed is the type confusion, not pointer-smuggling as a class:** T3 still inspects `resource` and nothing else, so an honest *string* under another key rides through exactly as before — `attester: attesters/sql_equality.py` is WARN, while the same path under `resource:` FAIL_SECUREs. That is by design (the string is scanned like any other frontmatter value under T1) and it is not what `1.1.0` changed. **The boundary is where YAML puts it**, ground-truthed against PyYAML 6.0.3: `": "` and a trailing `":"` open a mapping and are refused; a colon carrying neither a space nor a line end (`domain:security`, `https://e.com:8443/a`) does not and still parses, as does a quoted scalar (`- "uri: x"`). Quotes are retained rather than stripped — a divergence from YAML that remains, pinned in `tests/test_okf.py`. - **T2 constrains import, not emission.** The frontmatter grammar runs on `okf.import_bundle` (door C) only — `parse_frontmatter` is referenced nowhere in the door A/B persist path, so frontmatter that fails secure on import passes `screen_output` unremarked. The grammar therefore bounds what a consumer can *receive*, never what a producer can *emit*. Verified identical on 0.2.0 and 0.3.1. - **Consequence: an OKF v0.2 concept cannot traverse the external-import path.** Both of v0.2's backward-breaking migration targets are nested — `timestamp` → `generated.at`, and body `# Citations` → a `sources` block list of mappings — so a conformant v0.2 concept fails secure at the frontmatter gate. This is the correct direction but it is a compatibility wall, not a policy: v0.2 support requires a deliberate parse-safety decision about widening the grammar, and the dangling-or-substituted `executor`/ `attester` pointer question only becomes live once that decision is made. - **A persist gate cannot cover execution risk.** OKF v0.2 introduces concepts whose purpose is to *name code to be run* (`runtime`, `executor.resource`, `attester.resource`). This library answers "is this safe to **store**"; executable code carries its risk at **run**. A file that is harmless to persist can be harmful to point at. Upstream defers the attester ABI and sandboxing to a future revision, so there is no runtime contract to gate against — the execution boundary is *unowned* across the stack rather than covered by anyone's roadmap, and no tightening of a write-time scanner would change that. - **A document that *describes* attacks is a false positive.** Content documenting prompt-injection payloads (security notes, this project's own corpus) trips carrier-strip / fail-secure. At the text layer "*about* an attack" and "*carrying* an attack" are indistinguishable; such content needs a deliberate, explicitly escaped path, never a silent allow. - **Bilingual text trips the Cyrillic/Latin homoglyph rule.** `homoglyph:cyrillic-latin-mix` (MEDIUM) flags a Latin letter adjacent to a Cyrillic look-alike, so genuine bilingual prose → MEDIUM → under untrusted → QUARANTINE_REVIEW — a real false positive for an inbox that expects multilingual content. **Conceded for 1.x: no fix is promised.** The rule fires on codepoint adjacency, which genuine bilingual prose produces as readily as a substitution attack does. Narrowing it is a calibration question, not an API one, so a fix may land in any 1.x release without breaking the contract — but none is scheduled, and a caller that ingests multilingual prose should raise its untrusted-tier threshold rather than wait for one. `SECURITY.md` lists this as a documented boundary, not a vulnerability. - **Insider in-place edits** by a trusted author are out of the untrusted-content threat model. - **Text-only.** The core is `text -> findings`: it parses no files (no `pypdf`/`python-docx`/archive deps). Extract text first, then scan it with the high-untrust upload provenance. OCR-embedded instructions and multimodal stego are out of scope beyond the sanitizer's character-layer stripping. - **Uploaded files: only the *extracted text* is scanned.** The dev-scoped OKF inbox showcase (`tests/test_okf_inbox_uploads.py` + `tests/inbox_frontend.py`; parsers in the `[dev]` extra, never core `dependencies`) reads `.txt`/`.md`/`.csv`/`.docx`/`.pptx`/`.xlsx`, folders and `.zip`, materializes an OKF bundle, then guards it. What survives extraction is **out of scope**: macros, OLE/embedded objects, OCR-needing images, font/render stego, encrypted files — the binary layer needs a separate scanner. The front-end owns the container threats it *can* see (zip-slip → path gate, zip-bomb → size cap, symlink refusal, CSV/XLSX formula-lead cells). **`.pdf` is a deliberate concession:** a top-level `.pdf` is *refused as unsupported* rather than half-scanned (a PDF parser is disproportionate for a dev showcase, and the OCR/stego it would smuggle is already out of scope). One known gap: the numeric `-`/`+` CSV false positive (a typed XLSX numeric cell does not trip it). - **Lexicon findings are deduplicated by pattern id** — `count=1` and the first offset are reported, so a class matched across several channels collapses to one finding at its first location: a deliberate readability tradeoff. - **Active-content severity grades on URL shape, so a pure *beacon* is only LOW.** Since 0.3.1 a URL that merely names a remote document (bare path, no query, no userinfo, no percent-escapes, no opaque segment) is LOW, and only a URL that can move bytes outward keeps HIGH/MEDIUM. The deliberate hole: `![x](https:// evil.test/pixel.png)` on an attacker-controlled host still *fetches* when a renderer touches it, leaking reader IP, user-agent and timing. Grading the fetch itself would re-block every ordinary document, which is precisely the 0.3.0 regression this replaced — so beaconing is conceded, not covered. - **Short opaque URL segments slip through the same grading.** Opacity is decided by `entropy`'s primitives: base64 that decodes to text (≥20 chars), a hex id (≥32 chars), or Shannon entropy ≥4.4 at ≥24 chars. A shorter payload segment — `https://evil.test/aGVsbG8gd29ybGQ` — cannot be told from a name, because entropy is bounded by `log2(length)` at short lengths. Mitigation in depth, not in this detector: a literal credential in a URL is still caught by the LLM02 egress patterns in the same `scan_output` pass, whatever severity the carrier gets. - **Percent-escapes count as data-carrying — a `%20` in a path is a false positive.** An ordinary link with an encoded space grades as carrying and reaches QUARANTINE_REVIEW / FAIL_SECURE on an untrusted upload. Obfuscated encoding is a core exfil primitive and the ambiguous case is put on the review side deliberately; it is listed here because it is the same *class* of over-block that 0.3.1 fixed, in a rarer shape. **It is a non-ASCII-language tax, and that is the finding.** Three consumer corpora measured it (2026-07-25/26). The two English ones found zero — 0 of 347 external URLs in a 527-document vendor-docs corpus, 0 of 81 in a capture store. The third, a 389-file Norwegian/Microsoft reference corpus, found 10 distinct real escapes and **every one of them Norwegian**: `%C3%B8` and `%C3%A5` are simply *ø* and *å* in UTF-8, and legal/government sources turn titles into paths (`lovdata.no/…/kap2/%C2%A710`). "Accepted false positive" reads differently as "URLs in your own language grade above LOW". **Two distinct axes, which an earlier revision of this file conflated.** For *third-party link* corpora — URLs an ingester collects from other people's sites — language does predict: the corpus of Norwegian legal/government sources carried escapes where two link corpora of mostly English-language domains carried none. For *generated paths* the predictor is not language but **slugger class**: a whitelist slugger (`[^a-z0-9]+ → "-"`) cannot emit an escape in any language, because it discards the character before anything encodes it — measured structurally, not statistically, by a consumer whose content is Norwegian and whose slugger output is `"Løkkene i produksjonslinja" → "l-kkene-i-produksjonslinja"`. A path built with `encodeURIComponent` produces escapes systematically the moment titles are non-ASCII. So non-ASCII language is a *confounder* for encode-vs-whitelist, and the slugger class is the testable thing at a consumer — a one-line code read, not a corpus census. **Both consumer sluggers we have now read are whitelists**, so their generated-path exposure is structurally zero rather than measured-zero; the second was read on 2026-07-31 and reduces the same Norwegian input to the same output. What this does *not* give us is the other half of the axis: we have never seen an `encodeURIComponent`-class slugger in the field, so "produces escapes systematically" above remains a prediction from the transform, not an observation. One sub-class worth separating: `{tenant}`/`{agent-id}` template placeholders in API code samples encode to `%7B`/`%7D` (26 of that corpus's 36 hits — an artifact of harvesting, not of prose). On the *generated-path* side those same placeholders cannot reach that shape at all: braces and `/` are outside the grammar, so `"{tenant}/{agent-id} mal"` reduces to `tenant-agent-id-mal`. - **A percent-escaped path also defeats tokenization, which feeds the entropy branch.** `%` is not in the separator class, so `NSMs%20Grunnprinsipper%20for%20IKT` is one 34-character token where the same title with literal spaces would be four short ones. Measured at H=4.04 — under the 4.4 floor, and moot in practice because the `%` rule already disqualifies the URL. It is recorded because it means the length floor does less work in non-ASCII paths than the calibration assumed. **A field re-measurement (2026-07-31) sharpens this by roughly 3x.** The highest-entropy *legitimate* token in a consumer's 2400-URL corpus is not the one above but a 78-character percent-escaped lovdata title, `Fra%20%C3%A5ndsverk%20…`, at H=4.301 — leaving **0.099 of headroom to the 4.4 floor, not 0.36**. Re-run here through the real `_URL_TOKEN_RE` and `shannon_entropy` rather than a reconstruction: our tokenizer does emit it as one 78-character token. Still moot per URL (`is_ordinary_url` is False on the `%` rule) and still not a reason to move the threshold — but the margin this bullet reports is much thinner than 4.04 implies. Note the corpus count: 2400 is a rebuilt harvester's, where the 2401 in the next bullet is the earlier run's. Same knowledge base, one URL apart. An earlier version of this note offered a second moving variable — a corpus that "grew from 389 files to 394" — and that explanation is **withdrawn**: the consumer corrected it the same day, and the counts reproduce here against their tree. 394 is every `.md` under `skills/`; 389 is the `references/**` path the measurement actually scoped to; the 5 `SKILL.md` files between them contributed no unique URLs. That is one snapshot counted two ways, not two snapshots. So only the script moved, and the one-URL gap stays *unexplained* — the original is gone from a scratchpad and nobody has chased it. The counts remain non-interchangeable: never summed, never quoted as one figure. - **Legitimate CDN content-asset ids trip the hex branch, permanently.** A ≥32-char hex path segment is opaque by design, and several public CMSes mint asset URLs that way (measured: 9 distinct on `regjeringen.no`, `ks.no`, `datatilsynet.no`). This is the "legitimate build hash or doc id" case the calibration predicted, now confirmed present in the field. The class does not decay — it is how those systems generate URLs — so it is a standing false positive rather than a transient one. The branch is otherwise precise (no other false positives in 2401 distinct URLs) and stays. - **A non-empty query is graded as data-carrying — the over-block that actually occurs in the field.** Three corpora have now measured it, and each found a *disjoint* benign population: **(1)** 16 of 16 query-carrying external URLs in a vendor-docs corpus were publisher-authored campaign tracking (`utm_*` on the publisher's own domains); **(2)** 28 of 28 in a capture store were content identity (`?v=`, `?channel_id=`, `?all=true`) where the parameter *is* the resource; **(3)** 149 of 1694 distinct `learn.microsoft.com` URLs in a reference corpus carried `?view=`, Microsoft Learn's own documentation-version selector, plus 23 `?api-version=`. **No parameter-level remedy covers any two of them, let alone all three:** an allowlist keyed on tracking-parameter names resolves (1) entirely and (2) and (3) not at all; stripping the query is lossless for (1), dereferences nothing for (2), and silently changes *which document is cited* for (3) — the worst failure mode of the three, because the result stays plausible. This is a settled constraint on any future middle tier, not a hypothesis. The cost is bounded — a query-carrying *link* is MEDIUM, so it disposes QUARANTINE_REVIEW under `PRESET_USER_UPLOAD` and WARN under `PRESET_TRUSTED_SOURCE`: held or warned, never hard-failed (pinned in `tests/test_wiring.py`, because both consumers inferred a hard block rather than running it). An *image* keeps HIGH, and that is the carrier where this would bite — the two corpora that reported a carrier breakdown contained zero remote images and the third did not report one, so the image row of this limitation remains unmeasured in the field. - **What `active:raw-html` still costs benign documentation is mostly ordinary HTML, not over-reach — and that residual is not a narrowing away.** Two over-reach classes were closed in 0.6.0: the URL-attribute branch now requires an *external* target (an element outside the active name set carrying `href="/en/agent-sdk/quickstart"` reaches no attacker-controlled host), and `` left the active *name* set (APIM's attribute-less `` means "run the inherited policy"; HTML's `` has its whole affordance in an `href` the attribute branch still catches). `` appears in **25 of reference-corpus's 389** documents — count it with `grep -rlE ''`, because a loose `grep '` placeholder, which is not a tag this detector ever fired on. Measured before and after in one session against one corpus state — `docs/rawhtml-census.py`, whose `PRODUCTION` row re-measures the shipped predicate rather than a hypothesis: reference-corpus 133 → **108** of 389, vendor-harvest 100 → **98** of 187, generated-notes 90 → **88** of 550. Against a ceiling of 26 / 38 / 41 (raw-HTML detection switched off entirely) that is **96% of the achievable reduction in reference-corpus and 5% in the two wiki corpora** — the over-reach was nearly the whole raw-html cost in APIM policy XML and nearly none of it in vendor documentation. **The classes had to be measured together.** Alone, the URL-attribute branch frees 3 documents in reference-corpus and `` frees 13; together they free 25. A document carrying one usually carries the other, so closing either alone leaves it blocked by its twin — one-at-a-time measurement understates both, and reading "frees 3" as "this over-reach is cheap" is how this document got the attribution wrong for three releases. **What remains is real HTML in vendor documentation**, flagged by the *name* branch, correctly: per wiki corpus `` 298 occurrences, `` 94, `` 63 — identical in both because 184 of their documents share ancestors. `` is the one arguable member: names are lower-cased and `frame` is in the active set for legacy HTML framesets, which appear in essentially no modern documentation, while `Frame` is a common MDX component name. Case is not an available discriminator: HTML is case-insensitive, so PascalCase cannot be treated as "component, not tag". Recovering the rest is **not** a further narrowing — it needs a carrier split and a new label (`active:raw-html-link`), because raw HTML has no ordinary form and the class collapses to one finding per document. **The external-target test now gates one more surface, and it is a literal one.** `_has_external_target` matches `^(?:[A-Za-z][A-Za-z0-9+.\-]*:|//)`, so a backslash-separated authority — `href="\\evil.example/x"` — reads as relative and now grades down, although WHATWG URL parsing normalizes backslashes to slashes for special schemes and a browser would resolve it externally. This is inherited, not introduced: the markdown paths have applied the same predicate since 0.3.1. It is recorded here rather than fixed because the fix belongs to the predicate, not to the raw-HTML branch that newly depends on it. **The scanner and the mutator no longer share a predicate.** Until 0.6.0 `neutralize` imported `is_active_tag` by name, so any narrowing moved the opt-in mutator too, and no test discriminated the two halves. They are now `is_active_tag` and `is_defangable_tag`; the mutator kept the broader behaviour deliberately, pinned by `tests/test_neutralize.py::test_mutator_still_defangs_what_the_scanner_now_lets_pass`. - **The carrier split TIGHTENS the trusted tier when both carrier classes are present.** 0.7.0 is sold as a loosening of the upload door, and on that door it is one. But splitting one class into two means a document carrying both an `` and an `` now emits *two* findings at MEDIUM+ where it emitted one, which trips the compound overlay (`>=2 findings at MEDIUM+ -> escalated one tier`). Such a document was WARN through 0.6.1 and is `quarantine_review` from 0.7.0 under `PRESET_TRUSTED_SOURCE`. On the trusted preset nothing was hard-failed to begin with, so this is the *only* direction the split can move it. Pinned by `tests/test_wiring.py::test_split_tightens_the_trusted_tier_when_both_carriers_are_present`, and `docs/rawhtml-census.py` now reports a `TIGHTENS` column against the previously-shipped row on both trust tiers — "frees N" without "tightens M" is a one-sided number. **Measured on all three populations, as shipped: 0 documents tightened, on both trust tiers — reference-corpus (389), vendor-harvest (187), generated-notes (552).** **That zero is empirical, not structural, and the census shows exactly how thin it is.** The split measured *alone* tightens **13** documents on the trusted tier in vendor-harvest and **14** in generated-notes. Adding the no-URL narrowing takes each of them back to 0: in these populations the document's second, HIGH-class carrier was itself a tag naming no target, which the narrowing makes inert, so the compound overlay never sees two findings. That is the census reporting a cancellation, not this repo proving one — a population whose second carrier is a real `` would still escalate, which is precisely the case `test_split_tightens_the_trusted_tier_when_both_carriers_are_present` constructs and pins. Read the zero as "not observed in any of the three populations, each counted against its own denominator", never as "cannot happen". - **Raw-HTML findings no longer count end tags, and that moved a published field.** Through 0.6.1 `` was active by name on its own, so `count` ran roughly 1.6× the opening-tag total (measured on one corpus) and a start/end pair counted 2. 0.7.0's no-URL narrowing makes an end tag inert — it names no target — so `count` is now the opening-tag total. A consumer reading `count` will see it *drop* for every document carrying ``, on a field whose meaning did not change. The finding count is unaffected: the class still collapses to one finding per class per document, and `count` was never a document count. - **"Clean" means *graded, no finding raised* — never *cleaned bytes* — and at one measured consumer's door, `warn` is the floor a document must clear to be persisted rather than rejected.** The word is the library's own: a WARN disposition with nothing to report carries the reason string `"clean: no findings"` (`disposition.py:264`), and this project has repeated that word in the tables it sends consumers. `screen_output` is a judgement API — its `DispositionResult` carries `assessment` / `disposition` / `max_severity` / `reasons`, with no sanitized-text field to read off it. Defanging lives in a separate, deliberate call — `neutralize` — that a caller must invoke itself; nothing upstream of that call transforms a byte. Measured against `llm-ingestion-okf`'s `0.7.0` pin (2026-08-13): `inbox.py:139` sets its persist floor to `warn`, and `inbox.py:323` persists anything carrying that disposition into the bundle; its adapter (`guard_adapter.py:70`) forwards the original extracted text, because nothing upstream ever handed it a transformed one. Four raw-HTML carrier forms the 0.7.0 no-URL narrowing grades inert — an `` with no `href`, a bare ``, ``, `