llm-ingestion-pipeline-security/docs/LIMITATIONS.md

Honest limitations

Conceding these plainly is itself a control — it prevents the false assurance that a green scan means safe content. The README carries a summary of the highest-impact items; this is the full list, each with the mechanism.

• Structural unsolvability at the text layer. Pattern/lexicon detection is bypassable in isolation; character-injection and novel phrasings evade it. The contract (tool-less transform, capability isolation, fail-secure) carries the security — the lexicon is defense-in-depth, not a wall.
• A lone HIGH finding in trusted prose disposes to WARN, not quarantine. Under PRESET_TRUSTED_SOURCE, trust-scaling downgrades a single HIGH to WARN, and one HIGH is not "compound" (escalation needs ≥2 findings at MEDIUM+). So a HIGH injection reproduced verbatim under a trusted policy persists with a WARN. This is by design: if your "trusted" sources can carry attacker-influenced text, run them as untrusted (or add a quarantine floor).
• The quarantine floor fires at MEDIUM+, and is a no-op under the shipped upload preset. Through 0.3.0 quarantine_default floored any finding to QUARANTINE_REVIEW. That premise ("a finding is the exception") broke when the active-content detector made every ordinary markdown link a finding, so 0.3.1 raised the floor to MEDIUM+. Under PRESET_USER_UPLOAD (untrusted) a MEDIUM already escalates on trust alone, so the floor still changes no outcome there; it is live only for a caller-defined trusted policy that opts into quarantine_default. Documented so the preset is not over-read: a LOW finding on an upload now disposes WARN.
• Semantic / factual poisoning is invisible to lexicon + entropy: a false claim in clean prose carries no suspicious token. Highest impact for a wiki. The grounding module ships only a SourceGroundingCheck seam — the deterministic core does not judge semantics; a [judge] implementation must be plugged in.
• Adversarial-ML evasion can survive normalization; tokenizer mismatch between scanner and model leaves gaps. Latent / dormant memory poisoning is not judgeable at write time.
• Dormant / broken-link injection in a linked corpus (e.g. an OKF bundle): a link to a not-yet-existing target passes a per-concept write-time scan clean — the payload is planted later, when that target is written. link_graph surfaces the dangling edge as the signal, but catching the payload needs cross-write re-scan over time (the caller's disposition call).
• OKF reserved files (index.md / log.md). In a received bundle these are legitimate structure, so mode-b import_bundle scans their body and frontmatter (an injection in a directory listing is caught) rather than path-rejecting the conformant bundle. A front-end materialising individual uploads keeps the opposite rule (allow_reserved=False): a reserved basename is a listing-shadow and refused.
• OKF frontmatter is a restricted grammar, and a one-key block-sequence item is silently misparsed. Gate T2 accepts a line-oriented subset deliberately — full YAML is a larger parse-attack surface than a write-time gate needs. Nested mappings and flow collections ([a, b], {k: v}) are rejected outright, which fails secure. All three routes to a mapping fail, each on a different rule — flow ({k: v}) on the disallowed value-start indicator, block (k:\n sub: v) on the nested-mapping check, and dotted keys (k.sub: v) on the key pattern — so the mapping class has no expressible form, rather than one form being preferable to another. What survives is scalars and flat lists of strings. The defect is between those two outcomes: a block sequence whose items carry exactly one key parses "successfully" into the wrong type — sources:\n - uri: https://e.com/a yields the string 'uri: https://e.com/a', not a mapping, while the same list with two keys per item hard-rejects. A pointer can therefore ride through in a key the resource allowlist never inspects (attester:\n - resource: attesters/sql_equality.py → WARN), whereas a top-level resource: with a relative path correctly fails secure. The shape is not conformant OKF, so a well-formed bundle will not produce it; a malformed or hostile one can, and mode-b import_bundle writes the merged concept verbatim. Note the three block-list shapes are not one case: flat scalars parse correctly, one key per item misparses silently, two keys per item hard-rejects.
• T2 constrains import, not emission. The frontmatter grammar runs on okf.import_bundle (door C) only — parse_frontmatter is referenced nowhere in the door A/B persist path, so frontmatter that fails secure on import passes screen_output unremarked. The grammar therefore bounds what a consumer can receive, never what a producer can emit. Verified identical on 0.2.0 and 0.3.1.
• Consequence: an OKF v0.2 concept cannot traverse the external-import path. Both of v0.2's backward-breaking migration targets are nested — timestamp → generated.at, and body # Citations → a sources block list of mappings — so a conformant v0.2 concept fails secure at the frontmatter gate. This is the correct direction but it is a compatibility wall, not a policy: v0.2 support requires a deliberate parse-safety decision about widening the grammar, and the dangling-or-substituted executor/ attester pointer question only becomes live once that decision is made.
• A persist gate cannot cover execution risk. OKF v0.2 introduces concepts whose purpose is to name code to be run (runtime, executor.resource, attester.resource). This library answers "is this safe to store"; executable code carries its risk at run. A file that is harmless to persist can be harmful to point at. Upstream defers the attester ABI and sandboxing to a future revision, so there is no runtime contract to gate against — the execution boundary is unowned across the stack rather than covered by anyone's roadmap, and no tightening of a write-time scanner would change that.
• A document that describes attacks is a false positive. Content documenting prompt-injection payloads (security notes, this project's own corpus) trips carrier-strip / fail-secure. At the text layer "about an attack" and "carrying an attack" are indistinguishable; such content needs a deliberate, explicitly escaped path, never a silent allow.
• Bilingual text trips the Cyrillic/Latin homoglyph rule. homoglyph:cyrillic-latin-mix (MEDIUM) flags a Latin letter adjacent to a Cyrillic look-alike, so genuine bilingual prose → MEDIUM → under untrusted → QUARANTINE_REVIEW — a real false positive for an inbox that expects multilingual content. A calibration fix is pending.
• Insider in-place edits by a trusted author are out of the untrusted-content threat model.
• Text-only. The core is text -> findings: it parses no files (no pypdf/python-docx/archive deps). Extract text first, then scan it with the high-untrust upload provenance. OCR-embedded instructions and multimodal stego are out of scope beyond the sanitizer's character-layer stripping.
• Uploaded files: only the extracted text is scanned. The dev-scoped OKF inbox showcase (tests/test_okf_inbox_uploads.py + tests/inbox_frontend.py; parsers in the [dev] extra, never core dependencies) reads .txt/.md/.csv/.docx/.pptx/.xlsx, folders and .zip, materializes an OKF bundle, then guards it. What survives extraction is out of scope: macros, OLE/embedded objects, OCR-needing images, font/render stego, encrypted files — the binary layer needs a separate scanner. The front-end owns the container threats it can see (zip-slip → path gate, zip-bomb → size cap, symlink refusal, CSV/XLSX formula-lead cells). .pdf is a deliberate concession: a top-level .pdf is refused as unsupported rather than half-scanned (a PDF parser is disproportionate for a dev showcase, and the OCR/stego it would smuggle is already out of scope). One known gap: the numeric -/+ CSV false positive (a typed XLSX numeric cell does not trip it).
• Lexicon findings are deduplicated by pattern id — count=1 and the first offset are reported, so a class matched across several channels collapses to one finding at its first location: a deliberate readability tradeoff.
• Active-content severity grades on URL shape, so a pure beacon is only LOW. Since 0.3.1 a URL that merely names a remote document (bare path, no query, no userinfo, no percent-escapes, no opaque segment) is LOW, and only a URL that can move bytes outward keeps HIGH/MEDIUM. The deliberate hole: ![x](https:// evil.test/pixel.png) on an attacker-controlled host still fetches when a renderer touches it, leaking reader IP, user-agent and timing. Grading the fetch itself would re-block every ordinary document, which is precisely the 0.3.0 regression this replaced — so beaconing is conceded, not covered.
• Short opaque URL segments slip through the same grading. Opacity is decided by entropy's primitives: base64 that decodes to text (≥20 chars), a hex id (≥32 chars), or Shannon entropy ≥4.4 at ≥24 chars. A shorter payload segment — https://evil.test/aGVsbG8gd29ybGQ — cannot be told from a name, because entropy is bounded by log2(length) at short lengths. Mitigation in depth, not in this detector: a literal credential in a URL is still caught by the LLM02 egress patterns in the same scan_output pass, whatever severity the carrier gets.
• Percent-escapes count as data-carrying — a %20 in a path is a false positive. An ordinary link with an encoded space grades as carrying and reaches QUARANTINE_REVIEW / FAIL_SECURE on an untrusted upload. Obfuscated encoding is a core exfil primitive and the ambiguous case is put on the review side deliberately; it is listed here because it is the same class of over-block that 0.3.1 fixed, in a rarer shape. It is a non-ASCII-language tax, and that is the finding. Three consumer corpora measured it (2026-07-25/26). The two English ones found zero — 0 of 347 external URLs in a 527-document vendor-docs corpus, 0 of 81 in a capture store. The third, a 389-file Norwegian/Microsoft reference corpus, found 10 distinct real escapes and every one of them Norwegian: %C3%B8 and %C3%A5 are simply ø and å in UTF-8, and legal/government sources turn titles into paths (lovdata.no/…/kap2/%C2%A710). "Accepted false positive" reads differently as "URLs in your own language grade above LOW". Our two zero-measuring corpora were both English; that was a sampling bias invisible from inside. Two sub-classes worth separating: {tenant}/{agent-id} template placeholders in API code samples encode to %7B/%7D (26 of that corpus's 36 hits — an artifact of harvesting, not of prose), and a slugger deriving filenames from titles would produce escapes systematically rather than incidentally — still unmeasured.
• A percent-escaped path also defeats tokenization, which feeds the entropy branch. % is not in the separator class, so NSMs%20Grunnprinsipper%20for%20IKT is one 34-character token where the same title with literal spaces would be four short ones. Measured at H=4.04 — under the 4.4 floor, and moot in practice because the % rule already disqualifies the URL. It is recorded because it means the length floor does less work in non-ASCII paths than the calibration assumed.
• Legitimate CDN content-asset ids trip the hex branch, permanently. A ≥32-char hex path segment is opaque by design, and several public CMSes mint asset URLs that way (measured: 9 distinct on regjeringen.no, ks.no, datatilsynet.no). This is the "legitimate build hash or doc id" case the calibration predicted, now confirmed present in the field. The class does not decay — it is how those systems generate URLs — so it is a standing false positive rather than a transient one. The branch is otherwise precise (no other false positives in 2401 distinct URLs) and stays.
• A non-empty query is graded as data-carrying — the over-block that actually occurs in the field. Three corpora have now measured it, and each found a disjoint benign population: (1) 16 of 16 query-carrying external URLs in a vendor-docs corpus were publisher-authored campaign tracking (utm_* on the publisher's own domains); (2) 28 of 28 in a capture store were content identity (?v=, ?channel_id=, ?all=true) where the parameter is the resource; (3) 149 of 1694 distinct learn.microsoft.com URLs in a reference corpus carried ?view=, Microsoft Learn's own documentation-version selector, plus 23 ?api-version=. No parameter-level remedy covers any two of them, let alone all three: an allowlist keyed on tracking-parameter names resolves (1) entirely and (2) and (3) not at all; stripping the query is lossless for (1), dereferences nothing for (2), and silently changes which document is cited for (3) — the worst failure mode of the three, because the result stays plausible. This is a settled constraint on any future middle tier, not a hypothesis. The cost is bounded — a query-carrying link is MEDIUM, so it disposes QUARANTINE_REVIEW under PRESET_USER_UPLOAD and WARN under PRESET_TRUSTED_SOURCE: held or warned, never hard-failed (pinned in tests/test_wiring.py, because both consumers inferred a hard block rather than running it). An image keeps HIGH, and that is the carrier where this would bite — the two corpora that reported a carrier breakdown contained zero remote images and the third did not report one, so the image row of this limitation remains unmeasured in the field.
• Raw HTML with a relative URL attribute is HIGH, though it can reach no attacker-controlled host. The markdown paths test for an external target before flagging; the raw-HTML path deliberately does not, because an active element needs no URL at all (an on*= handler executes on its own). That reasoning covers event handlers but over-reaches on the URL-attribute branch: an element outside the active name set carrying href="/en/agent-sdk/quickstart" — an internal doc route — grades HIGH. Measured on a vendor-docs corpus, where it lands on MDX components: <Card href="/…"> fires this way, and <Frame> fires on the name branch alone because names are lower-cased and frame is in the active set — legacy HTML framesets, which appear in essentially no modern documentation, while Frame is a common component name. Case is not an available discriminator: HTML is case-insensitive, so PascalCase cannot be treated as "component, not tag".
• Raw-HTML findings count end tags. </a> is active by name on its own, so a corpus census that counts only opening tags understates what this detector reports by roughly the ratio of closing to opening active tags (measured at 1.6× on one corpus). Severity and finding count are unaffected — the class collapses to one finding — but the count field is not a document count.
• URL fragments are not graded. A fragment is never sent to the server, so it cannot carry data to the host a renderer auto-fetches, and …/overview#section is the most common shape in real documentation. The residual: a clicked link to an attacker-controlled page can have its location.hash read by that page's script, so a fragment payload on a link (not an image) is uncovered.
• Secret egress: base64-wrapped is caught, hex-wrapped is not. The output gate decodes base64 blobs and re-scans the plaintext, so a base64-wrapped secret surfaces as decoded:egress:*. entropy exposes decoded plaintext for base64 only, so hex (and other encodings, or nested wraps) is a deliberate boundary — decode the transport layer first if you need it scanned.

The six documented gaps (tracked by the coverage matrix)

These are asserted to still hold by tests/test_coverage_matrix.py — a closed gap fails the test, forcing this doc to be updated:

1. Hex-wrapped secret egress — entropy decodes base64 only (above).
2. Semantic / factual poisoning — invisible to token analysis (above).
3. A lone HIGH in trusted prose → WARN — the §4.7 trust-scaling design (above).
4. Lexicon dedup (count=1) — first offset only, by design (above).
5. Pure beaconing — a bare-path remote image on a hostile host is LOW (above).
6. Short opaque URL segment (<24 chars) — below what entropy can resolve (above).

Out-of-scope (documented boundary)

Embedding/vector-layer defenses (OWASP LLM08, downstream of persist); multimodal steganography; query-time / runtime guardrails; semantic factuality verification.