1
0
Fork 0
llm-ingestion-pipeline-secu.../docs
Kjell Tore Guttormsen c72e648a94
fix(sanitize): catch every invisible codepoint, exempt only on context
Variation-selector smuggling (A01) and Sneaky Bits (A02) gave 0 findings
on 1.4.1 and stayed in the sanitized text. The denominator is now the
frozen Unicode 18.0 set (DICP | Cf | Co, 141 674 codepoints); each one is
stripped and flagged unless a known legitimate use exempts it on context:
one VS15/16 after an emoji base, one ideographic selector after a CJK
ideograph, ZWNJ/ZWJ beside Arabic-script or Indic letters, LRM/RLM/ALM
in a document with RTL text, an Arabic number sign before a digit. A run
is always a carrier; private use is caught in full (operator decisions
2026-10-28). One predicate serves sanitize, the output gate and the
lexicon's word-split row, whose regex stays byte-identical with commons.

A01 and A02 are green; the FP populations and the pinned OKF corpus are
identical before and after.
2026-10-29 08:31:43 +01:00
..
threat-2026 test(threat-2026): one case per known 2025-2026 attack, red where not caught 2026-10-28 21:27:33 +01:00
2026-09-07-limitations-44-maaling.md docs(okf): measure the tags/description gap against the pinned SPEC and corpus 2026-09-06 23:58:02 +02:00
ADOPTION-BRIEF.md chore(release): 1.4.1 2026-09-23 12:46:32 +02:00
BRIEF.md chore(release): 1.4.1 2026-09-23 12:46:32 +02:00
catalog-census.py test(threat-2026): freeze the catalog-number census at v1.4.1 2026-10-28 21:27:31 +01:00
fp-sweep.py measure(fp): the upload door costs a human on 16-53% of benign documents 2026-08-10 21:32:56 +02:00
GATE-G-v1.md docs(gate): the surface claim covers okf too, and one branch of D4 is already a defect 2026-08-13 22:07:18 +02:00
LIMITATIONS.md fix(sanitize): catch every invisible codepoint, exempt only on context 2026-10-29 08:31:43 +01:00
OKF-INGESTION-BRIEF.md docs: version-sync + SECURITY/CONTRIBUTING + honest-limits (Session E) 2026-07-15 10:08:24 +02:00
PLAN-v1.md docs(plan): Session G landed, D4's rationale lives where the promise does, and the ninth surface was verified not bumped 2026-08-13 22:45:11 +02:00
PLAN.md fix(sanitize): catch every invisible codepoint, exempt only on context 2026-10-29 08:31:43 +01:00
rawhtml-census.py feat(active-content): raw HTML graded on carrier, and a tag naming no target is inert 2026-08-12 00:42:44 +02:00
redos-sweep.py docs(redos): the sweep now measures on the clock its own numbers justify 2026-08-18 16:56:01 +02:00
review-2026-07.md docs: reframe novelty claim to composite write-time contract (review MAJOR #3) 2026-07-15 09:22:38 +02:00
URL-SHAPE.md docs(url-shape): make the rule reconstructable, and record what three corpora measured 2026-07-27 08:56:24 +02:00