open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Releases 4 Activity

measure(fp): the upload door costs a human on 16-53% of benign documents

Browse source 
Every field measurement this repo had published was per URL. None answered
what a consumer actually feels: how often does an ordinary document fail to
persist unattended? Three benign populations, each against its own
denominator, run through screen_output under PRESET_USER_UPLOAD and counted
at document granularity:

  vendor-harvest    98 of 185 (53.0%) non-WARN -- 64 fail-secure, 34 held
  generated-notes   88 of 547 (16.1%) non-WARN -- 61 fail-secure, 27 held
  reference-corpus 133 of 389 (34.2%) non-WARN -- 80 fail-secure, 53 held

The number is bad and ships as measured; PLAN-v1 committed to that in advance
("et roedt FP-resultat er like verdifullt"). The response is a documented
limitation, not a recalibration: moving the grading fires the locked
linkedin-studio notification promise, and the drivers are residuals
LIMITATIONS already concedes. Counted at each document's worst severity,
active:raw-html -- the MDX-component over-reach -- is a top driver in 52 of
vendor-harvest's 98 and 53 of generated-notes' 88; about ten per population
are genuinely injection-shaped text, which security-adjacent documentation
honestly contains.

Method traps closed rather than stepped in:

- The unit is in the number. Document-level rates are NOT comparable to the
  URL-level 16/16, 28/28, 149/1694 above them, and the three rows are not
  summable -- the 2400 != 2401 defect class one level up.
- The gate is the strict one. The trusted door WARNs every non-CRITICAL
  finding, so it would have handed back a beautiful, meaningless near-zero;
  it is printed as a footnote and labelled structurally blind.
- "not WARN" is only a risk statement while the default action map sends
  exactly NONE and LOW to WARN. action_map became a supported override last
  commit, so the equivalence is pinned in the suite and the sweep aborts if
  it breaks.
- Ground truth for benign is provenance, not inspection, and says so.
- The populations are disjoint as documents but not independent as content:
  184 of generated-notes' 547 are same-named derivatives of vendor-harvest.
  Measured, not assumed, and the two rows read as one observation.

Also fixed: tests/test_wiring.py credited a consumer's capture store with
35 of 35 query-carrying URLs. That consumer retracted the number the next day
and re-measured 28 of 28 on the same 81-URL corpus. LIMITATIONS was corrected
then; the comment was not, so a retracted figure has been sitting beside a
live one since 07-27.

717 tests (was 716, none changed), coverage matrix 128/128 with 6/6 gaps
holding, every population swept twice with identical counts.
This commit is contained in:
Kjell Tore Guttormsen 2026-08-10 21:32:56 +02:00
commit d1bff6047d
6 changed files with 358 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

56
docs/LIMITATIONS.md
View file

@ -233,6 +233,62 @@ items; this is the full list, each with the mechanism.
by roughly the ratio of closing to opening active tags (measured at 1.6× on one
corpus). Severity and finding count are unaffected — the class collapses to one
finding — but the `count` field is not a document count.
- **Measured, document by document: a large minority of *benign* documents do not
persist unattended at the upload door.** The bullets above bound single rules on
single URLs. This one bounds the thing a consumer actually feels — how often an
ordinary document costs a human — and the honest answer is *often*, on corpora of
technical documentation. Three benign populations, each reported against its own
denominator (`docs/fp-sweep.py`, run on the post-0.4.0 tree carrying the axis
separation, which renders every 0.4.0 disposition identically):
| population | provenance | n | disposed non-WARN |
|---|---|---|---|
| vendor-harvest | vendor-published doc pages, harvested verbatim | 185 | **98 (53.0%)** — 64 fail-secure, 34 held |
| generated-notes | model-written notes at their own persist gate | 547 | **88 (16.1%)** — 61 fail-secure, 27 held |
| reference-corpus | first-party authored reference material | 389 | **133 (34.2%)** — 80 fail-secure, 53 held |
**The unit is a document and the gate is the strict one:** `screen_output(doc,
PRESET_USER_UPLOAD)`, counting `disposition is not WARN`. WARN is the benign
outcome (persisted, with a note), so a *finding* is not a false positive — only a
document the pipeline cannot persist unattended is. Under the default action map
that count is equivalent to *assessed `ELEVATED` or worse*, and the equivalence is
pinned by `tests/test_corpus.py::test_the_published_fp_metric_is_a_risk_statement`
so an `action_map` override cannot silently redefine the published number.
**These are not comparable to the URL-level measurements above** (16 of 16, 28 of
28, 149 of 1694): different unit, different corpora, and they must never be
combined or read as an update to each other. **Nor are the three rows summable** —
different provenance, different denominators.
**What moved them is mostly residuals this document already concedes**, counted by
the labels at each document's *worst* severity (a histogram of every label present
would credit the over-block to whatever else happened to be in the document). In
vendor-harvest, `active:raw-html` is a top driver in **52 of the 98** — the MDX
component / relative-attribute over-reach two bullets up — `markdown:link-anchor-
injection` in 23, and only about ten documents are moved by genuinely
injection-shaped text, which is what security-adjacent documentation contains
honestly. Generated-notes tracks it almost exactly, as 184 shared ancestors imply
— `active:raw-html` in 53 of its 88, `markdown:link-anchor-injection` in 23 — so
read those two rows as one observation, not two. In reference-corpus, which shares
no upstream with either, the same shape holds with a different mix:
`active:markdown-link` 38 (largely the `?view=` documentation-version class from
the query bullet above), `active:data-uri` 36, `active:raw-html` 27,
`markdown:link-anchor-injection` 27, and eleven injection-shaped.
**Ground truth for "benign" is provenance, not inspection:** nobody hand-read
these corpora — each is benign by where it came from. A planted injection sitting
in a harvested corpus is scored here as a false positive, which is a real caveat
and not a formality.
**The populations are disjoint as documents but not independent as content:** 184
of generated-notes' 547 are same-named derivatives of vendor-harvest's 185, which
is most of why their driver labels agree. The third population shares no upstream
with either, and is the one whose provenance is first-party.
**The trusted door cannot produce this number and is printed only as a footnote**
(162 of 185, 527 of 547, 366 of 389 WARN there): every non-CRITICAL finding WARNs
under trust, which is the structural blindness that let the 0.3.0 active-content
regression pass a green suite. Read the contrast as the intended one — the same
corpus is cheap to persist from a source you trust and expensive from one you do
not.
**Every population was swept twice and reproduced its counts exactly**, and the
largest document in any of them is 362 kB — no document approached the 1 000 000-
character input cap, so truncation confounds nothing here.
- **URL fragments are not graded.** A fragment is never sent to the server, so it
cannot carry data to the host a renderer auto-fetches, and `…/overview#section` is
the most common shape in real documentation. The residual: a *clicked* link to an

233
docs/fp-sweep.py Normal file
View file

@ -0,0 +1,233 @@
"""False-positive sweep — run *benign* document populations through the persist
gate and count the ones it does not wave through.
WHAT THE NUMBER MEANS, EXACTLY. The unit is a **document**, not a URL, and the
metric is `screen_output(doc, PRESET_USER_UPLOAD).disposition is not WARN`. Every
word of that is load-bearing:
- **Document, not URL.** `docs/LIMITATIONS.md` also carries URL-level field
measurements (16 of 16, 28 of 28, 149 of 1694). Those are a different unit over
partly-overlapping corpora. A document rate is NOT comparable to them and must
never be combined with them, or quoted as an update to them.
- **The upload preset, not the trusted one.** `PRESET_TRUSTED_SOURCE` will hand
you a beautiful near-zero and mean nothing: every non-CRITICAL finding WARNs
under trust, which is exactly the structural blindness that let the 0.3.0
active-content regression ship through a green suite (see the docstring on
`tests/test_corpus.py::test_false_positive_is_not_blocked_on_the_upload_gate`).
The trusted door is printed as a footnote, never as the headline.
- **`screen_output`, not `_scan_input`.** The output gate is where
`active_content` lives; the input path never reaches it.
- **not WARN**, not "has findings". A finding is not a false positive — the
library reports and the pipeline decides (BRIEF design principle 4). WARN means
*persisted, with a note*, which is the benign outcome.
GROUND TRUTH for "benign" is **provenance, not inspection**: nobody hand-read
these documents. Each population is benign by where it came from — vendor-
published documentation, this machine's own generated notes, first-party authored
reference material. That is the only ground truth available at this scale, and it
is a real caveat, not a formality: an injected document sitting in a harvested
corpus would be scored as a false positive here.
POPULATIONS ARE NEVER SUMMED. Every population has its own denominator and its
own provenance; a pooled rate would be arithmetic over incommensurable things and
would inherit the `2400 != 2401` defect one level up. This script refuses to print
a total.
USAGE — corpus roots are arguments, never hardcoded; the corpora live in private
consumer repos and their paths must not reach a public mirror:
python docs/fp-sweep.py LABEL=/path/to/corpus [LABEL=/path ...]
[--ext=.md,.txt] [--include=/subtree/]
Each LABEL should name the population's *class* (`vendor-harvest`,
`generated-notes`, `reference-corpus`), not the repo it came from.
"""
from __future__ import annotations
import sys
from collections import Counter
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src"))
from llm_ingestion_guard import ( # noqa: E402
DEFAULT_ACTION_MAP,
PRESET_TRUSTED_SOURCE,
PRESET_USER_UPLOAD,
Disposition,
Risk,
Source,
__version__,
scan_output,
screen_output,
severity_rank,
)
from llm_ingestion_guard.calibration import RISK_RANK # noqa: E402
BENIGN = Disposition.WARN
"""The benign outcome: persisted, with a note. Anything else costs a human."""
def check_metric_is_a_risk_statement() -> None:
"""Fail loudly if "not WARN" has stopped meaning "assessed ELEVATED or worse".
The published number is a count of non-WARN documents, but what it *claims*
is a statement about assessed risk. The two are the same statement only while
the default action map sends exactly `NONE` and `LOW` to WARN. Re-map that
and the published number silently changes meaning with no test failing — the
method trap this script exists to stay out of. Mirrored in the suite by
`tests/test_corpus.py::test_the_published_fp_metric_is_a_risk_statement`.
"""
elevated = RISK_RANK[Risk.ELEVATED.value]
for risk in Risk:
benign = DEFAULT_ACTION_MAP[risk] is BENIGN
below = RISK_RANK[risk.value] < elevated
if benign != below:
raise SystemExit(
f"metric invalid: {risk.value} maps to "
f"{DEFAULT_ACTION_MAP[risk].value}; 'not WARN' no longer means "
"'assessed ELEVATED or worse' and the published rate would be "
"a different claim than the doc makes"
)
def documents(root: Path, exts: tuple[str, ...], include: str = "") -> list[Path]:
"""Every non-hidden file under ``root`` with a wanted extension.
``include`` is a substring the *relative* path must contain, so a population
can be scoped to a subtree (`--include=/references/`) without pretending a
differently-scoped count is the same population. Two scopings of one tree are
two counts, and the difference between them is exactly the kind of thing
`docs/LIMITATIONS.md` has had to correct in public before.
"""
files = []
for p in sorted(root.rglob("*")):
if not p.is_file() or p.suffix not in exts:
continue
rel = p.relative_to(root)
if any(part.startswith(".") for part in rel.parts):
continue
if include and include not in f"/{rel}":
continue
files.append(p)
return files
def measure(label: str, root: Path, exts: tuple[str, ...], include: str = "") -> dict:
paths = documents(root, exts, include)
dispositions: Counter[str] = Counter()
assessments: Counter[str] = Counter()
trusted_dispositions: Counter[str] = Counter()
labels: Counter[str] = Counter()
drivers: Counter[str] = Counter()
offenders: list[tuple[str, str, tuple[str, ...]]] = []
empty = 0
for path in paths:
text = path.read_text(encoding="utf-8", errors="replace")
if not text.strip():
empty += 1
continue
result = screen_output(text, PRESET_USER_UPLOAD)
dispositions[result.disposition.value] += 1
assessments[result.assessment.value] += 1
trusted_dispositions[
screen_output(text, PRESET_TRUSTED_SOURCE).disposition.value
] += 1
if result.disposition is not BENIGN:
findings = scan_output(text, source=Source.OUTPUT).findings
found = tuple(sorted({f.label for f in findings}))
labels.update(found)
# What actually moved this document: the labels at its *worst*
# severity. A histogram of every label present would credit the
# over-block to whatever else happened to be in the document, which
# is how a residual gets blamed on the lexicon.
worst = max((severity_rank(f.severity) for f in findings), default=-1)
drivers[" + ".join(sorted({
f.label for f in findings if severity_rank(f.severity) == worst
})) or "(no findings)"] += 1
offenders.append((path.name, result.disposition.value, found))
n = sum(dispositions.values())
return {
"label": label,
"n": n,
"empty": empty,
"dispositions": dispositions,
"assessments": assessments,
"trusted": trusted_dispositions,
"labels": labels,
"drivers": drivers,
"offenders": offenders,
"non_warn": n - dispositions[BENIGN.value],
}
def report(m: dict) -> None:
n, non_warn = m["n"], m["non_warn"]
rate = f"{non_warn / n:.1%}" if n else "n/a"
print(f"\n## {m['label']} — {non_warn} of {n} documents disposed non-WARN ({rate})")
if m["empty"]:
print(f" ({m['empty']} empty file(s) skipped — no document, no verdict)")
print(" upload gate :", dict(m["dispositions"]))
print(" assessment :", dict(m["assessments"]))
print(" trusted gate :", dict(m["trusted"]), " <- footnote only, structurally blind")
if m["drivers"]:
print(" what MOVED them (labels at each document's worst severity):")
for label, count in m["drivers"].most_common():
print(f" {count:5d} {label}")
if m["labels"]:
print(" what was merely present on them:")
for label, count in m["labels"].most_common():
print(f" {count:5d} {label}")
for name, disposition, found in m["offenders"][:10]:
print(f" - {name}: {disposition} via {', '.join(found) or '(no labels)'}")
if len(m["offenders"]) > 10:
print(f" ... and {len(m['offenders']) - 10} more")
def main() -> None:
argv = sys.argv[1:]
exts = (".md", ".txt")
include = ""
rest = []
for arg in argv:
if arg.startswith("--ext="):
exts = tuple(e if e.startswith(".") else f".{e}"
for e in arg.split("=", 1)[1].split(","))
elif arg.startswith("--include="):
include = arg.split("=", 1)[1]
else:
rest.append(arg)
if not rest:
print(__doc__)
raise SystemExit(2)
check_metric_is_a_risk_statement()
print(f"llm-ingestion-guard {__version__} — persist gate, PRESET_USER_UPLOAD")
print("metric: disposition is not WARN (== assessed ELEVATED or worse)")
print(f"extensions: {', '.join(exts)}")
measurements = []
for spec in rest:
if "=" not in spec:
raise SystemExit(f"expected LABEL=PATH, got {spec!r}")
label, _, path = spec.partition("=")
root = Path(path).expanduser()
if not root.is_dir():
raise SystemExit(f"{label}: {root} is not a directory")
measurements.append(measure(label, root, exts, include))
for m in measurements:
report(m)
print("\n---")
print("Populations are reported separately by construction. They have "
"different\nprovenance and different denominators; a pooled rate would "
"be arithmetic over\nincommensurable things. This script prints no total.")
if __name__ == "__main__":
main()