measure(fp): the upload door costs a human on 16-53% of benign documents
Every field measurement this repo had published was per URL. None answered
what a consumer actually feels: how often does an ordinary document fail to
persist unattended? Three benign populations, each against its own
denominator, run through screen_output under PRESET_USER_UPLOAD and counted
at document granularity:
vendor-harvest 98 of 185 (53.0%) non-WARN -- 64 fail-secure, 34 held
generated-notes 88 of 547 (16.1%) non-WARN -- 61 fail-secure, 27 held
reference-corpus 133 of 389 (34.2%) non-WARN -- 80 fail-secure, 53 held
The number is bad and ships as measured; PLAN-v1 committed to that in advance
("et roedt FP-resultat er like verdifullt"). The response is a documented
limitation, not a recalibration: moving the grading fires the locked
linkedin-studio notification promise, and the drivers are residuals
LIMITATIONS already concedes. Counted at each document's worst severity,
active:raw-html -- the MDX-component over-reach -- is a top driver in 52 of
vendor-harvest's 98 and 53 of generated-notes' 88; about ten per population
are genuinely injection-shaped text, which security-adjacent documentation
honestly contains.
Method traps closed rather than stepped in:
- The unit is in the number. Document-level rates are NOT comparable to the
URL-level 16/16, 28/28, 149/1694 above them, and the three rows are not
summable -- the 2400 != 2401 defect class one level up.
- The gate is the strict one. The trusted door WARNs every non-CRITICAL
finding, so it would have handed back a beautiful, meaningless near-zero;
it is printed as a footnote and labelled structurally blind.
- "not WARN" is only a risk statement while the default action map sends
exactly NONE and LOW to WARN. action_map became a supported override last
commit, so the equivalence is pinned in the suite and the sweep aborts if
it breaks.
- Ground truth for benign is provenance, not inspection, and says so.
- The populations are disjoint as documents but not independent as content:
184 of generated-notes' 547 are same-named derivatives of vendor-harvest.
Measured, not assumed, and the two rows read as one observation.
Also fixed: tests/test_wiring.py credited a consumer's capture store with
35 of 35 query-carrying URLs. That consumer retracted the number the next day
and re-measured 28 of 28 on the same 81-URL corpus. LIMITATIONS was corrected
then; the comment was not, so a retracted figure has been sitting beside a
live one since 07-27.
717 tests (was 716, none changed), coverage matrix 128/128 with 6/6 gaps
holding, every population swept twice with identical counts.
This commit is contained in:
parent
398eb7407b
commit
d1bff6047d
6 changed files with 358 additions and 3 deletions
|
|
@ -235,8 +235,14 @@ a green scan means safe content. The highest-impact items:
|
|||
- **Six documented gaps** the coverage matrix keeps honest: hex-wrapped secret
|
||||
egress, semantic poisoning, trusted-prose lone-HIGH, lexicon dedup (`count=1`),
|
||||
pure beaconing, and short opaque URL segments.
|
||||
- **The upload door is a review queue, not an auto-persist path — measured.** On
|
||||
three benign document populations, `PRESET_USER_UPLOAD` disposed **98 of 185**
|
||||
(53.0%), **88 of 547** (16.1%) and **133 of 389** (34.2%) documents to something
|
||||
other than WARN. Technical documentation is the expensive case: it is dense in the
|
||||
exact constructs the gate grades. Budget human review, or run a source you actually
|
||||
trust as trusted. Re-run it yourself with [`docs/fp-sweep.py`](docs/fp-sweep.py).
|
||||
|
||||
**Full list — 32 items, each with the mechanism, plus the out-of-scope boundary:**
|
||||
**Full list — 33 items, each with the mechanism, plus the out-of-scope boundary:**
|
||||
[`docs/LIMITATIONS.md`](docs/LIMITATIONS.md). Several carry field measurements from
|
||||
consumer corpora, including the false positives the URL-shape rule actually produces.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue