open/llm-security-commons
1
0
Fork 0
Code Issues Releases 1 Activity

docs: close three honesty gaps the validation pass missed

Browse source 
1. The layout block still described calibration.json as holding "entropy
   floors, scan caps, disposition ranks" — the same never-delivered content
   the seed-source row and the README row were already corrected for. Third
   copy, same defect. Grepped the repo to confirm no fourth: the remaining
   hits either describe the absence or belong to active-content.json's own
   Shannon-entropy floor, which is a real thing from a different source.
   Tightened the calibration log entry so it cannot be read as denying that.

2. Definition of Done item 2 claimed every file is present or "explicitly and
   visibly stubbed". Three are neither — they are absent and marked Planned.
   Item 2 now records that it is unmet, which three files are missing, why
   absent beats stubbed (an empty conformance/ and a contentless normative
   spec would both PASS the mechanical checks in this document while making
   the repo look more finished than it is), and that the tag decision is the
   operator's. The DoD is what a future session reads to decide whether
   v0.1.0 is done; it must not assert a bar the repo does not clear.

3. The lexicon's "83/83 compile in both runtimes" claim had been produced
   against the raw dump, not against the committed JSON — this repo's stated
   standard is that a fidelity claim comes from a command over the artifact.
   Re-run against lexicon/injection-lexicon.json: 83/83 in Python re, 83/83
   in Node bare and under `u`. Also confirmed the \uXXXX escapes survive the
   JSON round-trip with the right semantics — the zero-width class matches
   U+200B and rejects U+00AD, the Cyrillic class matches U+0430 and rejects
   U+0445. The claim stands as written.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FaYqid3mejFmd9ZHsiHgp3
This commit is contained in:
Kjell Tore Guttormsen 2026-08-09 21:20:06 +02:00
commit 93f7bc024f
1 changed files with 25 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

30
docs/extraction-plan.md
View file

@ -27,7 +27,7 @@ llm-security-commons/
signatures/secret-egress.json
signatures/malware-signatures.json
signatures/active-content.json # EchoLeak: MD image/link/refdef/autolink, data:, active HTML
calibration/calibration.json # entropy floors, scan caps, disposition ranks
calibration/calibration.json # risk-score tiers, verdict/band cutoffs, grade thresholds
mapping/owasp-map.json # prefix -> LLM/ASI/AST/MCP
schema/finding.schema.json # + SARIF & JSONL profiles. Status: normative
spec/decode-pipeline.md # normative RFC-2119 decode order
@ -244,10 +244,13 @@ specific checks that were skipped.
| Differential scoring over a corpus | **not run** — the formulas are engine and were not supplied in runnable form | — |
Recorded in the file as `not_supplied`, so the absence is visible rather than inferred: the
README and this plan both promised **entropy floors, scan caps and disposition ranks**. None
were delivered. Searched across the entire dump message: `entropy` 0 occurrences,
`disposition` 0, `rank` 0, `floor` 0. Both descriptions have been corrected to name what
arrived rather than what was expected.
README, this plan's seed-source table and this plan's layout block all described *this file*
as holding **entropy floors, scan caps and disposition ranks**. None of the three arrived in
any dump — searched across the entire dump message: `entropy` 0 occurrences, `disposition` 0,
`rank` 0, `floor` 0. All three descriptions have been corrected to name what arrived rather
than what was expected. (`signatures/active-content.json` does carry a Shannon-entropy floor,
extracted from the guard repository; that is a different file with a different source, and is
not what the calibration rows promised.)
The consequence is stated in the file and is worth repeating here, because it inverts this
repository's central rule: for every other file, a consumer that disagrees is wrong. For this
@ -260,6 +263,23 @@ consumer's bug.
gitignored.
2. Every file in the layout above present and populated from verified seed data — or
explicitly and visibly stubbed where the source was unavailable.
**Status 2026-08-09: unmet, deliberately, and this is why no tag has been cut.** Six of
the nine data files are present and verified. Three are neither present nor stubbed:
`signatures/malware-signatures.json` (seed data not delivered),
`spec/decode-pipeline.md` (needs the decode implementation; a normative spec inferred
from a data dump would carry a `Status: normative` marker and pass every convention
check while asserting something nobody verified) and `conformance/` (corpus form is an
open decision — the guard's coverage matrix is probe-based, not a static
input/expected table).
They are absent rather than stubbed on purpose: an empty `conformance/` and a
contentless normative spec would both *pass* the mechanical checks in this document
while making the repository look more finished than it is. They are named as **Planned**
in the README instead, and are not linked, so nothing points at a file that does not
exist. Whether v0.1.0 ships without a conformance corpus is an operator decision, not a
session one: the README's own pitch is that consumers can be held to the same answer on
the same input, and a tag with zero cases cannot deliver that.
3. All JSON well-formed, every data file carrying `"version"`, every spec carrying
`Status: normative`.
4. Tagged `v0.1.0` and pushed.