docs: retract the "neither runtime misses an attack" claim, fix three counts
Four corrections across the two informative documents, all of them to
claims this repository made and none of them to a data file.
The retraction is the important one. lexicon-port-divergence.md said
neither runtime misses an attack: every witness payload still produced a
finding, so the divergence was only in which labels were raised. That was
wrong in method rather than in arithmetic. The payloads had been run
against the UNION of every pattern table this repository holds - 111 rules
across the lexicon, active-content.json and secret-egress.json - and the
hit that rescued them came from active-content.json, which is the Python
guard's table. llm-security has no active-content table at all. Unioning
two runtimes' tables and reading the result as a statement about each of
them turns combined reach into individual coverage.
Re-measured through llm-security's own entry point scanForInjection() at
b0de0ca - the whole engine, normalisation, homoglyph folding, rot13 and all
four arrays - all three witness payloads return found: false. Not a
different label set: no finding at all. Controls in the same run behave
normally, so the {0,256} window is a real evasion window and the <script>
pattern really does require a closing tag. llm-security reached the same
three results independently and has logged them as a v8.x task.
Second: one of the 13 divergences does not reach report level. On the
iframe witness the guard misses hybrid-xss:iframe-src but its
hybrid-xss:javascript-uri fires at the same severity and the same OWASP
anchor, so a reader of its report still sees the payload flagged. Both
sides were re-measured to confirm. The report-level number is 12; the 13
stands as the number that blocks conformance/, because a fixture names
labels.
Third: extraction-plan.md said six of the confusables have no entry in the
fold map. It is zero - all 13 are keys in the 28-entry map, and
carriers.json has said so correctly since it was written. The prose, not
the data, carried the error, and the number six most likely came from
running the comparison the other way (five fold-map Cyrillic keys are not
confusables). It also called the diverging sets deliberately distinct;
llm-security states the U+0456 / U+0445 divergence is drift, not design,
and has logged it for v8.x. Those two readings imply opposite things about
whether the sets should be expected to converge.
Fourth: the seed-sources section still asserted that sessions here have no
read access to llm-security and that content arrives only as a dump. The
operator changed that on 2026-08-09. It now records the read-only pinned-
commit boundary, the still-absolute prohibition on the sibling working copy
and on writing, and the reason the change matters - four attribution
defects caught so far, every one a transcription artefact.
Status blocks updated: all eight JSON artefacts now exist, seven verified
differentially and calibration.json still honestly verified: false; the
conformance blocker is half-cleared, with naming ratified and the 13
divergent patterns still unresolved.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SNMcqrfNyoLRQ7qXUFZnb9
This commit is contained in:
parent
47760d2264
commit
ff18768b71
3 changed files with 192 additions and 43 deletions
|
|
@ -40,10 +40,26 @@ Every JSON file carries a top-level `"version"` field. Every spec carries a
|
|||
|
||||
## v0.1.0 seed sources
|
||||
|
||||
`llm-security` is the canonical and richest source. **This repository's sessions have no
|
||||
read access to it** — content arrives only as an operator-supplied dump. Security-critical
|
||||
tables (homoglyph map, secret patterns, malware signatures) MUST come from real source
|
||||
data, never from recollection or inference.
|
||||
`llm-security` is the canonical and richest source. Security-critical tables (homoglyph map,
|
||||
secret patterns, malware signatures) MUST come from real source data, never from recollection
|
||||
or inference.
|
||||
|
||||
**The read boundary changed on 2026-08-09 (operator decision), and it matters to how the
|
||||
rows below should be read.** This section previously stated that sessions here have no read
|
||||
access and that content arrives only as an operator-supplied dump. Sessions may now **read
|
||||
`llm-security` read-only from its public remote at a pinned commit** —
|
||||
`ssh://git@git.fromaitochitta.com/open/llm-security.git` @ `b0de0ca`, verified as
|
||||
`origin/main` — by cloning into a scratchpad. The sibling working copy on this machine
|
||||
remains **absolutely off limits**, as does any form of writing.
|
||||
|
||||
The consequence is a method change, not just a convenience: files seeded from a dump are
|
||||
**differentially verified against the module** rather than trusted as transcriptions. That
|
||||
change has already caught four attribution defects — a re-escaping story that never happened
|
||||
in the lexicon, an inverted escape in one pattern, a severity citation pointing at a file with
|
||||
no severities in it, and two family names misremembered in a working note. Every one was a
|
||||
transcription artefact. Where a row below still says a dump was the delivery, the
|
||||
`provenance` block in the file itself records whether that dump has since been superseded by
|
||||
a direct read.
|
||||
|
||||
| Target | Seed source in `llm-security` (unless noted) |
|
||||
|---|---|
|
||||
|
|
@ -159,10 +175,28 @@ Recorded and deliberately **not** reconciled, in `cross_table_notes`: the reposi
|
|||
holds three overlapping Cyrillic sets and two overlapping zero-width sets, and none agree
|
||||
exactly. The zero-width carrier table includes U+00AD while the lexicon's pattern class does
|
||||
not; the lexicon's class contains U+0456 which `CYRILLIC_CONFUSABLES` lacks, and
|
||||
`CYRILLIC_CONFUSABLES` contains U+0445 which the class lacks; and six of the confusables
|
||||
have no entry in the fold map. The dump states the presence set and the fold map are
|
||||
deliberately distinct. The U+0456 / U+0445 divergence is reported to `llm-security` rather
|
||||
than fixed here.
|
||||
`CYRILLIC_CONFUSABLES` contains U+0445 which the class lacks.
|
||||
|
||||
**Two claims that stood here until 2026-08-09 were wrong, and both are corrected rather
|
||||
than quietly dropped.**
|
||||
|
||||
The first was a count: this paragraph said *six of the confusables have no entry in the fold
|
||||
map*. It is **zero**. All 13 members of `CYRILLIC_CONFUSABLES` are keys in the 28-entry fold
|
||||
map, and `codepoints/carriers.json` has said so correctly since it was written —
|
||||
`cross_table_notes.cyrillic.in_confusables_but_not_foldable` is an empty array. The prose,
|
||||
not the data, carried the error. The number six most likely came from the comparison run the
|
||||
other way: the fold map holds **five** Cyrillic keys that are not confusables (U+0423,
|
||||
U+0455, U+0456, U+0458, U+04CF), which `llm-security` confirms independently. A count
|
||||
asserted in prose beside a data file that contradicts it is the cheapest kind of defect to
|
||||
catch and the easiest to propagate, so the check is now: read the array, do not restate it.
|
||||
|
||||
The second was a characterisation. This paragraph said the sets are *deliberately distinct*.
|
||||
`llm-security` states plainly that the U+0456 / U+0445 divergence **is drift, not design**.
|
||||
They are not harmonising it now — that would be a behaviour change inside a
|
||||
behaviour-preservation release — and have logged it as a v8.x task on their side. So the
|
||||
divergence stays in the data exactly as it is, but it is recorded here as a known defect
|
||||
awaiting a fix upstream, not as an intentional distinction. Those two readings imply opposite
|
||||
things about whether a future consumer should expect the sets to converge.
|
||||
|
||||
### `signatures/secret-egress.json` — extracted 2026-08-09
|
||||
|
||||
|
|
@ -322,15 +356,33 @@ pick a winner.
|
|||
2. Every file in the layout above present and populated from verified seed data — or
|
||||
explicitly and visibly stubbed where the source was unavailable.
|
||||
|
||||
**Status 2026-08-09: unmet, deliberately, and this is why no tag has been cut.** Six of
|
||||
the nine data files are present and verified. Three are neither present nor stubbed:
|
||||
`signatures/malware-signatures.json` (seed data not delivered),
|
||||
`spec/decode-pipeline.md` (needs the decode implementation; a normative spec inferred
|
||||
from a data dump would carry a `Status: normative` marker and pass every convention
|
||||
check while asserting something nobody verified) and `conformance/` (blocked — see
|
||||
*Conformance sources* above: 105 of the guard's 134 cases are convertible, but a fixture
|
||||
cannot name a finding until commons owns a pattern id both runtimes map to, and 13
|
||||
patterns behave differently between the two ports).
|
||||
**Status 2026-08-09, second revision: still unmet, and still why no tag has been cut —
|
||||
but the remaining gap is two items, not three.** All **eight** JSON artefacts in the
|
||||
layout now exist. `signatures/malware-signatures.json` was the last, added once
|
||||
`knowledge/signatures.json` became readable at `b0de0ca` and reproduced from the file
|
||||
rather than from a dump.
|
||||
|
||||
Seven of the eight are differentially verified against their source; three of those
|
||||
(`lexicon`, `mapping/owasp-map`, `signatures/malware-signatures`) are verified against the
|
||||
source **module** at a pinned commit rather than against a transcription, and
|
||||
`schema/finding.schema.json` is closed against its producer. The eighth,
|
||||
`calibration/calibration.json`, still carries `verified: false` — its source arrived as a
|
||||
prose summary with no code behind it, so no differential check was possible, and the file
|
||||
names the checks that were not run rather than attaching a caveat to a pass.
|
||||
|
||||
Two items remain absent, and only one of them still blocks:
|
||||
|
||||
- `spec/decode-pipeline.md` — needs the decode implementation. A normative spec inferred
|
||||
from a data dump would carry a `Status: normative` marker and pass every convention
|
||||
check in this document while asserting something nobody verified.
|
||||
- `conformance/` — **half-unblocked.** The naming blocker is gone: the pattern id space
|
||||
is ratified by both runtimes as of 2026-08-09, so a fixture can name a finding. What
|
||||
remains is expected behaviour for the 13 patterns whose two ports genuinely differ.
|
||||
`llm-security` has decided not to change strategy inside a behaviour-preservation
|
||||
release, so those 13 stay divergent for now and are registered as known divergence per
|
||||
pattern rather than resolved. Of the guard's 134 coverage cases, 105 are convertible to
|
||||
static `input.txt` / `expected.json`; the other 29 assert a runtime's API surface, which
|
||||
this repository does not own.
|
||||
|
||||
They are absent rather than stubbed on purpose: an empty `conformance/` and a
|
||||
contentless normative spec would both *pass* the mechanical checks in this document
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue