open/llm-security-commons
1
0
Fork 0
Code Issues Releases 1 Activity
llm-security-commons/conformance
History
Kjell Tore Guttormsen 49e1e79807 feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 
The corpus was blocked on a decision nobody had to take. The 13 divergent
lexicon patterns were measured on witness inputs -- an attribute run padded
past 256 characters, an interior '<', an unclosed <script> -- and the corpus
payloads contain none of those shapes. Run through both runtimes' public
entry points, all 83 patterns produce identical lexicon finding sets, the 13
included. No winner picked, because the question was never reachable from
these inputs.

Measured at each runtime's entry point (scanForInjection() at b0de0ca,
scan_output(source=OUTPUT) at 0bf0729), never at a rebuilt regex table --
the layer mistake the divergence document already had to retract once.

- conformance/<case>/{input.txt,expected.json} x83, plus manifest.json
- spec/conformance-corpus.md, normative: bytes not text, id-only findings,
  exact-within-scope, and observed_out_of_scope as evidence not expectation

Verified by a harness that does not share the generator's knowledge: reads
only the case directories, re-runs both runtimes, checks every field
including digests -- 83 cases, 0 failures. Severity agrees with what commons
publishes 83/83. Deleting the middle third of each input breaks 76 of 83
expectations; the 7 survivors are the shortest payloads, a weak mutation
rather than a weak fixture, and are recorded as such.

Scope is 83 and not 94 for a different reason than expected: the 11
non-lexicon convertible cases have no ratified cross-runtime finding id, and
writing them would mint a contract unilaterally in the same stroke as the
tag. Named in manifest.json under scope_planned.

Docs corrected in place rather than edited away: extraction-plan and
lexicon-port-divergence both claimed the 13 blocked the corpus.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WhXDL82FRrQWedEmUg12Pj
2026-08-10 04:40:56 +02:00
..
config__bypass-safety feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
config__disable-filtering feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
config__ignore-claudemd feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
config__override-system-prompt feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
data-uri__executable feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
evasion__academic-context feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
evasion__educational feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
evasion__fictional feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
evasion__hypothetical feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
evasion__red-team feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
evasion__research-purpose feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
evasion__security-audit-sim feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
evasion__test-pretense feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
evasion__thought-experiment feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
hidden-comment__agent-directed feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
hitl-trap__approve-immediately feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
hitl-trap__brevity-skip feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
hitl-trap__details-omitted feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
hitl-trap__just-approve feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
hitl-trap__no-need-review feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
hitl-trap__scope-minimization feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
hitl-trap__suppress-review feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
hitl-trap__too-long feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
homoglyph__cyrillic-latin-mix feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
html-obfuscation__aria-label feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
html-obfuscation__display-none feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
html-obfuscation__offscreen feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
html-obfuscation__visibility-hidden feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
html-obfuscation__zero-font feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
html-obfuscation__zero-opacity feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
html-obfuscation__zero-size-overflow feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
hybrid-p2sql__inject-sql feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
hybrid-p2sql__sql-inject feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
hybrid-recursive__conditional-response feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
hybrid-recursive__inject-output feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
hybrid-xss__event-handler feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
hybrid-xss__iframe-src feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
hybrid-xss__javascript-uri feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
hybrid-xss__script-tag feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
identity__new-directive feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
identity__new-role feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
identity__no-longer feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
identity__pretend feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
identity__you-are-now feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
indirect__direct-address feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
indirect__instruction-to-ai feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
indirect__trigger-based feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
leetspeak__disregard feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
leetspeak__forget-instructions feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
leetspeak__ignore-previous feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
leetspeak__override feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
markdown__link-anchor-injection feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
markdown__link-ref-comment feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
multi-lang__french feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
multi-lang__german feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
multi-lang__spanish feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
nl-indirection__download-execute feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
nl-indirection__extract-exfiltrate feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
nl-indirection__fetch-execute feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
nl-indirection__read-dotfile feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
nl-indirection__send-external feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
normalization__claudemd-conflict feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
normalization__prioritize-user feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
normalization__skip-confirmation feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
normalization__user-precedence feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
override__disregard feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
override__forget-instructions feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
override__ignore-previous feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
override__override-instructions feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
override__prior-constraints feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
override__suspend-constraints feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
reframing__actual-task feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
reframing__real-purpose feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
reframing__should-actually-do feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
spoofed-header__hash-system feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
spoofed-header__inst feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
spoofed-header__instructions feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
spoofed-header__pipe-system feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
spoofed-tag__system-close feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
spoofed-tag__system-open feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
sub-agent__delegate-bypass feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
sub-agent__spawn-dangerous feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
unicode__zero-width-in-word feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00
manifest.json feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement 2026-08-10 04:40:56 +02:00