Kjell Tore Guttormsen
8d990e06d3
Records the reference artifacts Phase 5 swaps will be measured against, and
the gate that reads them. No extraction yet: this is the "before" picture,
and it had to land first or every later comparison would be confounded.
Typed test/ rather than feat/ deliberately - the only production change is
one widened export; the rest is gate, artifacts and generator.
Written failing-first (7 red on missing artifacts), then generated.
Three layers, because the plan's "assert .source/.flags of every regex" is
necessary but not sufficient:
1. regex records - .source/.flags off the COMPILED object. After a swap a
pattern is new RegExp(jsonString, flags), so the plan's named hazard
(JSON backslash-doubling on 83+18 regexes) is visible here and nowhere
else. Source-text comparison cannot see it.
2. table records - key/value digests. Most of what Phase 4 moves is not a
regex at all: HOMOGLYPH_MAP (x3, AS-IS), the typosquat tokens and the
four OWASP maps are char->char and string->string data. A regex-only
dump is blind to a broken homoglyph swap, i.e. to the bulk of the
payload. Operator decision: widen the dump.
3. file records - sha256 of the five moving-set sources. This dissolves
STATE's open question (how to enumerate every regex): it is complete by
construction, covering inline regexes in function bodies that no export
walk reaches, with no JS parser in a zero-dep repo. A lexical count
would have pinned a lie - severity.mjs scores 4 "regexes" that way and
exports none.
Both layers were proven to fire, not assumed to: mutating one HOMOGLYPH_MAP
entry reddens the table layer, and widening an inline regex inside
decodeHexEscapes (unreachable by any export walk) reddens the file layer.
HOMOGLYPH_MAP is now exported from string-utils.mjs. That export is a source
change the plan already flags as a surface hazard ("private tables become
loaded"), so it is pre-paid here rather than confounding the before/after.
Reference run: the 61 showcase payloads through the real hook entry points,
sequentially - array order is semantic and the plan forbids key-sorting, so
concurrency is removed rather than sorted away. 61/61 match expectation,
which also settles the plan's open assumption that payloads.json expectations
match current behaviour. Coverage is recorded, not assumed: 47/83 patterns,
with the other 36 listed by key so the gate never implies coverage it lacks.
Suite counts are per-file, each file run alone - a total is unattributable,
and the three known timing-sensitive files flake only under concurrency.
2045 pass / 0 fail across 91 files, matching the pre-existing count exactly.
The gate's own file is excluded (it reads the artifact the run produces) and
that exclusion is named in the artifact.
Suite: 2053/2053, 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BB4vXvwvtW4dxbPRd6vsez
472 lines
9.4 KiB
JSON
472 lines
9.4 KiB
JSON
{
|
|
"artifact": "golden-suite-counts",
|
|
"schema": 1,
|
|
"totals": {
|
|
"files": 91,
|
|
"pass": 2045,
|
|
"fail": 0
|
|
},
|
|
"excluded": [
|
|
{
|
|
"file": "tests/lib/golden-baseline.test.mjs",
|
|
"reason": "self-reference: this file reads suite-counts.json"
|
|
}
|
|
],
|
|
"files": [
|
|
{
|
|
"file": "tests/e2e/attack-chain.test.mjs",
|
|
"pass": 17,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/e2e/multi-session.test.mjs",
|
|
"pass": 9,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/e2e/scan-pipeline.test.mjs",
|
|
"pass": 24,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/hooks/post-mcp-verify.test.mjs",
|
|
"pass": 73,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/hooks/post-session-guard.test.mjs",
|
|
"pass": 75,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/hooks/pre-bash-destructive.test.mjs",
|
|
"pass": 53,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/hooks/pre-compact-scan.test.mjs",
|
|
"pass": 6,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/hooks/pre-edit-secrets.test.mjs",
|
|
"pass": 23,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/hooks/pre-install-supply-chain.test.mjs",
|
|
"pass": 31,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/hooks/pre-prompt-inject-scan.test.mjs",
|
|
"pass": 43,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/hooks/pre-write-pathguard.test.mjs",
|
|
"pass": 29,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/hooks/supply-chain-injection.test.mjs",
|
|
"pass": 1,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/hooks/update-check.test.mjs",
|
|
"pass": 10,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/audit-trail.test.mjs",
|
|
"pass": 8,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/bash-normalize.test.mjs",
|
|
"pass": 32,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/diff-engine-exact-pass.test.mjs",
|
|
"pass": 3,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/distribution-stats.test.mjs",
|
|
"pass": 13,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/doc-consistency.test.mjs",
|
|
"pass": 45,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/file-discovery.test.mjs",
|
|
"pass": 4,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/git-clone-gitattributes.test.mjs",
|
|
"pass": 8,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/git-clone-sandbox.test.mjs",
|
|
"pass": 32,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/injection-patterns.test.mjs",
|
|
"pass": 171,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/mcp-description-cache.test.mjs",
|
|
"pass": 34,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/output.test.mjs",
|
|
"pass": 27,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/policy-loader.test.mjs",
|
|
"pass": 14,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/severity.test.mjs",
|
|
"pass": 87,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/skill-registry-atomic.test.mjs",
|
|
"pass": 1,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/string-utils-hidden-unicode.test.mjs",
|
|
"pass": 21,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/string-utils-homoglyph.test.mjs",
|
|
"pass": 27,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/string-utils-tokens.test.mjs",
|
|
"pass": 16,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/string-utils.test.mjs",
|
|
"pass": 117,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/v8-env-removal.test.mjs",
|
|
"pass": 14,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/workflow-yaml-state.test.mjs",
|
|
"pass": 19,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/lib/yaml-frontmatter.test.mjs",
|
|
"pass": 6,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/ai-bom.test.mjs",
|
|
"pass": 13,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/ast-taint-scanner.test.mjs",
|
|
"pass": 12,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/attack-simulator.test.mjs",
|
|
"pass": 94,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/auto-cleaner-rce.test.mjs",
|
|
"pass": 2,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/auto-cleaner-traversal.test.mjs",
|
|
"pass": 2,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/auto-cleaner.test.mjs",
|
|
"pass": 140,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/bash-normalize-t5-t6.test.mjs",
|
|
"pass": 9,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/bash-normalize-t7-t9.test.mjs",
|
|
"pass": 12,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/benchmark.test.mjs",
|
|
"pass": 2,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/ci-integration.test.mjs",
|
|
"pass": 12,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/cli-wrapper.test.mjs",
|
|
"pass": 7,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/compliance-mapping.test.mjs",
|
|
"pass": 19,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/content-extractor-strip.test.mjs",
|
|
"pass": 7,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/dashboard.test.mjs",
|
|
"pass": 16,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/dep-token-overlap.test.mjs",
|
|
"pass": 7,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/dep.test.mjs",
|
|
"pass": 13,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/entropy-context.test.mjs",
|
|
"pass": 24,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/entropy-path-suppression.test.mjs",
|
|
"pass": 7,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/entropy.test.mjs",
|
|
"pass": 9,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/git-injection.test.mjs",
|
|
"pass": 1,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/git-reflog-reset.test.mjs",
|
|
"pass": 2,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/git.test.mjs",
|
|
"pass": 8,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/ide-extension-data.test.mjs",
|
|
"pass": 11,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/ide-extension-discovery.test.mjs",
|
|
"pass": 6,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/ide-extension-null-manifest.test.mjs",
|
|
"pass": 5,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/ide-extension-parser-entities.test.mjs",
|
|
"pass": 11,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/ide-extension-scanner.test.mjs",
|
|
"pass": 49,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/ide-extension-url.test.mjs",
|
|
"pass": 7,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/jetbrains-fetch.test.mjs",
|
|
"pass": 8,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/jetbrains-parser.test.mjs",
|
|
"pass": 30,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/knowledge-atlas.test.mjs",
|
|
"pass": 23,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/mcp-baseline-reset.test.mjs",
|
|
"pass": 10,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/mcp-live-inspect-stdout-cap.test.mjs",
|
|
"pass": 1,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/memory-poisoning-hex-dedupe.test.mjs",
|
|
"pass": 2,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/memory-poisoning.test.mjs",
|
|
"pass": 21,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/network.test.mjs",
|
|
"pass": 11,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/permission.test.mjs",
|
|
"pass": 7,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/posture-trifecta-mode.test.mjs",
|
|
"pass": 4,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/posture.test.mjs",
|
|
"pass": 57,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/reference-config.test.mjs",
|
|
"pass": 23,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/sarif-version.test.mjs",
|
|
"pass": 1,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/sarif.test.mjs",
|
|
"pass": 13,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/signature-scanner-custom-rules.test.mjs",
|
|
"pass": 4,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/signature-scanner.test.mjs",
|
|
"pass": 17,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/skill-scanner-narrative.test.mjs",
|
|
"pass": 11,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/supply-chain-recheck.test.mjs",
|
|
"pass": 32,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/taint-destructuring.test.mjs",
|
|
"pass": 19,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/taint-tracer.test.mjs",
|
|
"pass": 4,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/taint.test.mjs",
|
|
"pass": 10,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/toxic-flow-keyword-boundary.test.mjs",
|
|
"pass": 2,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/trigger-scanner.test.mjs",
|
|
"pass": 25,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/unicode-bom.test.mjs",
|
|
"pass": 4,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/unicode.test.mjs",
|
|
"pass": 9,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/vsix-fetch.test.mjs",
|
|
"pass": 38,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/vsix-sandbox.test.mjs",
|
|
"pass": 12,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/workflow-scanner.test.mjs",
|
|
"pass": 22,
|
|
"fail": 0
|
|
},
|
|
{
|
|
"file": "tests/scanners/zip-extract.test.mjs",
|
|
"pass": 25,
|
|
"fail": 0
|
|
}
|
|
]
|
|
}
|