docs(llm-security): v8 Phase 2 — B10 docs consistency, counts pinned by test
Extends tests/lib/doc-consistency.test.mjs with 15 cases that derive every inventory count from source instead of trusting prose. Each count has one stated derivation; a doc surface that disagrees now fails the suite. Counts corrected (all were wrong before the test existed): - orchestrated scanners: docs said 10 (README, ci-cd-guide), CLAUDE.md said 12, the synthesizer agent said 9 — scan-orchestrator registers 14 - total scanners: README badge + 3 prose sites said 23; the counting rule in docs/scanner-reference.md (14 orchestrated + 8 standalone) yields 22 - knowledge files: README badge + prose said 22; knowledge/ holds 23 - output.mjs finding() prefix JSDoc listed 10 of the 17 prefixes actually passed to it (missing IDE, MCI, MEM, PST, SCR, TFA, WFL) - norwegian-context.md said "8 hooks, 10 scanners" -> 9 and 14 - ci-cd-guide "what gets scanned" table listed 10 of 14 rows; adds workflow, trigger abuse, signature, AST taint Two plan items changed after verifying against ground truth: - CLAUDE.md's synthesizer "(12 scanners)" was not a deliberate subset; the agent file itself claimed 9. Both bumped to 14. - compliance-mapping.md's "13 posture categories" is substantively correct — its matrix has exactly 13 data rows, and categories 14-16 are governance consumers of the file, not rows in it. The planned 13->16 bump would have made the document false. Wording clarified to "code-level" instead, and the test now pins row count against the stated claim. Framework currency (both verified against primary reporting): - EU AI Act: Digital Omnibus (EP 2026-06-16, Council 2026-06-29) deferred the high-risk obligations behind Art. 9/15 to 2027-12-02 (Annex III) and 2028-08-02 (Annex I); transparency still applies from 2026-08-02 - OWASP Agentic AI Top 10 labelled as the 2026 edition Also: CLAUDE.md Distribution section rewritten monorepo -> polyrepo (each plugin is its own repo; the catalog pins url + ref per plugin), and current-state test counts synced 2013 -> 2034. Release-note paragraphs keep their historical numbers. No scanner, hook, or command behaviour changes. Suite 2034/2034. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wt4YQGoXwRja5K2Zmv8RZE
This commit is contained in:
parent
ff4d8e8a31
commit
0f1be986d0
8 changed files with 287 additions and 22 deletions
|
|
@ -1,6 +1,6 @@
|
||||||
# LLM Security Plugin (v7.8.3)
|
# LLM Security Plugin (v7.8.3)
|
||||||
|
|
||||||
Security scanning, auditing, and threat modeling for Claude Code projects. 5 frameworks: OWASP LLM Top 10, Agentic AI Top 10 (ASI), Skills Top 10 (AST), MCP Top 10, AI Agent Traps (DeepMind). 2013+ unit, integration, and end-to-end tests (`tests/e2e/` covers the multi-hook attack chain, multi-session state simulation, and the full scan-orchestrator pipeline); mutation-testing coverage not published.
|
Security scanning, auditing, and threat modeling for Claude Code projects. 5 frameworks: OWASP LLM Top 10, Agentic AI Top 10 (ASI, 2026 edition), Skills Top 10 (AST), MCP Top 10, AI Agent Traps (DeepMind). 2034+ unit, integration, and end-to-end tests (`tests/e2e/` covers the multi-hook attack chain, multi-session state simulation, and the full scan-orchestrator pipeline); mutation-testing coverage not published.
|
||||||
|
|
||||||
Release notes for v7.0.0 → v7.8.2: see `docs/version-history.md` — read on demand.
|
Release notes for v7.0.0 → v7.8.2: see `docs/version-history.md` — read on demand.
|
||||||
|
|
||||||
|
|
@ -51,7 +51,7 @@ Release notes for v7.0.0 → v7.8.2: see `docs/version-history.md` — read on d
|
||||||
| `mcp-scanner-agent` | 5-phase MCP server analysis | opus |
|
| `mcp-scanner-agent` | 5-phase MCP server analysis | opus |
|
||||||
| `posture-assessor-agent` | Full audit narrative (posture-scanner.mjs handles quick mode) | opus |
|
| `posture-assessor-agent` | Full audit narrative (posture-scanner.mjs handles quick mode) | opus |
|
||||||
| `threat-modeler-agent` | STRIDE x MAESTRO interview | opus |
|
| `threat-modeler-agent` | STRIDE x MAESTRO interview | opus |
|
||||||
| `deep-scan-synthesizer-agent` | Scanner JSON → human-readable report (12 scanners) | opus |
|
| `deep-scan-synthesizer-agent` | Scanner JSON → human-readable report (14 scanners) | opus |
|
||||||
| `cleaner-agent` | Semi-auto remediation proposals | opus |
|
| `cleaner-agent` | Semi-auto remediation proposals | opus |
|
||||||
|
|
||||||
## Hooks (9)
|
## Hooks (9)
|
||||||
|
|
@ -90,12 +90,14 @@ Post-clone: size check (100MB max), cleanup guarantee (temp dir + evidence file
|
||||||
|
|
||||||
## Distribution
|
## Distribution
|
||||||
|
|
||||||
This plugin lives in the `ktg-plugin-marketplace` monorepo at `https://git.fromaitochitta.com/open/ktg-plugin-marketplace` under `plugins/llm-security/`. It is not published as a standalone repo — users install it via the Claude Code marketplace mechanism:
|
This plugin is its own repository at `https://git.fromaitochitta.com/open/llm-security`. It is distributed through the `ktg-plugin-marketplace` catalog, which is a **polyrepo**: the catalog (`catalog/.claude-plugin/marketplace.json`) holds no plugin source, only a `url` + `ref` pin per plugin, and each plugin repo is released independently by tag. Users install via the Claude Code marketplace mechanism:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
claude plugin marketplace add https://git.fromaitochitta.com/open/ktg-plugin-marketplace.git
|
claude plugin marketplace add https://git.fromaitochitta.com/open/ktg-plugin-marketplace.git
|
||||||
```
|
```
|
||||||
|
|
||||||
|
A version bump is therefore two-sided: tag `vX.Y.Z` in this repo, then bump the catalog's `ref` to the same tag.
|
||||||
|
|
||||||
Issues, bug reports, and security disclosures all route to the marketplace repo.
|
Issues, bug reports, and security disclosures all route to the marketplace repo.
|
||||||
|
|
||||||
## State
|
## State
|
||||||
|
|
|
||||||
20
README.md
20
README.md
|
|
@ -10,13 +10,13 @@
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
A Claude Code plugin that provides security scanning, auditing, and threat modeling for agentic AI projects. Built on [OWASP LLM Top 10 (2025)](https://genai.owasp.org/llm-top-10/), [OWASP Agentic AI Top 10 (ASI01-ASI10)](https://genai.owasp.org/agentic-ai/), OWASP Skills Top 10 (AST01-AST10), MCP Top 10, and the [AI Agent Traps](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6372438) taxonomy (Google DeepMind, 2025), grounded in published research from ToxicSkills, ClawHavoc, MCPTox, Pillar Security, Invariant Labs, GHSL Security Lab, and Operant AI.
|
A Claude Code plugin that provides security scanning, auditing, and threat modeling for agentic AI projects. Built on [OWASP LLM Top 10 (2025)](https://genai.owasp.org/llm-top-10/), [OWASP Agentic AI Top 10 (ASI01-ASI10, 2026 edition)](https://genai.owasp.org/agentic-ai/), OWASP Skills Top 10 (AST01-AST10), MCP Top 10, and the [AI Agent Traps](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6372438) taxonomy (Google DeepMind, 2025), grounded in published research from ToxicSkills, ClawHavoc, MCPTox, Pillar Security, Invariant Labs, GHSL Security Lab, and Operant AI.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -101,7 +101,7 @@ flowchart TB
|
||||||
H4["PreCompact<br/>transcript scan"]
|
H4["PreCompact<br/>transcript scan"]
|
||||||
end
|
end
|
||||||
|
|
||||||
subgraph Scanning["Deterministic analysis — 23 scanners"]
|
subgraph Scanning["Deterministic analysis — 22 scanners"]
|
||||||
direction LR
|
direction LR
|
||||||
S1["UNI · ENT · PRM · DEP<br/>TNT · GIT · NET · MEM · SCR · TFA"]
|
S1["UNI · ENT · PRM · DEP<br/>TNT · GIT · NET · MEM · SCR · TFA"]
|
||||||
S2["WFL workflow scanner"]
|
S2["WFL workflow scanner"]
|
||||||
|
|
@ -144,8 +144,8 @@ Each layer is independent. A failure in one (e.g. an injection that slips past t
|
||||||
|---------|-------------|
|
|---------|-------------|
|
||||||
| `/security` | Router with quick-start guide |
|
| `/security` | Router with quick-start guide |
|
||||||
| `/security scan [path\|url]` | Supply-chain gate — ALLOW/WARNING/BLOCK verdict on skills, MCP servers, directories, or remote repos |
|
| `/security scan [path\|url]` | Supply-chain gate — ALLOW/WARNING/BLOCK verdict on skills, MCP servers, directories, or remote repos |
|
||||||
| `/security scan [path\|url] --deep` | Adds 10 deterministic scanners on top of the LLM agents |
|
| `/security scan [path\|url] --deep` | Adds 14 deterministic scanners on top of the LLM agents |
|
||||||
| `/security deep-scan [path]` | Run only the 10 orchestrated deterministic scanners. Supports `--fail-on <severity>`, `--compact`, `--format sarif`, `--output-file <path>` |
|
| `/security deep-scan [path]` | Run only the 14 orchestrated deterministic scanners. Supports `--fail-on <severity>`, `--compact`, `--format sarif`, `--output-file <path>` |
|
||||||
| `/security audit` | Full project audit, A-F grade, prioritized action plan |
|
| `/security audit` | Full project audit, A-F grade, prioritized action plan |
|
||||||
| `/security plugin-audit [path\|url]` | Plugin trust assessment with Install/Review/Do Not Install verdict |
|
| `/security plugin-audit [path\|url]` | Plugin trust assessment with Install/Review/Do Not Install verdict |
|
||||||
| `/security mcp-audit [--live]` | Audit installed MCP server configs (`--live` adds runtime inspection) |
|
| `/security mcp-audit [--live]` | Audit installed MCP server configs (`--live` adds runtime inspection) |
|
||||||
|
|
@ -224,7 +224,7 @@ All hooks are Node.js `.mjs` for cross-platform compatibility (macOS, Linux, Win
|
||||||
|
|
||||||
## Deterministic scanners
|
## Deterministic scanners
|
||||||
|
|
||||||
23 scanners. Zero external dependencies. All output JSON.
|
22 scanners. Zero external dependencies. All output JSON.
|
||||||
|
|
||||||
### Orchestrated (14) — run via `node scanners/scan-orchestrator.mjs <target>` or `/security deep-scan`
|
### Orchestrated (14) — run via `node scanners/scan-orchestrator.mjs <target>` or `/security deep-scan`
|
||||||
|
|
||||||
|
|
@ -424,8 +424,8 @@ These gaps are surfaced advisorily through `/security threat-model` and `/securi
|
||||||
## Project scope
|
## Project scope
|
||||||
|
|
||||||
This is a **solo open-source project in stabilization mode** as of 2026-05-01.
|
This is a **solo open-source project in stabilization mode** as of 2026-05-01.
|
||||||
The current feature set (5 frameworks, 23 scanners, 9 hooks, 6 agents,
|
The current feature set (5 frameworks, 22 scanners, 9 hooks, 6 agents,
|
||||||
20 commands, 22 knowledge files, 2013+ tests including a dedicated end-to-end suite) is the natural plateau for
|
20 commands, 23 knowledge files, 2034+ tests including a dedicated end-to-end suite) is the natural plateau for
|
||||||
what a deterministic + advisory plugin can defend against without crossing
|
what a deterministic + advisory plugin can defend against without crossing
|
||||||
into commercial-grade territory. Going forward, work focuses on:
|
into commercial-grade territory. Going forward, work focuses on:
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -2,7 +2,7 @@
|
||||||
name: deep-scan-synthesizer-agent
|
name: deep-scan-synthesizer-agent
|
||||||
description: |
|
description: |
|
||||||
Synthesizes deterministic deep-scan JSON results into a human-readable security report.
|
Synthesizes deterministic deep-scan JSON results into a human-readable security report.
|
||||||
Takes raw scanner output (9 scanners, structured findings) and produces an executive summary,
|
Takes raw scanner output (14 scanners, structured findings) and produces an executive summary,
|
||||||
prioritized recommendations, and per-scanner analysis.
|
prioritized recommendations, and per-scanner analysis.
|
||||||
Use when /security deep-scan or /security scan --deep has completed scanner execution.
|
Use when /security deep-scan or /security scan --deep has completed scanner execution.
|
||||||
model: opus
|
model: opus
|
||||||
|
|
@ -17,7 +17,7 @@ You are a security report synthesizer for the llm-security plugin's deterministi
|
||||||
## Input
|
## Input
|
||||||
|
|
||||||
You receive:
|
You receive:
|
||||||
1. **Raw JSON output** from `scan-orchestrator.mjs` — contains findings from 9 scanners (including TFA toxic flow analysis)
|
1. **Raw JSON output** from `scan-orchestrator.mjs` — contains findings from 14 scanners (including TFA toxic flow analysis)
|
||||||
2. **Path to the report template** at `templates/unified-report.md` (ANALYSIS_TYPE: deep-scan)
|
2. **Path to the report template** at `templates/unified-report.md` (ANALYSIS_TYPE: deep-scan)
|
||||||
3. **Knowledge base paths** for OWASP context
|
3. **Knowledge base paths** for OWASP context
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
# CI/CD Integration Guide
|
# CI/CD Integration Guide
|
||||||
|
|
||||||
Integrate llm-security into your CI/CD pipeline for automated security scanning of AI/LLM projects. The standalone CLI runs 10 deterministic Node.js scanners — no AI models, no external API calls, no data leaves your pipeline environment.
|
Integrate llm-security into your CI/CD pipeline for automated security scanning of AI/LLM projects. The standalone CLI runs 14 deterministic Node.js scanners — no AI models, no external API calls, no data leaves your pipeline environment.
|
||||||
|
|
||||||
## Data Sovereignty
|
## Data Sovereignty
|
||||||
|
|
||||||
|
|
@ -20,7 +20,7 @@ Integrate llm-security into your CI/CD pipeline for automated security scanning
|
||||||
|
|
||||||
- **NSM Grunnprinsipper:** Automated security scanning fulfills GP 3.1 (vulnerability management) and GP 2.4 (secure development)
|
- **NSM Grunnprinsipper:** Automated security scanning fulfills GP 3.1 (vulnerability management) and GP 2.4 (secure development)
|
||||||
- **Digitaliseringsdirektoratet:** Aligns with recommended practices for AI system development lifecycle security
|
- **Digitaliseringsdirektoratet:** Aligns with recommended practices for AI system development lifecycle security
|
||||||
- **EU AI Act (expected Aug 2026):** Directly supports Art. 9 (risk management) and Art. 15 (cybersecurity) requirements
|
- **EU AI Act:** Directly supports Art. 9 (risk management) and Art. 15 (cybersecurity) requirements. Note the Digital Omnibus (European Parliament 16 June 2026, Council 29 June 2026) deferred the high-risk obligations these articles sit under — to 2 December 2027 for stand-alone Annex III systems and 2 August 2028 for AI embedded in Annex I regulated products. Transparency obligations still apply from 2 August 2026, so this remains preparatory rather than deadline-driven work
|
||||||
|
|
||||||
## 5-Minute Setup
|
## 5-Minute Setup
|
||||||
|
|
||||||
|
|
@ -137,7 +137,7 @@ With `--fail-on`, exit codes are binary: 0 (clean) or 1 (threshold exceeded). Wi
|
||||||
|
|
||||||
## What Gets Scanned
|
## What Gets Scanned
|
||||||
|
|
||||||
The 10 deterministic scanners cover:
|
The 14 deterministic scanners cover:
|
||||||
|
|
||||||
| Scanner | Detects |
|
| Scanner | Detects |
|
||||||
|---------|---------|
|
|---------|---------|
|
||||||
|
|
@ -150,6 +150,10 @@ The 10 deterministic scanners cover:
|
||||||
| Network | Suspicious URLs, exfiltration endpoints, C2 patterns |
|
| Network | Suspicious URLs, exfiltration endpoints, C2 patterns |
|
||||||
| Memory poisoning | Injection patterns in CLAUDE.md, memory files, rules |
|
| Memory poisoning | Injection patterns in CLAUDE.md, memory files, rules |
|
||||||
| Supply chain | Lockfile audit, blocklists, OSV.dev (opt-in) |
|
| Supply chain | Lockfile audit, blocklists, OSV.dev (opt-in) |
|
||||||
|
| Workflow | CI/CD workflow injection — untrusted triggers, unpinned actions, spoofed bots |
|
||||||
|
| Trigger abuse | Activation-surface abuse in command/agent/skill frontmatter — shadowing, baiting, overly broad triggers |
|
||||||
|
| Signature | Known-malware identity match (webshells, reverse shells, cryptominers, hacktools) |
|
||||||
|
| AST taint | Scope-aware Python taint analysis (parse-only), falls back to regex taint tracing |
|
||||||
| Toxic flow | Lethal trifecta correlation (input + access + exfil) |
|
| Toxic flow | Lethal trifecta correlation (input + access + exfil) |
|
||||||
|
|
||||||
## Local Testing
|
## Local Testing
|
||||||
|
|
|
||||||
|
|
@ -1,12 +1,12 @@
|
||||||
# Compliance Mapping
|
# Compliance Mapping
|
||||||
|
|
||||||
Maps the llm-security plugin's 13 posture categories and mitigation controls to three enterprise compliance frameworks: EU AI Act, NIST AI RMF, and ISO 42001.
|
Maps the llm-security plugin's 13 code-level posture categories and mitigation controls to three enterprise compliance frameworks: EU AI Act, NIST AI RMF, and ISO 42001.
|
||||||
|
|
||||||
Used by `posture-assessor-agent` and compliance-aware posture categories (14-16) to evaluate framework alignment.
|
Used by `posture-assessor-agent` and compliance-aware posture categories (14-16) to evaluate framework alignment.
|
||||||
|
|
||||||
## How to Read This Matrix
|
## How to Read This Matrix
|
||||||
|
|
||||||
- **Plugin Control:** One of the 13 posture scanner categories
|
- **Plugin Control:** One of the 13 code-level posture categories (1-13). The three governance categories (14-16) are consumers of this matrix, not rows in it
|
||||||
- **Control Type:** Automated (hooks), Configured (settings), Advisory (scans/audits)
|
- **Control Type:** Automated (hooks), Configured (settings), Advisory (scans/audits)
|
||||||
- **EU AI Act:** Regulation (EU) 2024/1689 article(s) the control satisfies
|
- **EU AI Act:** Regulation (EU) 2024/1689 article(s) the control satisfies
|
||||||
- **NIST AI RMF:** AI 100-1 function(s) the control supports (Govern, Map, Measure, Manage)
|
- **NIST AI RMF:** AI 100-1 function(s) the control supports (Govern, Map, Measure, Manage)
|
||||||
|
|
|
||||||
|
|
@ -118,7 +118,7 @@ regulatory sandbox for controlled testing under the AI Act.
|
||||||
| Data protection in AI | GDPR, Datatilsynet sandbox | Secrets protection hooks, path guarding, credential scanning | Full |
|
| Data protection in AI | GDPR, Datatilsynet sandbox | Secrets protection hooks, path guarding, credential scanning | Full |
|
||||||
| Transparency and explainability | Digdir principles, AI Act Art. 13 | Scan reports, posture reports, AI-BOM | Partial |
|
| Transparency and explainability | Digdir principles, AI Act Art. 13 | Scan reports, posture reports, AI-BOM | Partial |
|
||||||
| Human oversight | Digdir principles, AI Act Art. 14 | Human Review Requirements (PST-07), Rule of Two, deny-first config | Full |
|
| Human oversight | Digdir principles, AI Act Art. 14 | Human Review Requirements (PST-07), Rule of Two, deny-first config | Full |
|
||||||
| Cybersecurity | AI Act Art. 15, NSM grunnprinsipper | All 8 hooks, 10 scanners, prompt injection hardening | Full |
|
| Cybersecurity | AI Act Art. 15, NSM grunnprinsipper | All 9 hooks, 14 scanners, prompt injection hardening | Full |
|
||||||
| Record-keeping | AI Act Art. 12, NSM detect principle | Audit trail (JSONL), session logging, baseline diffs | Full (v6.0) |
|
| Record-keeping | AI Act Art. 12, NSM detect principle | Audit trail (JSONL), session logging, baseline diffs | Full (v6.0) |
|
||||||
| Quality management | AI Act Art. 17 | Test suite (1147+ tests), posture scanner, scan-orchestrator | Partial |
|
| Quality management | AI Act Art. 17 | Test suite (1147+ tests), posture scanner, scan-orchestrator | Partial |
|
||||||
| Supply chain integrity | AI Act Art. 15, NSM identify principle | Supply chain hooks, dep audit scanner, AI-BOM | Full |
|
| Supply chain integrity | AI Act Art. 15, NSM identify principle | Supply chain hooks, dep audit scanner, AI-BOM | Full |
|
||||||
|
|
|
||||||
|
|
@ -16,7 +16,9 @@ export function resetCounter() {
|
||||||
/**
|
/**
|
||||||
* Create a finding object.
|
* Create a finding object.
|
||||||
* @param {object} opts
|
* @param {object} opts
|
||||||
* @param {string} opts.scanner - Scanner prefix (UNI, ENT, PRM, DEP, TNT, GIT, NET, TRG, SIG, AST)
|
* Every prefix below is pinned by tests/lib/doc-consistency.test.mjs against the
|
||||||
|
* scanners that actually call this function — orchestrated first, then standalone.
|
||||||
|
* @param {string} opts.scanner - Scanner prefix (UNI, ENT, PRM, DEP, TNT, GIT, NET, MEM, SCR, WFL, TRG, SIG, AST, TFA, IDE, MCI, PST)
|
||||||
* @param {string} opts.severity - From SEVERITY constants
|
* @param {string} opts.severity - From SEVERITY constants
|
||||||
* @param {string} opts.title - Short finding title
|
* @param {string} opts.title - Short finding title
|
||||||
* @param {string} opts.description - Detailed description
|
* @param {string} opts.description - Detailed description
|
||||||
|
|
|
||||||
|
|
@ -159,3 +159,260 @@ describe('doc-consistency — Hooks count consistency (D4)', () => {
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// B10 (v8.0.0 Phase 2): inventory counts stated in prose must be derived from
|
||||||
|
// source, not hand-maintained. Before this block, four independent counts had
|
||||||
|
// drifted — orchestrated scanners (docs said 10, source had 14), total
|
||||||
|
// scanners (README said 23, the documented rule yields 22), knowledge files
|
||||||
|
// (README said 22, the directory holds 23), and the `finding()` prefix list in
|
||||||
|
// output.mjs (documented 10 of the 17 prefixes actually passed to it).
|
||||||
|
//
|
||||||
|
// Every count below has ONE derivation from source, stated next to the helper.
|
||||||
|
// A doc surface that disagrees fails here rather than misleading a reader.
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
describe('doc-consistency — inventory counts are derived from source (B10)', () => {
|
||||||
|
const SCANNERS_DIR = join(PLUGIN_ROOT, 'scanners');
|
||||||
|
const KNOWLEDGE_DIR = join(PLUGIN_ROOT, 'knowledge');
|
||||||
|
|
||||||
|
// Counting rule (scanner total): every `scanners/*.mjs` IS a scanner except
|
||||||
|
// these five, which are a runner, a remediator, an extraction library, a
|
||||||
|
// cache utility and a CLI wrapper around an already-counted scanner.
|
||||||
|
const NON_SCANNER_MODULES = [
|
||||||
|
'scan-orchestrator.mjs',
|
||||||
|
'auto-cleaner.mjs',
|
||||||
|
'content-extractor.mjs',
|
||||||
|
'mcp-baseline-reset.mjs',
|
||||||
|
'supply-chain-recheck-cli.mjs',
|
||||||
|
];
|
||||||
|
|
||||||
|
function scannerModules() {
|
||||||
|
return readdirSync(SCANNERS_DIR).filter(f => f.endsWith('.mjs'));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ground truth: the SCANNERS array literal in scan-orchestrator.mjs.
|
||||||
|
function orchestratedCount() {
|
||||||
|
const src = readFileSync(join(SCANNERS_DIR, 'scan-orchestrator.mjs'), 'utf-8');
|
||||||
|
const start = src.indexOf('const SCANNERS = [');
|
||||||
|
if (start < 0) throw new Error('scan-orchestrator.mjs: `const SCANNERS = [` not found');
|
||||||
|
const end = src.indexOf('];', start);
|
||||||
|
const block = src.slice(start, end);
|
||||||
|
return (block.match(/^\s*\{\s*name:/gm) || []).length;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ground truth: scanner modules minus the five non-scanner modules above.
|
||||||
|
function totalScannerCount() {
|
||||||
|
return scannerModules().filter(f => !NON_SCANNER_MODULES.includes(f)).length;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ground truth: every entry in knowledge/.
|
||||||
|
function knowledgeFileCount() {
|
||||||
|
return readdirSync(KNOWLEDGE_DIR).length;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ground truth: the CATEGORIES array literal in posture-scanner.mjs.
|
||||||
|
function postureCategoryCount() {
|
||||||
|
const src = readFileSync(join(SCANNERS_DIR, 'posture-scanner.mjs'), 'utf-8');
|
||||||
|
const start = src.indexOf('const CATEGORIES = [');
|
||||||
|
if (start < 0) throw new Error('posture-scanner.mjs: `const CATEGORIES = [` not found');
|
||||||
|
const end = src.indexOf('];', start);
|
||||||
|
return (src.slice(start, end).match(/^\s*\{\s*id:\s*\d+/gm) || []).length;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ground truth: the prefix every module that calls `finding()` passes as
|
||||||
|
// `opts.scanner`. Two idioms are in use — an inline `scanner: 'XXX'` literal
|
||||||
|
// and a module-level `const SCANNER`/`SCANNER_PREFIX`.
|
||||||
|
function findingPrefixes() {
|
||||||
|
const prefixes = new Set();
|
||||||
|
for (const file of scannerModules()) {
|
||||||
|
const src = readFileSync(join(SCANNERS_DIR, file), 'utf-8');
|
||||||
|
if (!/import\s*\{[^}]*\bfinding\b[^}]*\}\s*from\s*'\.\/lib\/output\.mjs'/.test(src)) continue;
|
||||||
|
for (const m of src.matchAll(/scanner:\s*'([A-Z]{2,4})'/g)) prefixes.add(m[1]);
|
||||||
|
for (const m of src.matchAll(/^const SCANNER(?:_PREFIX)?\s*=\s*'([A-Z]{2,4})'/gm)) prefixes.add(m[1]);
|
||||||
|
}
|
||||||
|
return prefixes;
|
||||||
|
}
|
||||||
|
|
||||||
|
it('NON_SCANNER_MODULES all exist (the exclusion list cannot silently rot)', () => {
|
||||||
|
const present = new Set(scannerModules());
|
||||||
|
for (const name of NON_SCANNER_MODULES) {
|
||||||
|
assert.equal(
|
||||||
|
present.has(name),
|
||||||
|
true,
|
||||||
|
`NON_SCANNER_MODULES lists scanners/${name}, which no longer exists. ` +
|
||||||
|
`A rename here silently inflates the total scanner count — update the list.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// -- Orchestrated scanner count -------------------------------------------
|
||||||
|
// Every "<N> deterministic|orchestrated scanners" claim across prose and the
|
||||||
|
// standalone CLI must equal the SCANNERS array length.
|
||||||
|
const ORCHESTRATED_CLAIM_FILES = [
|
||||||
|
'README.md',
|
||||||
|
'CLAUDE.md',
|
||||||
|
join('docs', 'ci-cd-guide.md'),
|
||||||
|
join('docs', 'scanner-reference.md'),
|
||||||
|
join('bin', 'llm-security.mjs'),
|
||||||
|
join('knowledge', 'norwegian-context.md'),
|
||||||
|
join('agents', 'deep-scan-synthesizer-agent.md'),
|
||||||
|
];
|
||||||
|
|
||||||
|
for (const rel of ORCHESTRATED_CLAIM_FILES) {
|
||||||
|
it(`${rel} states the orchestrated scanner count correctly`, () => {
|
||||||
|
const expected = orchestratedCount();
|
||||||
|
const content = readFileSync(join(PLUGIN_ROOT, rel), 'utf-8');
|
||||||
|
const patterns = [
|
||||||
|
/(\d+)\s+deterministic(?:\s+Node\.js)?\s+scanners/gi,
|
||||||
|
/(\d+)\s+orchestrated\s+deterministic\s+scanners/gi,
|
||||||
|
/\*\*Orchestrated\s*\((\d+)\):?\*\*/g,
|
||||||
|
/^(\d+)\s+scanners:/gm,
|
||||||
|
/\((\d+)\s+scanners\)/g,
|
||||||
|
/findings\s+from\s+(\d+)\s+scanners/gi,
|
||||||
|
/scanner\s+output\s+\((\d+)\s+scanners/gi,
|
||||||
|
/hooks,\s*(\d+)\s+scanners/gi,
|
||||||
|
];
|
||||||
|
for (const pattern of patterns) {
|
||||||
|
for (const m of content.matchAll(pattern)) {
|
||||||
|
assert.equal(
|
||||||
|
Number(m[1]),
|
||||||
|
expected,
|
||||||
|
`${rel} claims ${m[1]} orchestrated scanners in "${m[0].trim()}", but ` +
|
||||||
|
`scan-orchestrator.mjs registers ${expected}. Update the prose.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// -- Total scanner count (orchestrated + standalone) -----------------------
|
||||||
|
it('README scanner badge matches the total scanner count', () => {
|
||||||
|
const expected = totalScannerCount();
|
||||||
|
const content = readFileSync(join(PLUGIN_ROOT, 'README.md'), 'utf-8');
|
||||||
|
const m = content.match(/badge\/scanners-(\d+)-/);
|
||||||
|
assert.notEqual(m, null, 'README.md has no `scanners-<N>` badge');
|
||||||
|
assert.equal(
|
||||||
|
Number(m[1]),
|
||||||
|
expected,
|
||||||
|
`README scanner badge says ${m[1]} but scanners/ holds ${expected} scanner modules ` +
|
||||||
|
`(${scannerModules().length} .mjs files minus ${NON_SCANNER_MODULES.length} non-scanner modules).`,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('docs/scanner-reference.md orchestrated + standalone equals the total', () => {
|
||||||
|
const content = readFileSync(join(PLUGIN_ROOT, 'docs', 'scanner-reference.md'), 'utf-8');
|
||||||
|
const orch = content.match(/\*\*Orchestrated\s*\((\d+)\):?\*\*/);
|
||||||
|
const standalone = content.match(/\*\*Standalone\s*\((\d+)\):?\*\*/);
|
||||||
|
assert.notEqual(orch, null, 'scanner-reference.md has no `**Orchestrated (N)**` marker');
|
||||||
|
assert.notEqual(standalone, null, 'scanner-reference.md has no `**Standalone (N)**` marker');
|
||||||
|
assert.equal(
|
||||||
|
Number(orch[1]) + Number(standalone[1]),
|
||||||
|
totalScannerCount(),
|
||||||
|
`scanner-reference.md declares ${orch[1]} orchestrated + ${standalone[1]} standalone ` +
|
||||||
|
`= ${Number(orch[1]) + Number(standalone[1])}, but scanners/ holds ${totalScannerCount()}. ` +
|
||||||
|
`This split is the counting rule the README badge depends on.`,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('README prose scanner totals match the badge', () => {
|
||||||
|
const expected = totalScannerCount();
|
||||||
|
const content = readFileSync(join(PLUGIN_ROOT, 'README.md'), 'utf-8');
|
||||||
|
const patterns = [
|
||||||
|
/Deterministic analysis\s*[—-]\s*(\d+)\s+scanners/g,
|
||||||
|
/^(\d+)\s+scanners\.\s+Zero external dependencies/gm,
|
||||||
|
/frameworks,\s*(\d+)\s+scanners/g,
|
||||||
|
];
|
||||||
|
for (const pattern of patterns) {
|
||||||
|
for (const m of content.matchAll(pattern)) {
|
||||||
|
assert.equal(
|
||||||
|
Number(m[1]),
|
||||||
|
expected,
|
||||||
|
`README claims ${m[1]} scanners in "${m[0].trim()}" but the total is ${expected}.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// -- Knowledge-file count --------------------------------------------------
|
||||||
|
it('README knowledge badge and prose match knowledge/ contents', () => {
|
||||||
|
const expected = knowledgeFileCount();
|
||||||
|
const content = readFileSync(join(PLUGIN_ROOT, 'README.md'), 'utf-8');
|
||||||
|
const badge = content.match(/badge\/knowledge_docs-(\d+)-/);
|
||||||
|
assert.notEqual(badge, null, 'README.md has no `knowledge_docs-<N>` badge');
|
||||||
|
assert.equal(
|
||||||
|
Number(badge[1]),
|
||||||
|
expected,
|
||||||
|
`README knowledge badge says ${badge[1]} but knowledge/ holds ${expected} files.`,
|
||||||
|
);
|
||||||
|
for (const m of content.matchAll(/(\d+)\s+knowledge files/g)) {
|
||||||
|
assert.equal(
|
||||||
|
Number(m[1]),
|
||||||
|
expected,
|
||||||
|
`README claims ${m[1]} in "${m[0]}" but knowledge/ holds ${expected} files.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// -- output.mjs finding() prefix list --------------------------------------
|
||||||
|
it('output.mjs JSDoc lists every prefix passed to finding()', () => {
|
||||||
|
const actual = findingPrefixes();
|
||||||
|
const src = readFileSync(join(SCANNERS_DIR, 'lib', 'output.mjs'), 'utf-8');
|
||||||
|
const m = src.match(/@param\s+\{string\}\s+opts\.scanner\s*-\s*Scanner prefix\s*\(([^)]*)\)/);
|
||||||
|
assert.notEqual(m, null, 'output.mjs has no `opts.scanner - Scanner prefix (...)` JSDoc line');
|
||||||
|
const documented = new Set(m[1].split(',').map(s => s.trim()).filter(Boolean));
|
||||||
|
|
||||||
|
const missing = [...actual].filter(p => !documented.has(p)).sort();
|
||||||
|
const extra = [...documented].filter(p => !actual.has(p)).sort();
|
||||||
|
assert.deepEqual(
|
||||||
|
{ missing, extra },
|
||||||
|
{ missing: [], extra: [] },
|
||||||
|
`output.mjs JSDoc prefix list is out of sync with the scanners that call finding(). ` +
|
||||||
|
`Missing from JSDoc: [${missing.join(', ')}]. Documented but unused: [${extra.join(', ')}].`,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
// -- Posture category count ------------------------------------------------
|
||||||
|
it('posture category count agrees across CLAUDE.md, README and scanner-reference', () => {
|
||||||
|
const expected = postureCategoryCount();
|
||||||
|
for (const rel of ['CLAUDE.md', 'README.md', join('docs', 'scanner-reference.md')]) {
|
||||||
|
const content = readFileSync(join(PLUGIN_ROOT, rel), 'utf-8');
|
||||||
|
for (const m of content.matchAll(/(\d+)\s+posture categories/gi)) {
|
||||||
|
assert.equal(Number(m[1]), expected, `${rel}: "${m[0]}" but posture-scanner has ${expected}.`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it('compliance-mapping.md maps its stated number of posture categories', () => {
|
||||||
|
const content = readFileSync(join(KNOWLEDGE_DIR, 'compliance-mapping.md'), 'utf-8');
|
||||||
|
// The matrix deliberately covers only the code-level categories; the
|
||||||
|
// governance categories (14-16) are consumers of this file, not rows in it.
|
||||||
|
const start = content.indexOf('## Mapping Matrix');
|
||||||
|
assert.notEqual(start, -1, 'compliance-mapping.md has no `## Mapping Matrix` section');
|
||||||
|
const tail = content.slice(start);
|
||||||
|
const end = tail.indexOf('\n---');
|
||||||
|
const section = end > 0 ? tail.slice(0, end) : tail;
|
||||||
|
// Data rows only: skip the header row and the `|---|` separator.
|
||||||
|
const rows = (section.match(/^\|(?!\s*Plugin Control)(?!-)[^|]+\|/gm) || [])
|
||||||
|
.filter(r => !/^\|\s*-+/.test(r));
|
||||||
|
const claimed = content.match(/(\d+)\s+code-level posture categories/);
|
||||||
|
assert.notEqual(
|
||||||
|
claimed,
|
||||||
|
null,
|
||||||
|
'compliance-mapping.md must state "<N> code-level posture categories" so the ' +
|
||||||
|
'claim is distinguishable from the plugin\'s full posture category count.',
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
rows.length,
|
||||||
|
Number(claimed[1]),
|
||||||
|
`compliance-mapping.md claims ${claimed[1]} code-level posture categories but the ` +
|
||||||
|
`Mapping Matrix has ${rows.length} data rows.`,
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
rows.length < postureCategoryCount(),
|
||||||
|
true,
|
||||||
|
`The matrix is documented as a subset of the ${postureCategoryCount()} posture categories; ` +
|
||||||
|
`it now has ${rows.length} rows. If it grew to cover all of them, update the prose too.`,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue