Kjell Tore Guttormsen
bbada84e9f
Fifth and last consumer swap of v8 Phase 5 step 4. The seven known-bad-identity
signatures stop living in knowledge/signatures.json and are built from the
vendored commons artifact signatures/malware-signatures.json instead.
Measured before the swap over all seven positions -- id, family, severity,
pattern, description, provenance, key order, and recompilation identity under
the engine's unconditional `i` flag: zero divergences over 56 checks, in order.
The commons copy was extracted from this repository's own file at b0de0ca and
had not drifted.
knowledge/signatures.json is REMOVED rather than left in place. Keeping it would
have left two files spelling one table with nothing gating the drift, and its
golden `file:` pin would have gone on passing while pinning bytes no scanner
reads -- a gate reporting success without running. The pin is replaced by a
walked-module anchor over SIGNATURE_RULES, which is strictly stronger: the pin
covered the bytes on disk, the walk covers what `new RegExp` made of them.
Golden diff was exactly that and nothing else: 7 ADDED, 1 REMOVED, 0 CHANGED
(102/7/5 -> 109/7/4), each added source verified equal to the recompiled commons
pattern.
compileRules() moves into the new lib module and is exported, so the built-in
ruleset and the operator's sig.custom_rules_path path keep one implementation
rather than two copies of the defaulting logic.
Coverage by construction, not by memory: the probe table in the scanner test is
asserted against the LOADED ruleset, so a rule commons adds cannot arrive
without an end-to-end probe. Mutation of the vendored JSON fires in three
directions -- under-match (xmrig alternative dropped) reddens two scanner tests
plus golden; over-match (webshell rule widened to a bare `shell`) reddens the
clean-fixture false-positive probe plus golden; reorder reddens the declared-
order test plus golden.
Loud failure is contract: an unresolvable commons writes one line to stderr
rather than silently disabling known-malware detection, and never throws.
Suite 2247 / 2241 pass / 6 skipped / 0 fail. suite-counts.json untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0151x4FVg9Mn55C2LvHLpHKo
128 lines
17 KiB
Markdown
128 lines
17 KiB
Markdown
# LLM Security — Scanner reference
|
||
|
||
Detailed scanner, CLI, CI/CD, knowledge-file and example documentation. Imported from `CLAUDE.md` via `@docs/scanner-reference.md`.
|
||
|
||
## Scanners
|
||
|
||
**Orchestrated (14):** Run via `node scanners/scan-orchestrator.mjs <target> [--fail-on <severity>] [--compact] [--output-file <path>] [--baseline] [--save-baseline]`.
|
||
`--fail-on <critical|high|medium|low>`: exit 1 if findings at/above severity, exit 0 otherwise. `--compact`: one-liner per finding format. Both configurable via `policy.json` `ci` section.
|
||
With `--output-file`: full JSON to file, compact aggregate to stdout. `--baseline` diffs against stored baseline. `--save-baseline` saves results for future diffs. Baselines stored in `reports/baselines/<target-hash>.json`.
|
||
|
||
14 scanners: unicode, entropy, permission, dep-audit, taint, git-forensics, network, memory-poisoning, supply-chain-recheck, workflow, toxic-flow, trigger, signature, ast-taint.
|
||
|
||
Lib: `mcp-description-cache.mjs` — caches MCP tool descriptions in `~/.cache/llm-security/mcp-descriptions.json`, detects per-update drift via Levenshtein (>10% = alert), 7-day TTL. v7.3.0 (E14) adds a sticky baseline slot per tool plus a 10-event rolling history; cumulative drift = `levenshtein(current, baseline) / max(|current|,|baseline|)`. When ratio ≥ `mcp.cumulative_drift_threshold` (default 0.25), emits `mcp-cumulative-drift` advisory through `post-mcp-verify.mjs`. Baseline survives TTL purge so slow-burn drift is preserved across the 7-day window. `clearBaseline(tool?)` exposed for the `/security mcp-baseline-reset` command. `LLM_SECURITY_MCP_CACHE_FILE` env var overrides the cache path for testing.
|
||
|
||
Supply-chain-recheck (SCR) re-audits installed dependencies from lockfiles (package-lock.json, yarn.lock, requirements.txt, Pipfile.lock) against blocklists, OSV.dev batch API, and typosquat detection. Offline fallback available. Shared data module: `scanners/lib/supply-chain-data.mjs`.
|
||
|
||
Memory-poisoning (MEM) detects cognitive state poisoning in CLAUDE.md, memory files, and .claude/rules — injection patterns, shell commands, credential paths, permission expansion, suspicious URLs, encoded payloads.
|
||
|
||
Toxic-flow (TFA) is a post-processing correlator that runs LAST — detects "lethal trifecta" (untrusted input + sensitive data access + exfiltration sink) by correlating output from prior scanners.
|
||
|
||
Trigger-abuse (TRG) inspects command/agent/skill `name`+`description` frontmatter for activation-surface abuse: built-in shadowing (a name colliding with a built-in tool), activation baiting (maximally-activating description phrases), and overly broad triggers (generic name + universal-applicability claim → HIGH). Descriptions are run through the decode pipeline (zero-width strip, homoglyph fold, `normalizeForScan`) so obfuscated baiting still trips. Policy: `trg` section (`baiting_phrases`, `builtin_names`, `broad_single_words`). OWASP: LLM06, AST04.
|
||
|
||
Signature (SIG) is a known-bad-identity engine: a small, high-confidence, family-grouped ruleset (`signatures/malware-signatures.json` in the vendored commons, built by `scanners/lib/malware-signatures.mjs` — webshell, reverse_shell, cryptominer, hacktool) tested against each file's decode pipeline (`normalizeForScan`/`foldHomoglyphs`/`rot13`), so obfuscated known-malware that a raw byte-matcher misses is still caught. Path-excludes `knowledge/`, `tests/`, `docs/`. Policy: `sig.enabled_families`. OWASP: LLM03, LLM02.
|
||
|
||
AST-taint (AST) shells out to a PARSE-ONLY python3 helper (`scanners/lib/py-ast-taint.py`) for scope-aware, variable-level Python taint analysis (sources → sinks), with graceful fallback to the regex taint-tracer when python3 is absent. The helper only `ast.parse`s the target — it never executes it. 5s timeout per file. Policy: `ast` section (`enabled`, `python_path`, `timeout_ms`). OWASP: LLM01, LLM02, AST02.
|
||
|
||
Utility: `node scanners/lib/fs-utils.mjs <backup|restore|cleanup|tmppath> [args]`.
|
||
|
||
Lib: `sarif-formatter.mjs` — converts scan output to OASIS SARIF 2.1.0 format. Used by `--format sarif` flag.
|
||
Lib: `audit-trail.mjs` — writes structured JSONL audit events (ISO 8601, OWASP tags, SIEM-ready). Enabled by the `audit.log_path` key in `.llm-security/policy.json` (v8.0.0; was `LLM_SECURITY_AUDIT_LOG`).
|
||
Lib: `policy-loader.mjs` — reads `.llm-security/policy.json` for distributable hook configuration. Includes `ci` section (`failOn`, `compact`) for CI/CD defaults. Defaults match hardcoded values.
|
||
|
||
**Standalone (8):** `posture-scanner.mjs` — deterministic posture assessment, 16 categories (incl. EU AI Act, NIST AI RMF, ISO 42001), <50ms. NOT in scan-orchestrator (meta-level, not code-level).
|
||
Run: `node scanners/posture-scanner.mjs [path]` → JSON stdout. Scanner prefix: PST. Used by `/security posture` and `/security audit`.
|
||
|
||
`mcp-live-inspect.mjs` — NOT in scan-orchestrator. MCP servers are running processes, not files.
|
||
Run: `node scanners/mcp-live-inspect.mjs [target] [--timeout 10000] [--skip-global]`
|
||
Scanner prefix: MCI. OWASP: MCP03, MCP06, MCP09. Invoked by `mcp-inspect` and `mcp-audit --live`.
|
||
|
||
`watch-cron.mjs` — standalone cron wrapper. Reads `reports/watch/config.json`, scans all targets, writes `reports/watch/latest.json`. Run: `node scanners/watch-cron.mjs [--config <path>]`
|
||
|
||
`reference-config-generator.mjs` — generates Grade A reference config based on posture gaps. Detects project type (plugin/monorepo/standalone). Templates in `templates/reference-config/`. Run: `node scanners/reference-config-generator.mjs [path] [--apply]`
|
||
|
||
`dashboard-aggregator.mjs` — cross-project security dashboard. Discovers Claude Code projects under ~/ (depth 3) and ~/.claude/plugins/, runs posture-scanner on each, aggregates to machine-grade (weakest link). Cache in `~/.cache/llm-security/dashboard-latest.json` (24h staleness). Run: `node scanners/dashboard-aggregator.mjs [--no-cache] [--max-depth N]`
|
||
|
||
`attack-simulator.mjs` — red-team harness. Data-driven: 72 scenarios in 12 categories from `knowledge/attack-scenarios.json`. Payloads constructed at runtime (fragment assembly to avoid triggering hooks on source). Uses `runHook()` from test helper. Adaptive mode (`--adaptive`): 5 mutation rounds per passing scenario (homoglyph, encoding, zero-width, case alternation, synonym). Mutation rules in `knowledge/attack-mutations.json`. Benchmark mode (`--benchmark`): outputs structured pass/fail metrics. Run: `node scanners/attack-simulator.mjs [--category <name>] [--json] [--verbose] [--adaptive] [--benchmark]`
|
||
|
||
`ai-bom-generator.mjs` — AI Bill of Materials generator. Discovers AI components (models, MCP servers, plugins, knowledge, hooks) and outputs CycloneDX 1.6 JSON. Scanner prefix: BOM. Run: `node scanners/ai-bom-generator.mjs <target> [--output-file <path>]`
|
||
|
||
`ide-extension-scanner.mjs` — scans installed VS Code (and forks: Cursor, Windsurf, VSCodium, code-server, Insiders, Remote-SSH) extensions and JetBrains IDE plugins (IntelliJ IDEA, PyCharm, GoLand, WebStorm, RubyMine, PhpStorm, CLion, DataGrip, RustRover, Rider, Aqua, Writerside, Android Studio). Fleet + Toolbox excluded. OS-aware discovery via `lib/ide-extension-discovery.mjs` (`~/.vscode/extensions/` + `~/Library/Application Support/JetBrains/<IDE><version>/plugins/` on macOS, `%APPDATA%\JetBrains\...` on Windows, `~/.config/JetBrains/...` on Linux). Parses VS Code `package.json` via `lib/ide-extension-parser.mjs` and JetBrains `META-INF/plugin.xml` + `META-INF/MANIFEST.MF` (with nested-jar extraction) via `lib/ide-extension-parser-jb.mjs`. 7 VS Code checks: blocklist match, theme-with-code, sideload (vsix), broad activation (`*` / `onStartupFinished`), typosquat (Levenshtein ≤2 vs top-100), extension-pack expansion, dangerous `vscode:uninstall` hooks. 7 JetBrains checks: theme-with-code, broad activation (`application-components`), `Premain-Class` instrumentation (HIGH — javaagent retransform), native binaries (`.so`/`.dylib`/`.dll`/`.jnilib`), long `<depends>` chains, typosquat vs top JetBrains plugins, shaded-jar advisory. Both branches orchestrate reused scanners (UNI/ENT/NET/TNT/MEM/SCR) per extension with bounded concurrency (default 4). Scanner prefix: IDE. OWASP: LLM01, LLM02, LLM03, LLM06, ASI02, ASI04. Offline by default, `--online` opt-in for Marketplace/OSV.dev lookups. Knowledge: `knowledge/top-vscode-extensions.json`, `knowledge/top-jetbrains-plugins.json`, `knowledge/ide-extension-threat-patterns.md`, `knowledge/marketplace-api-notes.md`, `knowledge/jetbrains-marketplace-api-notes.md`.
|
||
|
||
**v6.4.0 — URL support.** Targets can be Marketplace, OpenVSX, or direct `.vsix` URLs. Pipeline: `lib/vsix-fetch.mjs` (HTTPS-only fetch with 50MB cap, 30s timeout, SHA-256, manual redirect host whitelist) → `lib/zip-extract.mjs` (zero-dep ZIP parser, rejects zip-slip/symlink/absolute/drive-letter/encrypted/ZIP64, caps: 10 000 entries, 500MB uncomp, 100x ratio, depth 20) → existing scan pipeline against extracted `extension/` subdir → temp dir always cleaned in `try/finally`. Envelope.meta.source = `{ type: "url", kind, url, finalUrl, sha256, size, publisher?, name?, version? }`.
|
||
|
||
**v6.5.0 — OS sandbox.** Fetch + extract for URL targets now spawns `lib/vsix-fetch-worker.mjs` in a sub-process wrapped by `sandbox-exec` (macOS) or `bwrap` (Linux) — same primitives reused from `git-clone.mjs`. Helper: `lib/vsix-sandbox.mjs` exports `buildSandboxProfile`, `buildBwrapArgs`, `buildSandboxedWorker`, `runVsixWorker`. Worker IPC: argv `--url <url> --tmpdir <dir>` → single JSON line on stdout (`{ok, sha256, size, finalUrl, source, extRoot}` or `{ok:false, error, code?}`). Defense-in-depth — if the in-process ZIP parser ever has a bypass, the kernel still refuses writes outside `<tmpdir>`. `scan(target, { useSandbox })` defaults to `true`; tests pass `false` since `globalThis.fetch` mocks do not cross process boundaries. Windows fallback: in-process with `meta.warnings` advisory. Envelope `meta.source.sandbox`: `'sandbox-exec' | 'bwrap' | 'none' | 'in-process'`.
|
||
|
||
**v6.6.0 — JetBrains Marketplace URL fetch + JetBrains branch.** URL targets can also be `https://plugins.jetbrains.com/plugin/<numericId>-<slug>` (metadata-resolved → xmlId download) or `https://plugins.jetbrains.com/plugin/download?pluginId=<xmlId>&version=<v>` (direct). `lib/vsix-fetch.mjs` gains `detectUrlType` JetBrains kinds, `fetchJetBrainsPlugin`, host allowlist `plugins.jetbrains.com`. `buildSandboxedWorker(dirs, workerPath)` now accepts a custom worker path — `lib/jetbrains-fetch-worker.mjs` reuses the same IPC contract. Envelope `meta.source.kind` can be `'jetbrains-marketplace' | 'jetbrains-download'`. Installed-plugin scan runs JB-specific checks (see scanner bullet above) and shares the UNI/ENT/NET/TNT/MEM/SCR orchestration. `.kt`, `.groovy`, `.scala` added to `taint-tracer` code extensions.
|
||
|
||
Run: `node scanners/ide-extension-scanner.mjs [target|url] [--vscode-only] [--intellij-only] [--include-builtin] [--online] [--format json|compact] [--fail-on <sev>] [--output-file <path>]`. Invoked by `/security ide-scan`.
|
||
|
||
## Token Budget (ENFORCED)
|
||
|
||
All commands total ~600 lines. All commands use registered subagent types.
|
||
|
||
- Commands are short dispatchers (~30-60 lines) — no inline report templates or format specs
|
||
- All agents use registered `subagent_type` — agent instructions are system prompt, never file reads
|
||
- Max 1-2 knowledge files per agent invocation (threat-patterns + secrets-patterns)
|
||
- OWASP files are NEVER passed by commands — agents reference them from their own system prompt
|
||
- Agents run sequentially to avoid burst rate limits
|
||
- `pre-install-supply-chain.mjs` queries OSV.dev for CVEs on every package install
|
||
|
||
## CLI
|
||
|
||
`bin/llm-security.mjs` — standalone CLI entry point. Works without Claude Code via `npx llm-security` or `node bin/llm-security.mjs`.
|
||
Subcommands: `scan`, `deep-scan`, `posture`, `audit-bom`, `benchmark`. Dispatches to scanner scripts via `child_process.spawn`.
|
||
`package.json` `bin` field: `"llm-security": "./bin/llm-security.mjs"`. `files` whitelist: only `bin/` + `scanners/` published to npm.
|
||
|
||
## CI/CD Integration
|
||
|
||
Pipeline templates in `ci/`: `github-action.yml`, `azure-pipelines.yml`, `gitlab-ci.yml`. Documentation: `docs/ci-cd-guide.md`.
|
||
All templates use `--fail-on high --format sarif --output-file results.sarif` with SARIF upload per platform.
|
||
Standalone CLI makes zero network calls in default mode. Schrems II compatible in default offline mode. Optional OSV.dev enrichment (`supply-chain-recheck --online`) transmits package identifiers to a Google-operated API and is a separate compliance consideration.
|
||
|
||
## Knowledge Files (20)
|
||
|
||
| File | Content |
|
||
|------|---------|
|
||
| `skill-threat-patterns.md` | 7 threat categories for skill/command scanning |
|
||
| `mcp-threat-patterns.md` | 9 MCP threat categories (MCP01-MCP10) |
|
||
| `secrets-patterns.md` | Regex patterns for 10+ secret types |
|
||
| `owasp-llm-top10.md` | OWASP LLM Top 10 (2025) with Claude Code mappings |
|
||
| `owasp-agentic-top10.md` | OWASP Agentic AI Top 10 (ASI01-ASI10) |
|
||
| `owasp-skills-top10.md` | OWASP Skills Top 10 (AST01-AST10) — skill-specific threats |
|
||
| `mitigation-matrix.md` | Threat-to-control mappings |
|
||
| `top-packages.json` | Known package lists for supply chain checks |
|
||
| `skill-registry.json` | Seed data for skill signature registry |
|
||
| `prompt-injection-research-2025-2026.md` | 7 research papers (2025-2026) with implications for hook defenses |
|
||
| `deepmind-agent-traps.md` | DeepMind AI Agent Traps — 6 categories, 43 techniques, coverage matrix |
|
||
| `attack-scenarios.json` | 72 red-team scenarios across 12 categories for attack simulation |
|
||
| `attack-mutations.json` | Synonym tables and mutation rules for adaptive red-team testing |
|
||
| `compliance-mapping.md` | EU AI Act, NIST AI RMF, ISO 42001, MITRE ATLAS mappings to plugin capabilities |
|
||
| `norwegian-context.md` | Norwegian regulatory landscape — Datatilsynet, NSM, Digitaliseringsdirektoratet |
|
||
| `ide-extension-threat-patterns.md` | 10 IDE-extension detection categories (VS Code + JetBrains) with 2024-2026 case studies |
|
||
| `top-vscode-extensions.json` | Top ~100 VS Code Marketplace extension IDs (typosquat seed) + blocklist entries |
|
||
| `top-jetbrains-plugins.json` | Top JetBrains plugin IDs (typosquat seed) + blocklist entries (v6.6.0) |
|
||
| `marketplace-api-notes.md` | VS Code Marketplace + OpenVSX API endpoints used by `lib/vsix-fetch.mjs` (v6.4.0) |
|
||
| `jetbrains-marketplace-api-notes.md` | JetBrains Marketplace API endpoints used by `fetchJetBrainsPlugin` (v6.6.0) |
|
||
|
||
## Reports
|
||
|
||
Scan reports are stored in `reports/` as `.docx` (for sharing) with `.md` source.
|
||
|
||
## Examples (runnable demonstrations)
|
||
|
||
Self-contained, deterministic threat fixtures live under `examples/`. Each directory has a `README.md`, fixture/script/transcript, `run-*.{sh,mjs}`, and `expected-findings.md`. They are demonstrations — not unit tests.
|
||
|
||
| Directory | Demonstrates | Hooks/scanners | Sentinel |
|
||
|-------|--------------|----------------|----------|
|
||
| `malicious-skill-demo/` | Skill scanner end-to-end (UNI/ENT/PRM/DEP/TNT/NET + 7 LLM categories) | `scan-orchestrator` + agents | BLOCK 100/100 |
|
||
| `prompt-injection-showcase/` | 61 payloads × 19 categories against `pre-prompt-inject-scan`, `post-mcp-verify`, `pre-bash-destructive` | runtime hooks | per-category expected outcome |
|
||
| `lethal-trifecta-walkthrough/` | Rule-of-Two advisory on leg 3 (WebFetch → Read .env → Bash curl POST) + suppression | `post-session-guard` | advisory at stage 3 |
|
||
| `mcp-rug-pull/` | Cumulative drift advisory (E14, v7.3.0) — 7 stages below the per-update threshold, cumulatively over a 25% baseline shift | `post-mcp-verify` + `mcp-description-cache.mjs` | advisory at stage 7 |
|
||
| `supply-chain-attack/` | PreToolUse block on a compromised package + scope-hop advisory + dep-auditor typosquats + postinstall curl-pipe | `pre-install-supply-chain` + `dep-auditor` + `supply-chain-data` | 6+ findings, 2 advisories, 1 BLOCK |
|
||
| `poisoned-claude-md/` | 6 detectors (injection / shell / URL / credential paths / permission expansion / encoded payloads) including the E15 agent-file surface | `memory-poisoning-scanner` | ≥18 findings split across 2 files |
|
||
| `bash-evasion-gallery/` | T1-T9 disguised destructive commands → normalized + blocked (defense-in-depth over Claude Code 2.1.98+) | `pre-bash-destructive` + `bash-normalize` | 10 BLOCK exit codes |
|
||
| `toxic-agent-demo/` | Single-component lethal trifecta — an agent with [Bash, Read, WebFetch] and no hook guards = CRITICAL TFA finding | `toxic-flow-analyzer` (TFA) | 1 CRITICAL `Lethal trifecta:` |
|
||
| `pre-compact-poisoning/` | The PreCompact hook catches injection + an AWS-shaped credential in a synthetic transcript across off/warn/block modes | `pre-compact-scan` | 9 cases: block exit 2 + reason; warn systemMessage; off skip; benign passes |
|
||
|
||
State isolation: every example that mutates global state uses the run-script PID (post-session-guard via `${ppid}.jsonl`) or env overrides (`LLM_SECURITY_MCP_CACHE_FILE` for the MCP cache). The user's real `/tmp/llm-security-session-*.jsonl` and `~/.cache/llm-security/` are never touched.
|