docs(readme): true self-scan number, AV false-positive section, demo form
.llm-security-ignore: MEM/SIG/UNI rules added only for by-design findings, one explaining line each — the vendored conformance corpus, the Cyrillic confusables in injection-patterns.mjs and the homoglyph test inputs, the changelog's quoted homoglyph, the grade-F posture fixture, and the miner names in the supply-chain blocklist. Not suppressed on purpose: MEM on CLAUDE.md (the file a poisoning would target) and AST/TRG (no by-design findings once the ast-scan fixtures left the disk). Header prefix list completed (14 scanners) and the "150" replaced by the measured count. Self-scan, fresh clone of HEAD (node scanners/scan-orchestrator.mjs): before: BLOCK 80/100, 77 findings (1C 9H 29M 2L 36I), 362 suppressed after: WARNING 61/100, 58 findings (0C 5H 15M 2L 36I), 381 suppressed without the ignore file: BLOCK 95/100, 439 findings (unchanged) On `.` (working tree incl. local-only files): BLOCK 80/100, 80 findings -> WARNING 64/100, 61 findings. README: - Self-scan: the measured numbers replace "0 findings (ALLOW)". - The demo commands materialize the tree to a temp dir (the form the demo README uses); no evil-project-health/ path left in README. - New "Antivirus false positives" section + TOC entry: what the repo contains, why a classifier can still flag it, restore from quarantine, Microsoft's submission form, Microsoft's own advice against exclusions, a sparse-checkout recipe. Every Defender claim checked against learn.microsoft.com; the recipe run live against Forgejo (partial clone supported; 285 of 677 files on disk). The meaning of "!ml" is not claimed — Microsoft documents "!" suffixes only as internal indicators. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
35359b7d8c
commit
d9787aefb5
2 changed files with 78 additions and 6 deletions
|
|
@ -1,6 +1,6 @@
|
||||||
# .llm-security-ignore — Suppress expected findings when scanning this plugin
|
# .llm-security-ignore — Suppress expected findings when scanning this plugin
|
||||||
#
|
#
|
||||||
# Why 150 suppressed findings? A security plugin that documents attack patterns,
|
# Why 381 suppressed findings? A security plugin that documents attack patterns,
|
||||||
# ships a malicious demo fixture, and tests against deliberately evil code will
|
# ships a malicious demo fixture, and tests against deliberately evil code will
|
||||||
# trigger its own scanners. This is the "scanning the scanner" paradox:
|
# trigger its own scanners. This is the "scanning the scanner" paradox:
|
||||||
#
|
#
|
||||||
|
|
@ -9,10 +9,12 @@
|
||||||
# - tests/ contain deliberate taint flows and suspicious URLs as test input
|
# - tests/ contain deliberate taint flows and suspicious URLs as test input
|
||||||
# - hooks/ and scanners/ contain high-entropy regex for secret detection
|
# - hooks/ and scanners/ contain high-entropy regex for secret detection
|
||||||
#
|
#
|
||||||
# Every suppression below is explained. Run without this file to see all 150.
|
# Every suppression below is explained. Run without this file to see all of them.
|
||||||
|
# Measured 2026-09-22 (v8.1.0) on a fresh clone: 439 findings without this file,
|
||||||
|
# 58 with it (381 suppressed).
|
||||||
#
|
#
|
||||||
# Format: SCANNER:glob or just glob (applies to all scanners)
|
# Format: SCANNER:glob or just glob (applies to all scanners)
|
||||||
# Scanners: UNI, ENT, PRM, DEP, TNT, GIT, NET, TFA
|
# Scanners: UNI, ENT, PRM, DEP, TNT, GIT, NET, MEM, SCR, WFL, TRG, SIG, AST, TFA
|
||||||
|
|
||||||
# Demo fixture: intentionally malicious (the whole point of the demo)
|
# Demo fixture: intentionally malicious (the whole point of the demo)
|
||||||
examples/**
|
examples/**
|
||||||
|
|
@ -65,3 +67,24 @@ NET:CLAUDE.md
|
||||||
|
|
||||||
# Baseline files: generated JSON with scan results (high entropy expected)
|
# Baseline files: generated JSON with scan results (high entropy expected)
|
||||||
reports/baselines/**
|
reports/baselines/**
|
||||||
|
|
||||||
|
# Unicode: the vendored commons conformance corpus carries homoglyph and zero-width inputs on purpose
|
||||||
|
UNI:scanners/commons/conformance/**
|
||||||
|
|
||||||
|
# Unicode: Cyrillic confusables are detection patterns here (the lexicon matches the homoglyph form)
|
||||||
|
UNI:scanners/lib/injection-patterns.mjs
|
||||||
|
|
||||||
|
# Unicode: homoglyph test inputs for the string-utils folding tests
|
||||||
|
UNI:tests/lib/string-utils-homoglyph.test.mjs
|
||||||
|
|
||||||
|
# Unicode: the changelog quotes the homoglyph form a fix detects
|
||||||
|
UNI:CHANGELOG.md
|
||||||
|
|
||||||
|
# Memory: the grade-F posture fixture is poisoned on purpose (that is what grade F tests)
|
||||||
|
MEM:tests/fixtures/posture-scan/grade-f-project/**
|
||||||
|
|
||||||
|
# Signature: the supply-chain blocklist names miner packages so the install hook can block them
|
||||||
|
SIG:scanners/lib/supply-chain-data.mjs
|
||||||
|
|
||||||
|
# Not suppressed on purpose: MEM findings on CLAUDE.md (it documents the hooks, but it is the
|
||||||
|
# file a poisoning would target), and AST/TRG, which have no by-design findings here.
|
||||||
|
|
|
||||||
55
README.md
55
README.md
|
|
@ -38,6 +38,7 @@ A Claude Code plugin that provides security scanning, auditing, and threat model
|
||||||
- [Migrating to v8.0.0](#migrating-to-v800)
|
- [Migrating to v8.0.0](#migrating-to-v800)
|
||||||
- [Playground (v7.6.0)](#playground-v760)
|
- [Playground (v7.6.0)](#playground-v760)
|
||||||
- [Self-scan](#self-scan)
|
- [Self-scan](#self-scan)
|
||||||
|
- [Antivirus false positives](#antivirus-false-positives)
|
||||||
- [Recent versions](#recent-versions)
|
- [Recent versions](#recent-versions)
|
||||||
- [Changelog](#changelog)
|
- [Changelog](#changelog)
|
||||||
- [License & attribution](#license--attribution)
|
- [License & attribution](#license--attribution)
|
||||||
|
|
@ -660,15 +661,18 @@ you start:
|
||||||
npm test # node --test 'tests/**/*.test.mjs'
|
npm test # node --test 'tests/**/*.test.mjs'
|
||||||
```
|
```
|
||||||
|
|
||||||
Running `node scanners/scan-orchestrator.mjs .` on this plugin produces **0 findings (ALLOW)** with ~190 suppressions via `.llm-security-ignore`. Every suppression is explained — a security plugin that documents attack patterns, ships a malicious demo fixture, and tests against deliberately evil code will trigger its own scanners. The entropy scanner flags regex patterns in `knowledge/secrets-patterns.md`. The taint scanner flags `eval(user_input)` in test fixtures. The toxic flow analyzer flags the plugin's own commands that use Read+Bash. Remove the ignore file and re-run to see the unsuppressed picture.
|
Running `node scanners/scan-orchestrator.mjs .` on a fresh clone produces **WARNING 61/100 with 58 findings** (0 critical, 5 high, 15 medium, 2 low, 36 info; measured 2026-09-22), after `.llm-security-ignore` suppresses 381. Without the ignore file the same run reports BLOCK 95/100 with 439 findings. A security plugin that documents attack patterns, ships a malicious demo, and tests against deliberately evil code triggers its own scanners; every suppression is explained in the file. What is left is reported on purpose: high-entropy strings in docs and the HTML report renderer, example and research domains in docs, three CLI tools that write to a path given on the command line, and memory findings on `CLAUDE.md` — it documents the hooks, but it is the file a poisoning would target, so it is not suppressed. Remove the ignore file and re-run to see the unsuppressed picture.
|
||||||
|
|
||||||
The `examples/malicious-skill-demo/` directory contains a deliberately malicious "Project Health Dashboard" plugin and a [full security assessment](examples/malicious-skill-demo/security-assessment.md). The combined LLM + deterministic pipeline produced **85 findings** (24 critical, 24 high, 20 medium, 6 low, 11 info) and verdict **BLOCK 100/100** — both layers independently maxed the risk score. A human reviewing the plugin's `README.md` and `SKILL.md` would likely miss most of them; the Unicode Tag steganography is literally invisible.
|
The `examples/malicious-skill-demo/` directory contains a deliberately malicious "Project Health Dashboard" plugin and a [full security assessment](examples/malicious-skill-demo/security-assessment.md). The combined LLM + deterministic pipeline produced **85 findings** (24 critical, 24 high, 20 medium, 6 low, 11 info) and verdict **BLOCK 100/100** — both layers independently maxed the risk score. A human reviewing the plugin's `README.md` and `SKILL.md` would likely miss most of them; the Unicode Tag steganography is literally invisible.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
node scanners/scan-orchestrator.mjs examples/malicious-skill-demo/evil-project-health/ # ~5s
|
DEMO="$(node examples/malicious-skill-demo/materialize.mjs)" # writes the demo to a temp dir, prints the path
|
||||||
/security scan examples/malicious-skill-demo/evil-project-health/ --deep # full pipeline
|
node scanners/scan-orchestrator.mjs "$DEMO" # ~5s
|
||||||
|
rm -rf "$(dirname "$DEMO")" # when you are done
|
||||||
```
|
```
|
||||||
|
|
||||||
|
For the full pipeline, materialize the same way and run `/security scan <printed path> --deep` in Claude Code.
|
||||||
|
|
||||||
### Other runnable examples
|
### Other runnable examples
|
||||||
|
|
||||||
The `examples/` directory contains additional self-contained
|
The `examples/` directory contains additional self-contained
|
||||||
|
|
@ -726,6 +730,51 @@ demonstrations — each with `README.md`, fixture, run script, and
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Antivirus false positives
|
||||||
|
|
||||||
|
This repository is a security tool, so it contains detection rules and descriptions of attack
|
||||||
|
techniques: signature tables (regular expressions for known-bad code), knowledge files that explain
|
||||||
|
how prompt injection, exfiltration and persistence attacks work, and a demo of a malicious plugin.
|
||||||
|
Since v8.1.0 no payload sits on disk as a file that would run it. Test fixtures and the demo are
|
||||||
|
stored encoded or built from fragments, and exist only in a temp directory while a test or the demo
|
||||||
|
runs; `tests/av-surface.test.mjs` fails the suite if one comes back.
|
||||||
|
|
||||||
|
An antivirus product can still flag a file. Microsoft Defender Antivirus does not only match known
|
||||||
|
samples: it also uses local and cloud-based machine-learning models, behavior analysis and
|
||||||
|
heuristics ([Next-generation protection overview](https://learn.microsoft.com/defender-endpoint/next-generation-protection)).
|
||||||
|
A text file that explains many attack techniques in one place can be flagged although nothing in it
|
||||||
|
runs — a plain `SKILL.md` in another security repository was quarantined as
|
||||||
|
`Trojan:Script/Wacatac.H!ml`.
|
||||||
|
|
||||||
|
If Defender quarantines a file from this repository:
|
||||||
|
|
||||||
|
1. **Restore it:** Windows Security → Virus & threat protection → Protection history, select the
|
||||||
|
item, then **Restore** ([Restore quarantined files](https://learn.microsoft.com/defender-endpoint/restore-quarantined-files-microsoft-defender-antivirus)).
|
||||||
|
2. **Report the false positive** to Microsoft at <https://www.microsoft.com/wdsi/filesubmission>, so
|
||||||
|
Microsoft can review and correct the detection.
|
||||||
|
3. Open an issue in the marketplace repository with the file path and the detection name.
|
||||||
|
|
||||||
|
Excluding the folder is not recommended here, and Microsoft advises against it as a fix: "Every
|
||||||
|
exclusion is a protection gap that lowers your defenses, so use exclusions sparingly"
|
||||||
|
([Exclusions in Microsoft Defender Antivirus](https://learn.microsoft.com/defender-endpoint/microsoft-defender-antivirus-exclusions-overview)).
|
||||||
|
|
||||||
|
If you only want the scanners and hooks, a sparse checkout leaves knowledge files, tests, examples
|
||||||
|
and docs off disk (measured 2026-09-22: 285 of 677 files):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone --filter=blob:none --no-checkout https://git.fromaitochitta.com/open/llm-security.git
|
||||||
|
cd llm-security
|
||||||
|
git sparse-checkout set scanners hooks
|
||||||
|
git checkout main
|
||||||
|
```
|
||||||
|
|
||||||
|
Cone mode also checks out the files at the repository root. Some scanners read data from
|
||||||
|
`knowledge/` (the typosquat lists, the attack simulator's scenarios); add `knowledge` to the `set`
|
||||||
|
line if you run those. A sparse checkout of your own clone does not change what
|
||||||
|
`claude plugin install` puts in the plugin cache.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Recent versions
|
## Recent versions
|
||||||
|
|
||||||
| Version | Date | Highlights |
|
| Version | Date | Highlights |
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue