open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Releases 6 Activity

feat(watch): source list for the 16 catalogs and the 30-day release rule

Browse source 
docs/threat-2026/watch-sources.json is the catalog register in the edition
watch's input format: 12 entries probed (A, B, F, L x2, O x7), 4 manual as
class S with a reason. Run live on 2026-11-01: 16 current, 12 probes ok.

A frozen copy of the catalogs as v1.4.1 used them is reported behind
(OWASP LLM 2025) and on watch (RAG cheat sheet); the watch calls its Unicode
17.0 current, which only the 30-day clause catches.

tests/test_release_rule.py stops a release whose manual check or watch run
is more than 30 days older than the newest CHANGELOG date, or whose watch
output has a behind, notVerified or watch row. Each clause mutated away
turns a test red (5 of 5). No release: the package is unchanged.
This commit is contained in:
Kjell Tore Guttormsen 2026-11-01 12:50:51 +01:00
commit c4ecf0dba6
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
7 changed files with 973 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

64
docs/PLAN.md
View file

@ -24,7 +24,7 @@ Lest slik (antakelser, ikke operatørens ord):
- Ikke tatt med (operatøren valgte bort «bare rydding»): blokk-skalaren `a #b`,
støyen i ReDoS-sveipet og repo-standard-sjekken. De står under § V.5.
**Dekket i:** katalogene over tid → § V.2 rad 1, økt 2 · hullene → rad 2, økt 1
**Dekket i:** katalogene over tid → § V.2 rad 1, økt 2 (§ V.7) · hullene → rad 2, økt 1
(måling) og økt 3 og utover (lukking) · ferdig → siste økt.
---
@ -190,6 +190,68 @@ så de kommer med i samme release som den første lukkingen.
---
## V.7 Økt 2, vakten (01.11.2026, ingen release)
**Kildelista** er `docs/threat-2026/watch-sources.json`: de 16 i registeret, i vaktens
egen form (en liste med ett `probe`-objekt per oppføring). Den er generert fra
`catalogs.json`, og `tests/test_release_rule.py` holder dem like: samme id-er, samme
`cited`, `latest`, `source` og `checked`. 12 oppføringer har en probe som gjør en
forespørsel (A, B, F, L×2, O×7, som målt i § V.6). De 4 manuelle (unicode, rfc-3986,
commons, Willison) står med klasse S. Det er vaktens egen form for «ingen maskin
kan svare» (llm-security bruker den på 6 av sine 58). Hver har en begrunnelse som
starter med `manual:`, og sjekkdatoen er registerets `checked`.
**Kjørt live 01.11** med vakten ved `45d3459`, med `--used` og en egen last-run-fil
i scratch. llm-securitys stempel hadde samme sha256 før og etter.
| Liste | behind | watch | notVerified | current | Prober ok | exit |
|---|---|---|---|---|---|---|
| `watch-sources.json` | 0 | 0 | 0 | 16 | 12 | 0 |
| `watch-sources-v1.4.1.json` (frosset) | 1: owasp-llm-top10 (2025 → 2026) | 1: owasp-rag-cheat-sheet (41f67299 → 510ea484) | 0 | 12 | 10 | 1 |
Utdataene er committet som `watch-last.json` og `watch-v1.4.1-run.json`.
**Den frosne v1.4.1-kopien** inneholder bare det v1.4.1 (tagget 2026-09-23) siterte,
og hver oppføring har utgaven utgiveren serverte den dagen. OWASP Agentic og ATLAS er
utelatt fordi v1.4.1 ikke siterte dem. RAG-arket sto på `41f67299` (2026-05-06), lest
fra GitHubs commit-API med `until=2026-09-23`. Uversjonerte arXiv-lenker gir de samme
versjonene som nå, fordi alle ble publisert før tag-dagen. Fila er låst med sha256 i
testen.
**Funn:** vakten melder Unicode 17.0 som `current`, fordi klasse S ikke spør. Det er
bare 30-dagersregelen som fanger den. Positiv kontroll med alle seks problemene: 4
manuelle sjekker fra 23.09, 1 behind og 1 watch.
**Release-regelen** (`tests/test_release_rule.py`) leser releasedatoen fra den
nyeste CHANGELOG-overskriften. Den binder derfor på release-commiten og ikke på en
vilkårlig dag mellom to releaser. En test holder overskriften lik versjonen i
`pyproject.toml`. Regelen stopper når:
- en manuell sjekk er mer enn 30 dager eldre enn releasedatoen;
- vaktens siste kjøring er mer enn 30 dager eldre (den er sjekkdatoen for de 12);
- utdataene ikke er en kjøring av denne lista;
- det finnes en rad i `behind`, `notVerified` eller `watch`. En `watch`-rad teller
aldri med i vaktens exit-kode, så regelen leser den for seg. Den ryddes av en
sjekk for hånd som oppdaterer `known_value`, og deretter en ny kjøring.
Hver klausul er mutert bort én om gangen, og hver mutasjon gjorde en test rød (5 av 5).
**Før en release:** kjør
`LLM_SECURITY_EDITION_WATCH_LAST_RUN=<scratch>/lr.json node <llm-security>/scripts/edition-watch.mjs --used docs/threat-2026/watch-sources.json > docs/threat-2026/watch-last.json`
og commit utdataene. Uten den miljøvariabelen stempler kjøringen llm-securitys egen
last-run-fil.
**Ukentlig kjøring:** foreslått til llm-security via coord 01.11. Det er deres jobb
(`run.sh` i deres tilstandsmappe, launchd mandager 09:15), så den endres ikke herfra.
Rad 1 er ikke framme før jobben har kjørt lista vår de siste 7 dagene.
**Sett, ikke endret:** commons-proben (N) svarer `ok` under llm-securitys jobb, som
setter en ssh-remote. Deres `last.json` fra 26.10 har 0 i `notVerified`. Commons blir
likevel stående manuell her. llm-securitys egen liste følger de samme commons-taggene
hver uke. Under en annen kjøring uten ssh-remote ville proben her svare
`auth-required`.
Ingen release, fordi ingenting i den utgitte pakken er endret.
## Forrige retning (godkjent av operatøren 28.10.2026, nådd 31.10)
> «llm-ingestion-guard bygger på nyeste utgave av hver trusselkatalog den viser til,

25
docs/threat-2026/watch-last.json Normal file
View file

@ -0,0 +1,25 @@
{
"behind": [],
"notVerified": [],
"watch": [],
"current": [
"owasp-llm-top10",
"owasp-agentic-top10",
"mitre-atlas",
"owasp-rag-cheat-sheet",
"unicode",
"cve-2025-32711",
"arxiv-2402.07867",
"arxiv-2504.11168",
"arxiv-2509.14285",
"arxiv-2505.03574",
"arxiv-2503.18813",
"arxiv-2604.00387",
"arxiv-2605.00460",
"willison-dual-llm",
"rfc-3986",
"llm-security-commons"
],
"lastRun": "2026-11-01T11:45:11.499Z",
"weeklyJobLastRun": "never"
}

290
docs/threat-2026/watch-sources-v1.4.1.json Normal file
View file

@ -0,0 +1,290 @@
[
{
"id": "owasp-llm-top10",
"name": "OWASP Top 10 for LLM Applications",
"kind": "catalog",
"location": "Finding.owasp on every finding; README, LIMITATIONS, coverage matrix",
"used_edition": "2025",
"newest_edition": "2026",
"newest_source_url": "https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/",
"checked": "2026-10-31",
"probe": {
"class": "A",
"signal": "edition-year",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": "2025",
"known_at": "2026-09-23"
}
},
{
"id": "owasp-rag-cheat-sheet",
"name": "Retrieval-Augmented Generation (RAG) Security Cheat Sheet",
"kind": "guidance",
"location": "README \u00a7 Design & threat model (linked, with the check date)",
"used_edition": "living document, commit 41f67299 (2026-05-06), the newest on 2026-09-23",
"newest_edition": "living document, last modified 2026-10-04 (commit 510ea484)",
"newest_source_url": "https://cheatsheetseries.owasp.org/cheatsheets/RAG_Security_Cheat_Sheet.html",
"checked": "2026-10-31",
"probe": {
"class": "L",
"signal": "watch-only",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"url": "https://api.github.com/repos/OWASP/CheatSheetSeries/commits?path=cheatsheets/RAG_Security_Cheat_Sheet.md&per_page=1",
"parse": "0.sha",
"known_value": "41f67299945eeb85cc4387e073bea3d8343ad92d",
"known_at": "2026-09-23"
}
},
{
"id": "unicode",
"name": "The Unicode Standard (UCD)",
"kind": "standard",
"location": "invisible-character tables (sanitize, output); the character denominator in docs/threat-2026/codepoints.json",
"used_edition": "17.0",
"newest_edition": "18.0.0",
"newest_source_url": "https://www.unicode.org/versions/Unicode18.0.0/",
"checked": "2026-10-31",
"probe": {
"class": "S",
"signal": "none",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": null,
"reason": "manual: unicode.org is not on the watch's host list and the UCD ReadMe is text, not JSON; unicodetools on GitHub has final-17.0 as its newest release while 18.0 is current, so no probe class can read the edition",
"known_at": "2026-09-23"
}
},
{
"id": "cve-2025-32711",
"name": "CVE-2025-32711 (EchoLeak), M365 Copilot Information Disclosure",
"kind": "advisory",
"location": "active-content detector docs, coverage matrix, README",
"used_edition": "CVE-2025-32711",
"newest_edition": "MSRC revision 2026-02-20; NVD last modified 2026-06-17",
"newest_source_url": "https://nvd.nist.gov/vuln/detail/CVE-2025-32711",
"checked": "2026-10-31",
"probe": {
"class": "L",
"signal": "watch-only",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"url": "https://raw.githubusercontent.com/CVEProject/cvelistV5/main/cves/2025/32xxx/CVE-2025-32711.json",
"parse": "cveMetadata.dateUpdated",
"known_value": "2026-02-26T17:50:41.404Z",
"known_at": "2026-09-23"
}
},
{
"id": "arxiv-2402.07867",
"name": "PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models",
"kind": "paper",
"location": "README \u00a7 Design & threat model (linked); semantic-poisoning gap",
"used_edition": "2402.07867 (no version)",
"newest_edition": "v3",
"newest_source_url": "https://arxiv.org/abs/2402.07867",
"checked": "2026-10-31",
"probe": {
"class": "O",
"signal": "version-field",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": [
"2402.07867v3"
],
"known_at": "2026-09-23"
}
},
{
"id": "arxiv-2504.11168",
"name": "Bypassing LLM Guardrails: An Empirical Analysis of Evasion Attacks against Prompt Injection and Jailbreak Detection Systems",
"kind": "paper",
"location": "README and BRIEF, detection-is-evadable claim",
"used_edition": "2504.11168 (no version)",
"newest_edition": "v3",
"newest_source_url": "https://arxiv.org/abs/2504.11168",
"checked": "2026-10-31",
"probe": {
"class": "O",
"signal": "version-field",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": [
"2504.11168v3"
],
"known_at": "2026-09-23"
}
},
{
"id": "arxiv-2509.14285",
"name": "A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks",
"kind": "paper",
"location": "docs/BRIEF.md",
"used_edition": "2509.14285 (no version)",
"newest_edition": "v4",
"newest_source_url": "https://arxiv.org/abs/2509.14285",
"checked": "2026-10-31",
"probe": {
"class": "O",
"signal": "version-field",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": [
"2509.14285v4"
],
"known_at": "2026-09-23"
}
},
{
"id": "arxiv-2505.03574",
"name": "LlamaFirewall: An open source guardrail system for building secure AI agents",
"kind": "paper",
"location": "docs/BRIEF.md",
"used_edition": "2505.03574 (no version)",
"newest_edition": "v1",
"newest_source_url": "https://arxiv.org/abs/2505.03574",
"checked": "2026-10-31",
"probe": {
"class": "O",
"signal": "version-field",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": [
"2505.03574v1"
],
"known_at": "2026-09-23"
}
},
{
"id": "arxiv-2503.18813",
"name": "Defeating Prompt Injections by Design (CaMeL)",
"kind": "paper",
"location": "architecture lineage",
"used_edition": "2503.18813 (no version)",
"newest_edition": "v2",
"newest_source_url": "https://arxiv.org/abs/2503.18813",
"checked": "2026-10-31",
"probe": {
"class": "O",
"signal": "version-field",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": [
"2503.18813v2"
],
"known_at": "2026-09-23"
}
},
{
"id": "arxiv-2604.00387",
"name": "RAGShield: Detecting Numerical Claim Manipulation in Government RAG Systems",
"kind": "paper",
"location": "docs/PLAN.md v1 anchors (history)",
"used_edition": "2604.00387 (no version), labelled 'RAGShield'",
"newest_edition": "v2",
"newest_source_url": "https://arxiv.org/abs/2604.00387",
"checked": "2026-10-31",
"probe": {
"class": "O",
"signal": "version-field",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": [
"2604.00387v2"
],
"known_at": "2026-09-23"
}
},
{
"id": "arxiv-2605.00460",
"name": "CleanBase: Detecting Malicious Documents in RAG Knowledge Databases",
"kind": "paper",
"location": "docs/BRIEF.md \u00a7 11, write-time RAG-DB poisoning is under-tooled",
"used_edition": "2605.00460 (no version, pdf link)",
"newest_edition": "v1",
"newest_source_url": "https://arxiv.org/abs/2605.00460",
"checked": "2026-10-31",
"probe": {
"class": "O",
"signal": "version-field",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": [
"2605.00460v1"
],
"known_at": "2026-09-23"
}
},
{
"id": "willison-dual-llm",
"name": "The Dual LLM pattern for building AI assistants that can resist prompt injection",
"kind": "article",
"location": "architecture lineage",
"used_edition": "by name, 2023",
"newest_edition": "2023-04-25 (single post)",
"newest_source_url": "https://simonwillison.net/2023/Apr/25/dual-llm-pattern/",
"checked": "2026-10-31",
"probe": {
"class": "S",
"signal": "none",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": null,
"reason": "manual: a single dated blog post with no editions; there is nothing for a probe to follow",
"known_at": "2026-09-23"
}
},
{
"id": "rfc-3986",
"name": "RFC 3986, Uniform Resource Identifier (URI): Generic Syntax (STD 66)",
"kind": "standard",
"location": "active_content.py scheme grammar",
"used_edition": "RFC 3986",
"newest_edition": "RFC 3986, INTERNET STANDARD, not obsoleted (updated by RFC 7320, RFC 8820)",
"newest_source_url": "https://www.rfc-editor.org/rfc/rfc3986.json",
"checked": "2026-10-31",
"probe": {
"class": "S",
"signal": "none",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": null,
"reason": "manual: rfc-editor.org is not on the watch's host list; its rfc3986.json (obsoleted_by empty) would serve a class L probe once the host is added",
"known_at": "2026-09-23"
}
},
{
"id": "llm-security-commons",
"name": "llm-security-commons (injection lexicon, carrier code points)",
"kind": "data-source",
"location": "src/llm_ingestion_guard/injection_lexicon.json",
"used_edition": "lexicon 0.8.0 (83 patterns, byte-equal copy)",
"newest_edition": "v0.9.0 (lexicon/injection-lexicon.json 0.8.0, codepoints/carriers.json 0.2.0)",
"newest_source_url": "ssh://git@git.fromaitochitta.com/ktg/llm-security-commons.git",
"checked": "2026-10-31",
"probe": {
"class": "S",
"signal": "none",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": null,
"reason": "manual: the repository is private, so the class N probe answers auth-required without credentials; llm-security's own source list watches the same commons tags under its weekly job",
"known_at": "2026-09-23"
}
}
]

329
docs/threat-2026/watch-sources.json Normal file
View file

@ -0,0 +1,329 @@
[
{
"id": "owasp-llm-top10",
"name": "OWASP Top 10 for LLM Applications",
"kind": "catalog",
"location": "Finding.owasp on every finding; README, LIMITATIONS, coverage matrix",
"used_edition": "2026",
"newest_edition": "2026",
"newest_source_url": "https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/",
"checked": "2026-10-31",
"probe": {
"class": "A",
"signal": "edition-year",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": "2026",
"known_at": "2026-11-01"
}
},
{
"id": "owasp-agentic-top10",
"name": "OWASP Top 10 for Agentic Applications",
"kind": "catalog",
"location": "README \u00a7 What it protects against (ASI06)",
"used_edition": "for 2026",
"newest_edition": "for 2026",
"newest_source_url": "https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/",
"checked": "2026-10-31",
"probe": {
"class": "B",
"signal": "edition-year",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": "2026",
"known_at": "2026-11-01"
}
},
{
"id": "mitre-atlas",
"name": "MITRE ATLAS",
"kind": "catalog",
"location": "README \u00a7 What it protects against",
"used_edition": "v2026.10",
"newest_edition": "v2026.10",
"newest_source_url": "https://github.com/mitre-atlas/atlas-data/releases/tag/v2026.10",
"checked": "2026-10-31",
"probe": {
"class": "F",
"signal": "stable-release",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"url": "https://api.github.com/repos/mitre-atlas/atlas-data/releases/latest",
"known_value": "v2026.10",
"known_at": "2026-11-01"
}
},
{
"id": "owasp-rag-cheat-sheet",
"name": "Retrieval-Augmented Generation (RAG) Security Cheat Sheet",
"kind": "guidance",
"location": "README \u00a7 Design & threat model (linked, with the check date)",
"used_edition": "living document, last modified 2026-10-04 (commit 510ea484)",
"newest_edition": "living document, last modified 2026-10-04 (commit 510ea484)",
"newest_source_url": "https://cheatsheetseries.owasp.org/cheatsheets/RAG_Security_Cheat_Sheet.html",
"checked": "2026-10-31",
"probe": {
"class": "L",
"signal": "watch-only",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"url": "https://api.github.com/repos/OWASP/CheatSheetSeries/commits?path=cheatsheets/RAG_Security_Cheat_Sheet.md&per_page=1",
"parse": "0.sha",
"known_value": "510ea484cfc304ec2722289237e936bc372eee34",
"known_at": "2026-11-01"
}
},
{
"id": "unicode",
"name": "The Unicode Standard (UCD)",
"kind": "standard",
"location": "invisible-character tables (sanitize, output); the character denominator in docs/threat-2026/codepoints.json",
"used_edition": "18.0.0",
"newest_edition": "18.0.0",
"newest_source_url": "https://www.unicode.org/versions/Unicode18.0.0/",
"checked": "2026-10-31",
"probe": {
"class": "S",
"signal": "none",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": null,
"reason": "manual: unicode.org is not on the watch's host list and the UCD ReadMe is text, not JSON; unicodetools on GitHub has final-17.0 as its newest release while 18.0 is current, so no probe class can read the edition",
"known_at": "2026-10-31"
}
},
{
"id": "cve-2025-32711",
"name": "CVE-2025-32711 (EchoLeak), M365 Copilot Information Disclosure",
"kind": "advisory",
"location": "active-content detector docs, coverage matrix, README",
"used_edition": "MSRC revision 2026-02-20; NVD last modified 2026-06-17",
"newest_edition": "MSRC revision 2026-02-20; NVD last modified 2026-06-17",
"newest_source_url": "https://nvd.nist.gov/vuln/detail/CVE-2025-32711",
"checked": "2026-10-31",
"probe": {
"class": "L",
"signal": "watch-only",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"url": "https://raw.githubusercontent.com/CVEProject/cvelistV5/main/cves/2025/32xxx/CVE-2025-32711.json",
"parse": "cveMetadata.dateUpdated",
"known_value": "2026-02-26T17:50:41.404Z",
"known_at": "2026-11-01"
}
},
{
"id": "arxiv-2402.07867",
"name": "PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models",
"kind": "paper",
"location": "README \u00a7 Design & threat model (linked); semantic-poisoning gap",
"used_edition": "v3",
"newest_edition": "v3",
"newest_source_url": "https://arxiv.org/abs/2402.07867",
"checked": "2026-10-31",
"probe": {
"class": "O",
"signal": "version-field",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": [
"2402.07867v3"
],
"known_at": "2026-11-01"
}
},
{
"id": "arxiv-2504.11168",
"name": "Bypassing LLM Guardrails: An Empirical Analysis of Evasion Attacks against Prompt Injection and Jailbreak Detection Systems",
"kind": "paper",
"location": "README and BRIEF, detection-is-evadable claim",
"used_edition": "v3",
"newest_edition": "v3",
"newest_source_url": "https://arxiv.org/abs/2504.11168",
"checked": "2026-10-31",
"probe": {
"class": "O",
"signal": "version-field",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": [
"2504.11168v3"
],
"known_at": "2026-11-01"
}
},
{
"id": "arxiv-2509.14285",
"name": "A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks",
"kind": "paper",
"location": "docs/BRIEF.md",
"used_edition": "v4",
"newest_edition": "v4",
"newest_source_url": "https://arxiv.org/abs/2509.14285",
"checked": "2026-10-31",
"probe": {
"class": "O",
"signal": "version-field",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": [
"2509.14285v4"
],
"known_at": "2026-11-01"
}
},
{
"id": "arxiv-2505.03574",
"name": "LlamaFirewall: An open source guardrail system for building secure AI agents",
"kind": "paper",
"location": "docs/BRIEF.md",
"used_edition": "v1",
"newest_edition": "v1",
"newest_source_url": "https://arxiv.org/abs/2505.03574",
"checked": "2026-10-31",
"probe": {
"class": "O",
"signal": "version-field",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": [
"2505.03574v1"
],
"known_at": "2026-11-01"
}
},
{
"id": "arxiv-2503.18813",
"name": "Defeating Prompt Injections by Design (CaMeL)",
"kind": "paper",
"location": "architecture lineage",
"used_edition": "v2",
"newest_edition": "v2",
"newest_source_url": "https://arxiv.org/abs/2503.18813",
"checked": "2026-10-31",
"probe": {
"class": "O",
"signal": "version-field",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": [
"2503.18813v2"
],
"known_at": "2026-11-01"
}
},
{
"id": "arxiv-2604.00387",
"name": "RAGShield: Detecting Numerical Claim Manipulation in Government RAG Systems",
"kind": "paper",
"location": "docs/PLAN.md v1 anchors (history)",
"used_edition": "v2",
"newest_edition": "v2",
"newest_source_url": "https://arxiv.org/abs/2604.00387",
"checked": "2026-10-31",
"probe": {
"class": "O",
"signal": "version-field",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": [
"2604.00387v2"
],
"known_at": "2026-11-01"
}
},
{
"id": "arxiv-2605.00460",
"name": "CleanBase: Detecting Malicious Documents in RAG Knowledge Databases",
"kind": "paper",
"location": "docs/BRIEF.md \u00a7 11, write-time RAG-DB poisoning is under-tooled",
"used_edition": "v1",
"newest_edition": "v1",
"newest_source_url": "https://arxiv.org/abs/2605.00460",
"checked": "2026-10-31",
"probe": {
"class": "O",
"signal": "version-field",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": [
"2605.00460v1"
],
"known_at": "2026-11-01"
}
},
{
"id": "willison-dual-llm",
"name": "The Dual LLM pattern for building AI assistants that can resist prompt injection",
"kind": "article",
"location": "architecture lineage",
"used_edition": "2023-04-25 (single post)",
"newest_edition": "2023-04-25 (single post)",
"newest_source_url": "https://simonwillison.net/2023/Apr/25/dual-llm-pattern/",
"checked": "2026-10-31",
"probe": {
"class": "S",
"signal": "none",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": null,
"reason": "manual: a single dated blog post with no editions; there is nothing for a probe to follow",
"known_at": "2026-10-31"
}
},
{
"id": "rfc-3986",
"name": "RFC 3986, Uniform Resource Identifier (URI): Generic Syntax (STD 66)",
"kind": "standard",
"location": "active_content.py scheme grammar",
"used_edition": "RFC 3986, INTERNET STANDARD, not obsoleted (updated by RFC 7320, RFC 8820)",
"newest_edition": "RFC 3986, INTERNET STANDARD, not obsoleted (updated by RFC 7320, RFC 8820)",
"newest_source_url": "https://www.rfc-editor.org/rfc/rfc3986.json",
"checked": "2026-10-31",
"probe": {
"class": "S",
"signal": "none",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": null,
"reason": "manual: rfc-editor.org is not on the watch's host list; its rfc3986.json (obsoleted_by empty) would serve a class L probe once the host is added",
"known_at": "2026-10-31"
}
},
{
"id": "llm-security-commons",
"name": "llm-security-commons (injection lexicon, carrier code points)",
"kind": "data-source",
"location": "src/llm_ingestion_guard/injection_lexicon.json",
"used_edition": "v0.9.0 (lexicon/injection-lexicon.json 0.8.0, codepoints/carriers.json 0.2.0)",
"newest_edition": "v0.9.0 (lexicon/injection-lexicon.json 0.8.0, codepoints/carriers.json 0.2.0)",
"newest_source_url": "ssh://git@git.fromaitochitta.com/ktg/llm-security-commons.git",
"checked": "2026-10-31",
"probe": {
"class": "S",
"signal": "none",
"timeout_ms": 20000,
"max_bytes": 1048576,
"cadence_days": 7,
"known_value": null,
"reason": "manual: the repository is private, so the class N probe answers auth-required without credentials; llm-security's own source list watches the same commons tags under its weekly job",
"known_at": "2026-10-31"
}
}
]

36
docs/threat-2026/watch-v1.4.1-run.json Normal file
View file

@ -0,0 +1,36 @@
{
"behind": [
{
"id": "owasp-llm-top10",
"source": "OWASP Top 10 for LLM Applications",
"used": "2025",
"newest": "2026",
"url": "https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/",
"probe_status": "ok"
}
],
"notVerified": [],
"watch": [
{
"id": "owasp-rag-cheat-sheet",
"signal": "watch-only",
"observed": "510ea484cfc304ec2722289237e936bc372eee34"
}
],
"current": [
"unicode",
"cve-2025-32711",
"arxiv-2402.07867",
"arxiv-2504.11168",
"arxiv-2509.14285",
"arxiv-2505.03574",
"arxiv-2503.18813",
"arxiv-2604.00387",
"arxiv-2605.00460",
"willison-dual-llm",
"rfc-3986",
"llm-security-commons"
],
"lastRun": "2026-11-01T11:47:10.999Z",
"weeklyJobLastRun": "never"
}

7
tests/test_catalog_register.py
View file

@ -6,9 +6,10 @@ is `docs/threat-2026/catalogs.json`; its denominator was frozen at 15 entries in
session 1 and may grow, never shrink, without the operator's yes.
This file fails when an entry lacks a date or a link, or cites an edition other
than the one recorded as latest. It deliberately does **not** fail on age: an
automatic "this check is stale" alarm is outside the approved direction, so a date
from last year passes as long as it is a date.
than the one recorded as latest. It does not fail on age: that is the release
rule in `tests/test_release_rule.py` (direction of 2026-11-01), which judges the
check dates against the release date, so the suite does not turn red merely
because a day passed between releases.
The README may only name an OWASP Agentic or MITRE ATLAS entry the register
records as read at the publisher, and must name the ATLAS release the register

226
tests/test_release_rule.py Normal file
View file

@ -0,0 +1,226 @@
"""The release rule: no release with a catalog check older than 30 days, row 1 of the 2026-11 done-criterion.
Row 1 (`docs/PLAN.md` § V.2) asks that each of the 16 catalogs in
`docs/threat-2026/catalogs.json` is followed by llm-security's edition watch, or
stands as manual with a reason, and that a test stops a release whose manual check
is more than 30 days older than the release date.
The source list `docs/threat-2026/watch-sources.json` is the register in the
watch's own input format: 12 entries carry a probe that makes a request, and 4
carry class S (no request) with the reason no probe can answer. The newest watch
output for that list is committed as `docs/threat-2026/watch-last.json`.
The rule, read against the release date (the date in the newest CHANGELOG
heading, so it binds on the release commit that writes it and on nothing else):
- every manual entry's ``known_at`` is at most 30 days before the release date;
- the watch output is a run of this list (the same ids) and its ``lastRun`` is at
most 30 days before the release date, since that run is the check date of the
12 probed entries;
- the output has no ``behind`` row, no ``notVerified`` row and no ``watch`` row. A
``watch`` row (class L: a living document or a CVE record moved) never counts
as behind in the watch's own exit code, so the rule reads it separately: it is
cleared by a check by hand that updates the entry's ``known_value``, then a new
run.
To release: run the watch on the list (command in `docs/PLAN.md` § V.7) and commit
its output here. The positive control is a frozen copy of the catalogs as v1.4.1
(tagged 2026-09-23) used them, `watch-sources-v1.4.1.json`, and its live output
`watch-v1.4.1-run.json`: the watch reports it behind, and this rule stops it.
"""
from __future__ import annotations
import datetime
import hashlib
import json
import re
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
THREAT = ROOT / "docs" / "threat-2026"
REGISTER = THREAT / "catalogs.json"
SOURCES = THREAT / "watch-sources.json"
LAST_RUN = THREAT / "watch-last.json"
SOURCES_V141 = THREAT / "watch-sources-v1.4.1.json"
RUN_V141 = THREAT / "watch-v1.4.1-run.json"
CHANGELOG = ROOT / "CHANGELOG.md"
MAX_AGE_DAYS = 30
SOURCES_V141_SHA256 = "49551e7c142488b3a86da099583828471ed3d1bf8ed8875f7439865b00dd10da"
MANUAL_IDS = frozenset({"unicode", "rfc-3986", "llm-security-commons", "willison-dual-llm"})
# Probe classes that make no request (the watch's own NO_REQUEST_CLASSES).
NO_REQUEST = frozenset({"S", "R-age"})
_HEADING = re.compile(r"^## \[(\d+\.\d+\.\d+)\] \S+ (\d{4}-\d{2}-\d{2})\s*$", re.M)
def _json(path: Path):
return json.loads(path.read_text(encoding="utf-8"))
def release_date() -> datetime.date:
"""The date of the newest CHANGELOG heading: the release the tree describes."""
m = _HEADING.search(CHANGELOG.read_text(encoding="utf-8"))
assert m, "no release heading in CHANGELOG.md"
return datetime.date.fromisoformat(m.group(2))
def _run_ids(run: dict) -> list[str]:
ids = [r["id"] for key in ("behind", "notVerified", "watch") for r in run[key]]
return ids + list(run["current"])
def problems(sources: list[dict], run: dict, released: datetime.date) -> list[str]:
"""Why a release on ``released`` must stop; empty when it may go."""
out = []
oldest = released - datetime.timedelta(days=MAX_AGE_DAYS)
for e in sources:
if e["probe"]["class"] in NO_REQUEST:
checked = datetime.date.fromisoformat(e["probe"]["known_at"])
if checked < oldest:
out.append(f"manual check of {e['id']} on {checked} is more than {MAX_AGE_DAYS} days before {released}")
if sorted(_run_ids(run)) != sorted(e["id"] for e in sources):
out.append("the watch output is not a run of this source list")
ran = datetime.datetime.fromisoformat(run["lastRun"].replace("Z", "+00:00")).date()
if ran < oldest:
out.append(f"the watch last ran on {ran}, more than {MAX_AGE_DAYS} days before {released}")
for key in ("behind", "notVerified", "watch"):
for r in run[key]:
out.append(f"{key}: {r['id']}")
return out
# --- the rule on the tree as it stands -----------------------------------------
def test_this_tree_may_be_released():
assert problems(_json(SOURCES), _json(LAST_RUN), release_date()) == []
# --- the source list is the register in the watch's format -----------------------
def test_source_list_covers_every_register_entry():
register = {e["id"]: e for e in _json(REGISTER)["entries"]}
sources = _json(SOURCES)
assert [e["id"] for e in sources] == list(register)
for e in sources:
r = register[e["id"]]
assert e["used_edition"] == r["cited"], e["id"]
assert e["newest_edition"] == r["latest"], e["id"]
assert e["newest_source_url"] == r["source"], e["id"]
assert e["checked"] == r["checked"], e["id"]
def test_twelve_probed_and_four_manual_with_a_reason():
sources = _json(SOURCES)
manual = {e["id"] for e in sources if e["probe"]["class"] in NO_REQUEST}
assert manual == MANUAL_IDS
assert len(sources) - len(manual) == 12
register = {e["id"]: e for e in _json(REGISTER)["entries"]}
for e in sources:
p = e["probe"]
datetime.date.fromisoformat(p["known_at"])
if e["id"] in MANUAL_IDS:
assert p["class"] == "S" and p["known_value"] is None
assert p["reason"].startswith("manual: "), e["id"]
# The manual check date is the register's check date: one truth.
assert p["known_at"] == register[e["id"]]["checked"], e["id"]
else:
assert p["known_value"], e["id"]
def test_arxiv_entries_stand_one_per_paper():
# Seven ids in one entry exceed the watch's 80-character VERSION_RE (PLAN § V.6).
for e in _json(SOURCES):
if e["probe"]["class"] == "O":
assert e["probe"]["known_value"] == [e["id"].removeprefix("arxiv-") + e["used_edition"]]
# --- the positive control: the catalogs as v1.4.1 used them ---------------------
def test_v141_copy_is_frozen():
assert hashlib.sha256(SOURCES_V141.read_bytes()).hexdigest() == SOURCES_V141_SHA256
def test_v141_copy_holds_only_what_v141_cited():
register = {e["id"]: e for e in _json(REGISTER)["entries"]}
copy = {e["id"]: e for e in _json(SOURCES_V141)}
assert set(copy) == {i for i, r in register.items() if r["cited_at_v1_4_1"] != "not cited"}
assert copy["owasp-llm-top10"]["used_edition"] == "2025"
assert copy["unicode"]["used_edition"] == "17.0"
def test_watch_reports_v141_behind_and_the_rule_stops_it():
sources, run = _json(SOURCES_V141), _json(RUN_V141)
assert [r["id"] for r in run["behind"]] == ["owasp-llm-top10"]
assert [(r["id"], r["observed"][:8]) for r in run["watch"]] == [("owasp-rag-cheat-sheet", "510ea484")]
# The watch says "current" for unicode at 17.0 (class S makes no request);
# only the 30-day clause catches it, together with the other three manual
# entries whose last check was the v1.4.1 tag date.
assert "unicode" in run["current"]
found = problems(sources, run, datetime.date(2026, 11, 1))
assert found == [
f"manual check of {i} on 2026-09-23 is more than 30 days before 2026-11-01"
for i in ("unicode", "willison-dual-llm", "rfc-3986", "llm-security-commons")
] + ["behind: owasp-llm-top10", "watch: owasp-rag-cheat-sheet"]
# --- each clause of the rule goes red on its own ------------------------------
def _clean() -> tuple[list[dict], dict, datetime.date]:
return _json(SOURCES), _json(LAST_RUN), datetime.date(2026, 11, 1)
def test_manual_check_31_days_old_stops_and_30_does_not():
sources, run, _ = _clean()
checked = next(e for e in sources if e["id"] == "unicode")["probe"]["known_at"]
day = datetime.date.fromisoformat(checked)
ran = datetime.datetime.fromisoformat(run["lastRun"].replace("Z", "+00:00")).date()
assert ran >= day # the run is not the older of the two
assert problems(sources, run, day + datetime.timedelta(days=30)) == []
assert problems(sources, run, day + datetime.timedelta(days=31)) == [
f"manual check of {e['id']} on {e['probe']['known_at']} is more than 30 days before "
f"{day + datetime.timedelta(days=31)}"
for e in sources
if e["id"] in MANUAL_IDS
]
def test_old_watch_run_stops():
sources, run, released = _clean()
run["lastRun"] = (released - datetime.timedelta(days=31)).isoformat() + "T09:15:00.000Z"
assert problems(sources, run, released) == [
f"the watch last ran on {released - datetime.timedelta(days=31)}, more than 30 days before {released}"
]
def test_watch_row_stops_though_the_watch_does_not_count_it_behind():
sources, run, released = _clean()
run["current"].remove("cve-2025-32711")
run["watch"].append({"id": "cve-2025-32711", "signal": "watch-only", "observed": "2026-12-01T00:00:00.000Z"})
assert problems(sources, run, released) == ["watch: cve-2025-32711"]
def test_not_verified_stops():
sources, run, released = _clean()
run["current"].remove("mitre-atlas")
run["notVerified"].append({"id": "mitre-atlas", "probe_status": "unreachable", "reason": "deadline reached"})
assert problems(sources, run, released) == ["notVerified: mitre-atlas"]
def test_output_of_another_list_stops():
sources, run, released = _clean()
run["current"].remove("rfc-3986")
assert problems(sources, run, released) == ["the watch output is not a run of this source list"]
def test_release_date_belongs_to_the_version_being_released():
# The rule must read the heading of the version pyproject declares, or a
# release commit that forgets its CHANGELOG heading would be judged on the
# previous release's date.
m = _HEADING.search(CHANGELOG.read_text(encoding="utf-8"))
version = re.search(r'^version = "([^"]+)"', (ROOT / "pyproject.toml").read_text(encoding="utf-8"), re.M)
assert m and version and m.group(1) == version.group(1)