docs(catalogs): wire ASI06 and ATLAS v2026.10 into the README, date every source
Row 1 of the 2026 done-criterion: 16 of 16 register entries on the newest edition, each with a check date and a source link, re-read at the publisher on 2026-10-31. - README: new subsection mapping OWASP ASI06 and seven MITRE ATLAS v2026.10 techniques to what the guard stops at the write and what it does not; a dated catalog table; the threat-model anchors are linked. - ATLAS moved to v2026.10 (2026-10-28) after session 1; AML.T0057 no longer exists there and is not linked. - Register 1.1.0: `cited` beside `latest` on every entry; CleanBase added (cited in BRIEF, unregistered); unverifiable sources kept as candidates. - tests/test_catalog_register.py fails on a missing date or link, an older cited edition, or a README ATLAS/ASI id the register has not verified. - ZWJ measurement re-taken on Unicode 18.0 (1614 sequences, 122/122); the "458 unassigned" figure came from Python's Unicode 16.0, 406 in 18.0.
This commit is contained in:
parent
3eb8e9e9a9
commit
8f08b1906e
SSH key fingerprint:
SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
8 changed files with 314 additions and 43 deletions
29
CHANGELOG.md
29
CHANGELOG.md
|
|
@ -5,6 +5,35 @@ All notable changes to this project will be documented in this file.
|
|||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Documentation — every cited catalog on its newest edition, dated and linked
|
||||
|
||||
No code behaviour changed; one code comment was corrected.
|
||||
|
||||
- **README maps the persisted store to OWASP ASI06 and MITRE ATLAS v2026.10.** A
|
||||
new subsection pairs ASI06 (Memory & Context Poisoning, OWASP Top 10 for
|
||||
Agentic Applications for 2026) and seven ATLAS techniques with what the guard
|
||||
stops at the write **and what it does not**. Each technique was read in the
|
||||
publisher's v2026.10 data file. ATLAS released v2026.10 on 2026-10-28, after the
|
||||
register had recorded v2026.09. `AML.T0057` (LLM Data Leakage) is gone from
|
||||
v2026.10, so it is not linked.
|
||||
- **Catalog register 1.1.0** (`docs/threat-2026/catalogs.json`): every entry now
|
||||
records the edition the repo cites next to the newest one, re-checked at the
|
||||
publisher on 2026-10-31. CleanBase (arXiv 2605.00460) was cited but unregistered,
|
||||
so it is added: 15 → 16 entries. Sources that cannot be read at their publisher
|
||||
are listed as `candidates` and not counted. A new test,
|
||||
`tests/test_catalog_register.py`, fails when an entry lacks a check date or a
|
||||
link, or cites an older edition than the latest. It also fails when the README
|
||||
names an ATLAS or Agentic entry the register has not verified. It never fails on
|
||||
age.
|
||||
- **Unicode 17.0 → 18.0 in the ZWJ measurement.** Re-measured on 18.0: 1614 RGI
|
||||
sequences, 122/122 neighbouring codepoints covered. The `sanitize.py` comment's
|
||||
"458 unassigned" came from Python's Unicode 16.0 tables; the correct figure for
|
||||
18.0 is 406.
|
||||
- **BRIEF** no longer dates "AI Agent Traps" to 2025: the publisher (SSRN) could
|
||||
not be read, so the year is not stated.
|
||||
|
||||
## [1.5.0] — 2026-10-31
|
||||
|
||||
### Changed — every OWASP number carries the 2026 edition (**filters on the old value lose hits**)
|
||||
|
|
|
|||
46
README.md
46
README.md
|
|
@ -231,6 +231,42 @@ fails the test), and that every lexicon pattern has a case — so the matrix can
|
|||
fall behind the code. The four classes it deliberately does **not** stop are called
|
||||
out in [Known limitations](#known-limitations).
|
||||
|
||||
### The persisted store: OWASP ASI06 and MITRE ATLAS
|
||||
|
||||
The store this guard writes to is what the
|
||||
[OWASP Top 10 for Agentic Applications for 2026](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/)
|
||||
calls **ASI06 · Memory & Context Poisoning**: persistent corruption of agent memory,
|
||||
RAG stores or contextual knowledge that keeps acting long after the write. The rows
|
||||
below map it, and the [MITRE ATLAS v2026.10](https://github.com/mitre-atlas/atlas-data/releases/tag/v2026.10)
|
||||
techniques that reach a write-time pipeline, onto what the guard does. The right-hand
|
||||
column is the part it does **not** stop. Each one was measured or is a documented gap.
|
||||
|
||||
| Catalog entry | What the guard stops at the write | What it does not stop |
|
||||
|---|---|---|
|
||||
| **ASI06 · Memory & Context Poisoning** | Instruction-carrying and hidden-carrier content, before it is persisted (scan-before-persist); a trust tier from `Origin` × `Channel`; one provenance line per concept in `log.md` | A false claim in clean prose — semantic poisoning, the highest-impact gap |
|
||||
| **AML.T0070 · RAG Poisoning** | Prompt injections and hidden carriers in a document headed for the index | The "false or misleading information" half — same gap |
|
||||
| **AML.T0080.000 · AI Agent Context Poisoning: Memory** | A memory entry that carries instructions the lexicon or a carrier check sees | An entry that only steers in plain prose |
|
||||
| **AML.T0143 · LLM Prompt Propagation** | Instructions that the lexicon or a carrier check sees are caught at the output gate, so they never reach the store that would pass them on | Propagation over a channel the guard does not sit on |
|
||||
| **AML.T0051.001 · LLM Prompt Injection: Indirect** | The **LLM01:2026** row above | Novel phrasings with no lexicon hit; the tool-less transform carries that case |
|
||||
| **AML.T0068 · LLM Prompt Obfuscation** | Invisible characters, HTML comments, zero font size, base64 and rot13 (decoded, then re-scanned) | Text hidden with a tiny font or the background colour is not flagged by its styling; plain-prose instructions hidden that way give 0 findings |
|
||||
| **AML.T0077 · LLM Response Rendering** | The **LLM10:2026** row above: markdown-image auto-fetch, links, raw active HTML | Pure beaconing to a bare path on a hostile host (a documented gap) |
|
||||
| **AML.T0066 · Retrieval Content Crafting** | Only the crafted content that also carries an injection | The crafted content itself — semantic gap |
|
||||
|
||||
**Every catalog the guard cites is on its newest edition, with the date it was
|
||||
last checked and a link.** The full list is the register in
|
||||
[`docs/threat-2026/catalogs.json`](docs/threat-2026/catalogs.json), and
|
||||
[`tests/test_catalog_register.py`](tests/test_catalog_register.py) fails on an
|
||||
entry without a date or a link. The main ones, checked **2026-10-31**:
|
||||
|
||||
| Catalog | Edition | Published |
|
||||
|---|---|---|
|
||||
| [OWASP Top 10 for LLM Applications](https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/) | 2026 | 2026-08-04 |
|
||||
| [OWASP Top 10 for Agentic Applications](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/) | for 2026 | 2025-12-09 |
|
||||
| [MITRE ATLAS](https://github.com/mitre-atlas/atlas-data/releases/tag/v2026.10) | v2026.10 | 2026-10-28 |
|
||||
| [OWASP RAG Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/RAG_Security_Cheat_Sheet.html) | living, last changed 2026-10-04 | 2026-05-06 |
|
||||
| [The Unicode Standard](https://www.unicode.org/versions/Unicode18.0.0/) | 18.0.0 | 2026-09-16 |
|
||||
| [CVE-2025-32711](https://nvd.nist.gov/vuln/detail/CVE-2025-32711) (EchoLeak) | NVD record, modified 2026-06-17 | 2025-06-11 |
|
||||
|
||||
## The reusable contract (adopt-this checklist)
|
||||
|
||||
The actual product is this checklist, encoded as code you wire in order:
|
||||
|
|
@ -315,8 +351,14 @@ steganography; query-time / runtime guardrails; semantic factuality verification
|
|||
The contract is extracted from a working reference implementation (the
|
||||
`claude-code-llm-wiki` Stage B enrichment pipeline). Threat-model anchors: OWASP
|
||||
Top 10 for LLM Applications 2026 (LLM01:2026, LLM02:2026, LLM03:2026, LLM05:2026,
|
||||
LLM10:2026 strongest; LLM06:2026 self-safety; LLM09:2026 boundary), PoisonedRAG,
|
||||
guardrail-evasion (arXiv 2504.11168), EchoLeak (CVE-2025-32711).
|
||||
LLM10:2026 strongest; LLM06:2026 self-safety; LLM09:2026 boundary); ASI06 from the
|
||||
OWASP Top 10 for Agentic Applications for 2026; MITRE ATLAS v2026.10 (the
|
||||
[mapping above](#the-persisted-store-owasp-asi06-and-mitre-atlas)); the
|
||||
[OWASP RAG Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/RAG_Security_Cheat_Sheet.html)
|
||||
(checked 2026-10-31); [PoisonedRAG](https://arxiv.org/abs/2402.07867);
|
||||
guardrail evasion ([arXiv 2504.11168](https://arxiv.org/abs/2504.11168)); EchoLeak
|
||||
([CVE-2025-32711](https://nvd.nist.gov/vuln/detail/CVE-2025-32711)). Edition, check
|
||||
date and link for every source: [the register](docs/threat-2026/catalogs.json).
|
||||
|
||||
## License
|
||||
|
||||
|
|
|
|||
|
|
@ -208,8 +208,13 @@ The actual product is this checklist, encoded as code a pipeline wires in order:
|
|||
|
||||
- OWASP LLM Top 10: **LLM01:2026 Prompt Injection**, output-handling, and the RAG /
|
||||
knowledge-poisoning vectors.
|
||||
- Google DeepMind "AI Agent Traps" (Franklin et al., 2025): Latent Memory
|
||||
Poisoning and RAG Knowledge Poisoning map directly onto the write-time artifact.
|
||||
- Google DeepMind "AI Agent Traps" (Franklin et al., SSRN abstract 6372438): Latent
|
||||
Memory Poisoning and RAG Knowledge Poisoning map directly onto the write-time
|
||||
artifact. *Not verified at the publisher:* SSRN refuses non-browser requests, so
|
||||
the year is not stated (see `candidates` in `docs/threat-2026/catalogs.json`).
|
||||
- OWASP Top 10 for Agentic Applications for 2026: **ASI06 Memory & Context
|
||||
Poisoning** names the persisted store this guard writes to; the README maps it
|
||||
and the matching MITRE ATLAS v2026.10 techniques.
|
||||
- Research anchors: CleanBase (malicious docs in RAG DBs), DataFilter, and the
|
||||
2025 guardrail-evasion literature that motivates the defense-in-depth framing.
|
||||
|
||||
|
|
|
|||
|
|
@ -778,7 +778,7 @@ fails the test, forcing this doc to be updated:
|
|||
emoji, on ordinary first-party content. The joiner is now judged by context
|
||||
instead: exempt only when **both** neighbours are emoji-context codepoints,
|
||||
measured to cover 122/122 of the codepoints adjacent to a ZWJ across Unicode
|
||||
17.0's 1614 RGI sequences. A joiner placed *between two emoji* is therefore
|
||||
18.0's 1614 RGI sequences (re-measured 2026-10-31; 17.0 gave the same). A joiner placed *between two emoji* is therefore
|
||||
invisible to the carrier check and could carry a covert channel — it costs an
|
||||
emoji per bit and cannot split a word, which is the shape the word-splitting
|
||||
attack needs, so the narrowing is deliberate rather than complete. *(Until
|
||||
|
|
|
|||
11
docs/PLAN.md
11
docs/PLAN.md
|
|
@ -238,6 +238,17 @@ release.**
|
|||
alt som ikke er verifisert, utelates.
|
||||
- En test feiler når registeret mangler dato eller lenke. Den feiler ikke på alder:
|
||||
automatisk varsling er utenfor retningen.
|
||||
- *Gjort 31.10:* **rad 1: 16 av 16.** Alle 15 er sjekket på nytt hos utgiveren.
|
||||
Bare ATLAS hadde flyttet seg: v2026.10 kom 28.10 kl. 22:55 UTC, etter økt 1.
|
||||
`AML.T0057` finnes ikke lenger, så den er ikke koblet inn. CleanBase sto i BRIEF
|
||||
uten å være i registeret, og er lagt til (15 → 16). README kobler ASI06 og sju
|
||||
ATLAS-teknikker, hver med det vakta *ikke* stopper. Målt: tekst gjemt med hvit
|
||||
eller 1px skrift gir 0 funn, mens null skriftstørrelse, base64 og rot13 fanges.
|
||||
Unicode-målingen for ZWJ er tatt på nytt på 18.0 (1614 sekvenser, 122/122).
|
||||
«458» kom fra Pythons Unicode 16.0, og riktig tall for 18.0 er 406.
|
||||
`tests/test_catalog_register.py` holder rad 1. Utenfor tellingen
|
||||
(`candidates`): «AI Agent Traps» (SSRN svarer 403, ikke verifisert) og OKF-
|
||||
spesifikasjonen (operatørens beslutning).
|
||||
|
||||
**Økt 5…N: angrepssakene (rad 3).** Lista fra økt 1 tas i bolker. Hver bolk blir
|
||||
enten deteksjon (TDD, vaktregelen holder) eller en dokumentert rest i LIMITATIONS og
|
||||
|
|
|
|||
|
|
@ -1,10 +1,11 @@
|
|||
{
|
||||
"id": "threat-catalog-register",
|
||||
"version": "1.0.0",
|
||||
"description": "Every threat catalog, standard and source the guard cites: the edition cited at v1.4.1, the newest edition, when it was checked and where. Row 1 of the 2026 done-criterion (docs/PLAN.md § T.2) is N of N entries with cited == latest, a check date and a source link.",
|
||||
"denominator_rule": "Frozen 2026-10-28 in session 1. Entries may be added; none is removed without the operator's yes.",
|
||||
"verification_rule": "Each latest edition was read from the PUBLISHER (its own site, its official repository or API). A copy verified elsewhere is a premise, not a measurement.",
|
||||
"checked": "2026-10-28",
|
||||
"version": "1.1.0",
|
||||
"description": "Every threat catalog, standard and source the guard cites: the edition cited at v1.4.1, the edition cited now, the newest edition, when it was checked and where. Row 1 of the 2026 done-criterion (docs/PLAN.md § T.2) is N of N entries with cited == latest, a check date and a source link; tests/test_catalog_register.py holds it there.",
|
||||
"denominator_rule": "Frozen 2026-10-28 in session 1 at 15. Entries may be added; none is removed without the operator's yes. Session 4 (2026-10-31) added arxiv-2605.00460, cited in docs/BRIEF.md but missing here.",
|
||||
"verification_rule": "Each latest edition was read from the PUBLISHER (its own site, its official repository or API). A copy verified elsewhere is a premise, not a measurement. A source that cannot be read at its publisher goes under candidates, not entries.",
|
||||
"citation_rule": "An unversioned link (an arXiv abs page, a CVE record, a living document) cites whatever the publisher serves as newest; `cited` records the edition it served on the check date.",
|
||||
"checked": "2026-10-31",
|
||||
"entries": [
|
||||
{
|
||||
"id": "owasp-llm-top10",
|
||||
|
|
@ -13,8 +14,9 @@
|
|||
"publisher": "OWASP GenAI Security Project",
|
||||
"cited_at_v1_4_1": "2025 (numbers written without a year)",
|
||||
"latest": "2026",
|
||||
"cited": "2026",
|
||||
"latest_published": "2026-08-04",
|
||||
"checked": "2026-10-28",
|
||||
"checked": "2026-10-31",
|
||||
"source": "https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/",
|
||||
"source_data": "https://github.com/GenAI-Security-Project/GenAI-LLM-Top10/tree/main/2026/final",
|
||||
"evidence": "Project README: 'Current release: OWASP GenAI LLM Top 10 2026 — published August 4, 2026', 2025 marked 'archived release'. The page's datePublished is 2026-08-04T05:50:36Z; its visible date reads August 3, 2026. Headings in 2026/final/*.md: '## LLM01:2026 Prompt Injection' … '## LLM10:2026 Improper Output Handling'. The landing page genai.owasp.org/llm-top-10/ still shows the 2025 list.",
|
||||
|
|
@ -41,15 +43,16 @@
|
|||
"publisher": "OWASP GenAI Security Project",
|
||||
"cited_at_v1_4_1": "not cited",
|
||||
"latest": "for 2026",
|
||||
"cited": "for 2026",
|
||||
"latest_published": "2025-12-09",
|
||||
"checked": "2026-10-28",
|
||||
"checked": "2026-10-31",
|
||||
"source": "https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/",
|
||||
"evidence": "Visible date 'December 9, 2025' (datePublished 2025-12-10T07:55:54Z). IDs and titles from the publisher's 2026/final/mappings/asi-2026.json; no year-suffixed form such as 'ASI06:2026' was seen.",
|
||||
"evidence": "Visible date 'December 9, 2025' (datePublished 2025-12-10T07:55:54Z), unchanged on 2026-10-31. IDs and titles from the publisher's 2026/final/mappings/asi-2026.json; no year-suffixed form such as 'ASI06:2026' was seen. The PDF behind the page's download link answers 'No Access' to an anonymous request (2026-10-31), so the ASI06 scope the README relies on is read from two publisher sources instead: the announcement page (genai.owasp.org/2025/12/09/owasp-top-10-for-agentic-applications-the-benchmark-for-agentic-security-in-the-age-of-autonomous-ai/: 'Memory poisoning reshaped behaviour long after the initial interaction (ASI06 – Memory & Context Poisoning ...)') and the crosswalk repository (GenAI-Security-Project/crosswalk, agentic-top10/Agentic_MITREATLAS.md at 6d7f9a7a: 'Persistent corruption of agent memory, RAG stores, embeddings, or contextual knowledge'). The crosswalk marks its ATLAS rows as unreviewed drafts; no ATLAS link here is taken from it.",
|
||||
"relevant_entries": {
|
||||
"ASI06": "Memory & Context Poisoning",
|
||||
"ASI01": "Agent Goal Hijack"
|
||||
},
|
||||
"used_in": "not yet (session 4 links ASI06)"
|
||||
"used_in": "README § What it protects against (ASI06)"
|
||||
},
|
||||
{
|
||||
"id": "mitre-atlas",
|
||||
|
|
@ -57,23 +60,24 @@
|
|||
"name": "MITRE ATLAS",
|
||||
"publisher": "MITRE",
|
||||
"cited_at_v1_4_1": "not cited",
|
||||
"latest": "v2026.09",
|
||||
"latest_published": "2026-09-15",
|
||||
"checked": "2026-10-28",
|
||||
"source": "https://github.com/mitre-atlas/atlas-data/releases/tag/v2026.09",
|
||||
"evidence": "GitHub releases/latest: v2026.09, 2026-09-15T15:03:27Z; asset ATLAS-2026.09.yaml with version '2026.09', format-version 6.0.0; '1 matrix, 16 tactics, 120 techniques, 88 sub-techniques'. Versioning moved from 5.x (last v5.6.0) to calendar versions at v2026.05; dist/ATLAS.yaml is marked deprecated.",
|
||||
"latest": "v2026.10",
|
||||
"cited": "v2026.10",
|
||||
"latest_published": "2026-10-28",
|
||||
"checked": "2026-10-31",
|
||||
"source": "https://github.com/mitre-atlas/atlas-data/releases/tag/v2026.10",
|
||||
"evidence": "GitHub releases/latest on 2026-10-31: v2026.10, 2026-10-28T22:55:06Z, i.e. after session 1 read v2026.09. Release notes: '1 matrix, 16 tactics, 129 techniques, 92 sub-techniques'. Asset ATLAS-2026.10.yaml sha256 ac41ee9f99a47084ba5fdfc77ae83a28687e12390240fffeb37df7914ffe142d, collection version '2026.10', format-version 6.1.0, 221 technique records (129 + 92). Earlier: v2026.09 (2026-09-15); versioning moved from 5.x (last v5.6.0) to calendar versions at v2026.05.",
|
||||
"relevant_entries": {
|
||||
"AML.T0051.001": "LLM Prompt Injection: Indirect",
|
||||
"AML.T0068": "LLM Prompt Obfuscation",
|
||||
"AML.T0070": "RAG Poisoning",
|
||||
"AML.T0071": "False RAG Entry Injection",
|
||||
"AML.T0066": "Retrieval Content Crafting",
|
||||
"AML.T0057": "LLM Data Leakage",
|
||||
"AML.T0077": "LLM Response Rendering",
|
||||
"AML.T0080.000": "AI Agent Context Poisoning: Memory"
|
||||
"AML.T0080.000": "AI Agent Context Poisoning: Memory",
|
||||
"AML.T0143": "LLM Prompt Propagation"
|
||||
},
|
||||
"relevant_entries_note": "Read from the v2026.09 data file only; not cross-checked on atlas.mitre.org. Session 4 re-reads each before linking it.",
|
||||
"used_in": "not yet (session 4)"
|
||||
"relevant_entries_note": "Each ID, name and description read in ATLAS-2026.10.yaml on 2026-10-31. AML.T0057 'LLM Data Leakage' (listed here from v2026.09) is absent from v2026.10: the release notes move its conversation-context part to AML.T0085.002, so it was dropped rather than linked. AML.T0143 is new in v2026.10. atlas.mitre.org/techniques/<id> answers HTTP 404 to a non-browser request (the site renders client-side), so the README links the release, not the technique pages. AML.T0071 is read but not linked: whether the guard stops a fake RAG entry was not measured.",
|
||||
"used_in": "README § What it protects against"
|
||||
},
|
||||
{
|
||||
"id": "owasp-rag-cheat-sheet",
|
||||
|
|
@ -82,11 +86,12 @@
|
|||
"publisher": "OWASP Cheat Sheet Series",
|
||||
"cited_at_v1_4_1": "by name only, no date or link (docs/PLAN.md v1 positioning)",
|
||||
"latest": "living document, last modified 2026-10-04 (commit 510ea484)",
|
||||
"cited": "living document, last modified 2026-10-04 (commit 510ea484)",
|
||||
"latest_published": "2026-10-04",
|
||||
"checked": "2026-10-28",
|
||||
"checked": "2026-10-31",
|
||||
"source": "https://cheatsheetseries.owasp.org/cheatsheets/RAG_Security_Cheat_Sheet.html",
|
||||
"evidence": "OWASP/CheatSheetSeries history: created 2026-05-06 (41f67299, 'New Cheat Sheet: RAG Security (#2131)'), last modified 2026-10-04T18:12:29Z (510ea484). The repo had cited it under a shorter name.",
|
||||
"used_in": "positioning text"
|
||||
"used_in": "README § Design & threat model (linked, with the check date)"
|
||||
},
|
||||
{
|
||||
"id": "unicode",
|
||||
|
|
@ -95,14 +100,17 @@
|
|||
"publisher": "Unicode Consortium",
|
||||
"cited_at_v1_4_1": "17.0 (sanitize.py block comment, LIMITATIONS ZWJ measurement)",
|
||||
"latest": "18.0.0",
|
||||
"cited": "18.0.0",
|
||||
"latest_published": "2026-09-16",
|
||||
"checked": "2026-10-28",
|
||||
"checked": "2026-10-31",
|
||||
"source": "https://www.unicode.org/versions/Unicode18.0.0/",
|
||||
"evidence": "unicode.org/history/publicationdates.html: 'Version 18.0 | 2026 | September 16'. /Public/UCD/latest/ReadMe.txt: 'final data files and charts for version 18.0.0'.",
|
||||
"pinned_files": {
|
||||
"https://www.unicode.org/Public/18.0.0/ucd/DerivedCoreProperties.txt": "sha256:09c928886a178fcafd93c29e4bd59073a058e5a100b716d425cb563ab50f68c9",
|
||||
"https://www.unicode.org/Public/18.0.0/ucd/UnicodeData.txt": "sha256:0736451de439ae7baf1425136617da495e09ee5afbe6e394374db7009ea08950"
|
||||
"https://www.unicode.org/Public/18.0.0/ucd/UnicodeData.txt": "sha256:0736451de439ae7baf1425136617da495e09ee5afbe6e394374db7009ea08950",
|
||||
"https://www.unicode.org/Public/18.0.0/emoji/emoji-zwj-sequences.txt": "sha256:f61b5213bdf85a57741a9f903cb38d0f39a3d13be6ff4aca3b39857e859a9191"
|
||||
},
|
||||
"remeasured_2026_10_31": "The ZWJ emoji-context claim in sanitize.py and LIMITATIONS still cited 17.0. Re-measured on 18.0: 1614 RGI ZWJ sequences, 122 distinct codepoints next to a ZWJ, 122/122 inside the emoji-context ranges (17.0 gives the same counts, the positive control). The '458 unassigned' figure beside it came from Python's own unicodedata (Unicode 16.0); UnicodeData.txt gives 433 in 17.0 and 406 in 18.0.",
|
||||
"used_in": "invisible-character tables (sanitize, output); the character denominator in docs/threat-2026/codepoints.json"
|
||||
},
|
||||
{
|
||||
|
|
@ -112,8 +120,9 @@
|
|||
"publisher": "MITRE CVE / NVD; Microsoft MSRC",
|
||||
"cited_at_v1_4_1": "CVE-2025-32711",
|
||||
"latest": "MSRC revision 2026-02-20; NVD last modified 2026-06-17",
|
||||
"cited": "MSRC revision 2026-02-20; NVD last modified 2026-06-17",
|
||||
"latest_published": "2025-06-11",
|
||||
"checked": "2026-10-28",
|
||||
"checked": "2026-10-31",
|
||||
"source": "https://nvd.nist.gov/vuln/detail/CVE-2025-32711",
|
||||
"evidence": "NVD: 'Ai command injection in M365 Copilot allows an unauthorized attacker to disclose information over a network.' CWE-74. MSRC title 'M365 Copilot Information Disclosure Vulnerability', Critical. NVD's references include the researcher's EchoLeak page; neither record says 'EchoLeak' in prose.",
|
||||
"used_in": "active-content detector docs, coverage matrix, README"
|
||||
|
|
@ -125,10 +134,11 @@
|
|||
"publisher": "arXiv",
|
||||
"cited_at_v1_4_1": "2402.07867 (no version)",
|
||||
"latest": "v3",
|
||||
"cited": "v3",
|
||||
"latest_published": "2024-08-13",
|
||||
"checked": "2026-10-28",
|
||||
"checked": "2026-10-31",
|
||||
"source": "https://arxiv.org/abs/2402.07867",
|
||||
"used_in": "claims we will not make; semantic-poisoning gap"
|
||||
"used_in": "README § Design & threat model (linked); semantic-poisoning gap"
|
||||
},
|
||||
{
|
||||
"id": "arxiv-2504.11168",
|
||||
|
|
@ -137,8 +147,9 @@
|
|||
"publisher": "arXiv",
|
||||
"cited_at_v1_4_1": "2504.11168 (no version)",
|
||||
"latest": "v3",
|
||||
"cited": "v3",
|
||||
"latest_published": "2025-07-14",
|
||||
"checked": "2026-10-28",
|
||||
"checked": "2026-10-31",
|
||||
"source": "https://arxiv.org/abs/2504.11168",
|
||||
"used_in": "README and BRIEF, detection-is-evadable claim"
|
||||
},
|
||||
|
|
@ -149,8 +160,9 @@
|
|||
"publisher": "arXiv",
|
||||
"cited_at_v1_4_1": "2509.14285 (no version)",
|
||||
"latest": "v4",
|
||||
"cited": "v4",
|
||||
"latest_published": "2025-12-17",
|
||||
"checked": "2026-10-28",
|
||||
"checked": "2026-10-31",
|
||||
"source": "https://arxiv.org/abs/2509.14285",
|
||||
"used_in": "docs/BRIEF.md"
|
||||
},
|
||||
|
|
@ -161,8 +173,9 @@
|
|||
"publisher": "arXiv",
|
||||
"cited_at_v1_4_1": "2505.03574 (no version)",
|
||||
"latest": "v1",
|
||||
"cited": "v1",
|
||||
"latest_published": "2025-05-06",
|
||||
"checked": "2026-10-28",
|
||||
"checked": "2026-10-31",
|
||||
"source": "https://arxiv.org/abs/2505.03574",
|
||||
"used_in": "docs/BRIEF.md"
|
||||
},
|
||||
|
|
@ -173,8 +186,9 @@
|
|||
"publisher": "arXiv",
|
||||
"cited_at_v1_4_1": "2503.18813 (no version)",
|
||||
"latest": "v2",
|
||||
"cited": "v2",
|
||||
"latest_published": "2025-06-24",
|
||||
"checked": "2026-10-28",
|
||||
"checked": "2026-10-31",
|
||||
"source": "https://arxiv.org/abs/2503.18813",
|
||||
"used_in": "architecture lineage"
|
||||
},
|
||||
|
|
@ -185,12 +199,27 @@
|
|||
"publisher": "arXiv",
|
||||
"cited_at_v1_4_1": "2604.00387 (no version), labelled 'RAGShield'",
|
||||
"latest": "v2",
|
||||
"cited": "v2",
|
||||
"latest_published": "2026-04-04",
|
||||
"checked": "2026-10-28",
|
||||
"checked": "2026-10-31",
|
||||
"source": "https://arxiv.org/abs/2604.00387",
|
||||
"note": "Narrower than the label suggests: numerical claim manipulation in government RAG systems.",
|
||||
"used_in": "docs/PLAN.md v1 anchors (history)"
|
||||
},
|
||||
{
|
||||
"id": "arxiv-2605.00460",
|
||||
"kind": "paper",
|
||||
"name": "CleanBase: Detecting Malicious Documents in RAG Knowledge Databases",
|
||||
"publisher": "arXiv",
|
||||
"cited_at_v1_4_1": "2605.00460 (no version, pdf link)",
|
||||
"latest": "v1",
|
||||
"cited": "v1",
|
||||
"latest_published": "2026-05-01",
|
||||
"checked": "2026-10-31",
|
||||
"source": "https://arxiv.org/abs/2605.00460",
|
||||
"added": "2026-10-31, session 4: cited in docs/BRIEF.md § 11 but missing from the register frozen in session 1",
|
||||
"used_in": "docs/BRIEF.md § 11, write-time RAG-DB poisoning is under-tooled"
|
||||
},
|
||||
{
|
||||
"id": "willison-dual-llm",
|
||||
"kind": "article",
|
||||
|
|
@ -198,8 +227,9 @@
|
|||
"publisher": "Simon Willison (author's own blog)",
|
||||
"cited_at_v1_4_1": "by name, 2023",
|
||||
"latest": "2023-04-25 (single post)",
|
||||
"cited": "2023-04-25 (single post)",
|
||||
"latest_published": "2023-04-25",
|
||||
"checked": "2026-10-28",
|
||||
"checked": "2026-10-31",
|
||||
"source": "https://simonwillison.net/2023/Apr/25/dual-llm-pattern/",
|
||||
"used_in": "architecture lineage"
|
||||
},
|
||||
|
|
@ -210,8 +240,9 @@
|
|||
"publisher": "RFC Editor",
|
||||
"cited_at_v1_4_1": "RFC 3986",
|
||||
"latest": "RFC 3986, INTERNET STANDARD, not obsoleted (updated by RFC 7320, RFC 8820)",
|
||||
"cited": "RFC 3986, INTERNET STANDARD, not obsoleted (updated by RFC 7320, RFC 8820)",
|
||||
"latest_published": "2005-01",
|
||||
"checked": "2026-10-28",
|
||||
"checked": "2026-10-31",
|
||||
"source": "https://www.rfc-editor.org/rfc/rfc3986.json",
|
||||
"used_in": "active_content.py scheme grammar"
|
||||
},
|
||||
|
|
@ -222,11 +253,30 @@
|
|||
"publisher": "the commons repository on the same Forgejo instance",
|
||||
"cited_at_v1_4_1": "lexicon 0.8.0 (83 patterns, byte-equal copy)",
|
||||
"latest": "v0.9.0 (lexicon/injection-lexicon.json 0.8.0, codepoints/carriers.json 0.2.0)",
|
||||
"cited": "v0.9.0 (lexicon/injection-lexicon.json 0.8.0, codepoints/carriers.json 0.2.0)",
|
||||
"latest_published": "2026-08-21",
|
||||
"checked": "2026-10-28",
|
||||
"checked": "2026-10-31",
|
||||
"source": "ssh://git@git.fromaitochitta.com/ktg/llm-security-commons.git",
|
||||
"evidence": "Fetched origin 2026-10-28: newest tag v0.9.0, origin/main c362717 (2026-08-21). Its mapping/owasp-map.json carries 2025-edition numbers; that is input for commons, not a change here.",
|
||||
"evidence": "Fetched origin 2026-10-28 and again 2026-10-31: newest tag v0.9.0, origin/main c362717 (2026-08-21). Its mapping/owasp-map.json carries 2025-edition numbers; that is input for commons, not a change here.",
|
||||
"used_in": "src/llm_ingestion_guard/injection_lexicon.json"
|
||||
}
|
||||
],
|
||||
"candidates": [
|
||||
{
|
||||
"id": "ssrn-ai-agent-traps",
|
||||
"name": "AI Agent Traps (Franklin, Tomasev, Jacobs, Leibo, Osindero; Google DeepMind)",
|
||||
"cited_in": "docs/BRIEF.md § 10",
|
||||
"source": "https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6372438",
|
||||
"checked": "2026-10-31",
|
||||
"why_not_counted": "SSRN answers HTTP 403 to non-browser requests, so the edition and date were not read at the publisher. Secondary coverage dates the posting to March 2026, where BRIEF said 2025. Neither is verified, so BRIEF no longer states a year."
|
||||
},
|
||||
{
|
||||
"id": "okf-spec",
|
||||
"name": "Open Knowledge Format specification (okf/SPEC.md)",
|
||||
"cited_in": "README, CLAUDE.md, the okf adapter (SPEC § 4.1, § 5.1, § 10)",
|
||||
"source": "https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md",
|
||||
"checked": "2026-10-31",
|
||||
"why_not_counted": "A format the adapter parses, not a threat catalog; whether it belongs in the row-1 denominator is the operator's decision. Measured for that decision: one SPEC.md commit after the corpus pin 3fcbb9f, 62432a0 (2026-08-20) 'make every timestamp an ISO 8601 datetime with an explicit offset'."
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -133,14 +133,15 @@ def _carrier_class(cp: int) -> str:
|
|||
# not context, so `a<ZWJ>👩` stays a carrier and an attacker cannot buy exemption
|
||||
# with a single emoji.
|
||||
#
|
||||
# The ranges are blocks, not an emoji table. Measured against Unicode 17.0's
|
||||
# `emoji-zwj-sequences.txt`: 1614 RGI sequences use 122 distinct codepoints
|
||||
# The ranges are blocks, not an emoji table. Measured against Unicode 18.0's
|
||||
# `emoji-zwj-sequences.txt` (sha256 f61b5213bdf85a57…, re-measured 2026-10-31;
|
||||
# 17.0 gives the same counts): 1614 RGI sequences use 122 distinct codepoints
|
||||
# adjacent to a ZWJ, and these five ranges cover 122/122. Blocks are the point —
|
||||
# shipping the RGI sequence list itself would be precise the day it landed and
|
||||
# stale at the next Unicode release, reintroducing this exact false positive for
|
||||
# every new emoji until someone bumped the file. Whole blocks include the
|
||||
# unassigned headroom (458 Cn codepoints here) that future emoji are allocated
|
||||
# into, so the table does not age.
|
||||
# unassigned headroom (406 Cn codepoints here in 18.0) that future emoji are
|
||||
# allocated into, so the table does not age.
|
||||
#
|
||||
# Residual, documented in LIMITATIONS: a ZWJ hidden *between two emoji* is
|
||||
# exempt and could carry a covert channel. It costs an emoji per bit and cannot
|
||||
|
|
|
|||
133
tests/test_catalog_register.py
Normal file
133
tests/test_catalog_register.py
Normal file
|
|
@ -0,0 +1,133 @@
|
|||
"""Every catalog the guard cites carries its edition, a check date and a source link — row 1 of the 2026 done-criterion.
|
||||
|
||||
Row 1 (`docs/PLAN.md` § T.2) is N of N register entries on the newest edition, each
|
||||
with the date it was last checked and a link to where it was checked. The register
|
||||
is `docs/threat-2026/catalogs.json`; its denominator was frozen at 15 entries in
|
||||
session 1 and may grow, never shrink, without the operator's yes.
|
||||
|
||||
This file fails when an entry lacks a date or a link, or cites an edition other
|
||||
than the one recorded as latest. It deliberately does **not** fail on age: an
|
||||
automatic "this check is stale" alarm is outside the approved direction, so a date
|
||||
from last year passes as long as it is a date.
|
||||
|
||||
The README may only name an OWASP Agentic or MITRE ATLAS entry the register
|
||||
records as read at the publisher, and must name the ATLAS release the register
|
||||
records as latest — that is how "what is not verified is left out" stays true
|
||||
after the session that wrote it. No network: the publisher reads are recorded in
|
||||
the register's ``evidence`` fields, not repeated here.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
REGISTER = ROOT / "docs" / "threat-2026" / "catalogs.json"
|
||||
README = ROOT / "README.md"
|
||||
|
||||
# The denominator frozen 2026-10-28 (session 1). Entries may be added; none of
|
||||
# these may disappear without the operator's yes.
|
||||
FROZEN_IDS = frozenset({
|
||||
"owasp-llm-top10", "owasp-agentic-top10", "mitre-atlas", "owasp-rag-cheat-sheet",
|
||||
"unicode", "cve-2025-32711", "arxiv-2402.07867", "arxiv-2504.11168",
|
||||
"arxiv-2509.14285", "arxiv-2505.03574", "arxiv-2503.18813", "arxiv-2604.00387",
|
||||
"willison-dual-llm", "rfc-3986", "llm-security-commons",
|
||||
})
|
||||
|
||||
LINK_SCHEMES = frozenset({"https", "ssh"})
|
||||
|
||||
|
||||
def _register() -> dict:
|
||||
return json.loads(REGISTER.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def _entries() -> list[dict]:
|
||||
return _register()["entries"]
|
||||
|
||||
|
||||
def _problems(entry: dict) -> list[str]:
|
||||
"""Why an entry does not count toward row 1; empty when it does."""
|
||||
problems = []
|
||||
for field in ("id", "name", "publisher", "latest", "cited", "checked", "source"):
|
||||
if not isinstance(entry.get(field), str) or not entry[field].strip():
|
||||
problems.append(f"missing {field}")
|
||||
checked = entry.get("checked")
|
||||
if isinstance(checked, str) and checked.strip():
|
||||
try:
|
||||
datetime.date.fromisoformat(checked)
|
||||
except ValueError:
|
||||
problems.append(f"checked is not an ISO date: {checked!r}")
|
||||
source = entry.get("source")
|
||||
if isinstance(source, str) and source.strip():
|
||||
parts = urlsplit(source)
|
||||
if parts.scheme not in LINK_SCHEMES or not parts.netloc:
|
||||
problems.append(f"source is not a link: {source!r}")
|
||||
if entry.get("cited") != entry.get("latest"):
|
||||
problems.append(f"cites {entry.get('cited')!r}, latest is {entry.get('latest')!r}")
|
||||
return problems
|
||||
|
||||
|
||||
def test_the_check_goes_red_on_a_missing_date_a_missing_link_or_an_old_edition():
|
||||
good = {"id": "x", "name": "X", "publisher": "P", "latest": "2", "cited": "2",
|
||||
"checked": "2026-10-31", "source": "https://example.test/x"}
|
||||
assert _problems(good) == []
|
||||
assert "missing checked" in _problems({**good, "checked": ""})
|
||||
assert any("ISO date" in p for p in _problems({**good, "checked": "31.10.2026"}))
|
||||
assert "missing source" in _problems({k: v for k, v in good.items() if k != "source"})
|
||||
assert any("not a link" in p for p in _problems({**good, "source": "arXiv 2402.07867"}))
|
||||
assert any("latest is" in p for p in _problems({**good, "cited": "1"}))
|
||||
|
||||
|
||||
def test_every_entry_has_a_check_date_a_source_link_and_the_latest_edition():
|
||||
failing = {e.get("id", "?"): _problems(e) for e in _entries()}
|
||||
failing = {k: v for k, v in failing.items() if v}
|
||||
assert not failing, "register entries that do not count toward row 1:\n" + "\n".join(
|
||||
f" {k}: {'; '.join(v)}" for k, v in failing.items())
|
||||
|
||||
|
||||
def test_the_frozen_denominator_never_shrinks():
|
||||
ids = [e["id"] for e in _entries()]
|
||||
assert len(ids) == len(set(ids)), "duplicate register ids"
|
||||
missing = FROZEN_IDS - set(ids)
|
||||
assert not missing, f"entries removed without the operator's yes: {sorted(missing)}"
|
||||
|
||||
|
||||
def test_unverified_candidates_are_kept_apart_from_the_count():
|
||||
# A source the repo cites but that could not be read at its publisher is
|
||||
# recorded as a candidate, never as an entry with a borrowed date.
|
||||
entry_ids = {e["id"] for e in _entries()}
|
||||
for candidate in _register().get("candidates", []):
|
||||
assert candidate["id"] not in entry_ids
|
||||
assert candidate.get("why_not_counted", "").strip()
|
||||
|
||||
|
||||
def _entry(entry_id: str) -> dict:
|
||||
(entry,) = [e for e in _entries() if e["id"] == entry_id]
|
||||
return entry
|
||||
|
||||
|
||||
def test_readme_names_only_atlas_techniques_read_in_the_latest_release():
|
||||
atlas = _entry("mitre-atlas")
|
||||
named = set(re.findall(r"AML\.T\d{4}(?:\.\d{3})?", README.read_text(encoding="utf-8")))
|
||||
assert named, "the README links no ATLAS technique; row 1 expects ATLAS to be wired in"
|
||||
unverified = named - set(atlas["relevant_entries"])
|
||||
assert not unverified, f"README names ATLAS techniques the register has not verified: {sorted(unverified)}"
|
||||
|
||||
|
||||
def test_readme_names_the_atlas_release_the_register_calls_latest():
|
||||
text = README.read_text(encoding="utf-8")
|
||||
latest = _entry("mitre-atlas")["latest"]
|
||||
assert f"ATLAS {latest}" in text, f"README does not say 'ATLAS {latest}'"
|
||||
stale = set(re.findall(r"ATLAS (v\d{4}\.\d{2})", text)) - {latest}
|
||||
assert not stale, f"README names an ATLAS release other than the latest: {sorted(stale)}"
|
||||
|
||||
|
||||
def test_readme_names_only_agentic_entries_read_at_the_publisher():
|
||||
agentic = _entry("owasp-agentic-top10")
|
||||
named = set(re.findall(r"\bASI\d{2}\b", README.read_text(encoding="utf-8")))
|
||||
assert "ASI06" in named, "the README does not link ASI06"
|
||||
unverified = named - set(agentic["relevant_entries"])
|
||||
assert not unverified, f"README names Agentic entries the register has not verified: {sorted(unverified)}"
|
||||
Loading…
Add table
Add a link
Reference in a new issue