1
0
Fork 0

Compare commits

..

No commits in common. "main" and "v1.2.0" have entirely different histories.

16 changed files with 108 additions and 1477 deletions

3
.gitignore vendored
View file

@ -24,6 +24,3 @@ coverage/
*.local.md
*.local.sh
.DS_Store
# --- scratchpad: measurement scripts are tracked, the extracted corpus copy is not ---
/scratchpad/corpus/

View file

@ -5,105 +5,6 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [1.4.0] — 2026-09-08
### Added — `tags: [a, b, c]`, the one flow-sequence form SPEC §4.1 writes out
`okf.parse_frontmatter` refused a flow sequence of plain scalars. That is the
form SPEC.md §4.1's own frontmatter skeleton uses for `tags`, and `tags` is
present in 53/53 upstream reference concepts, 9/53 of them in exactly this form.
Admitted now, parsing to the same value its block-sequence sibling already did:
```yaml
tags: [finance, revenue, headline-metric]
```
An element is a plain scalar only if it is non-empty and carries none of
`{ } [ ] : , " ' #`, and it then passes the unchanged scalar-indicator rule.
The refusals that make that a rule rather than a preference: a quoted element
(this parser retains quotes rather than stripping them, so reading one would
hand back a value YAML does not), an element carrying a colon or a comma, a
sequence inside a sequence, an empty element, an anchor or an alias. A sequence
may not **mix** scalars and mappings — the same rule the block list already
carries, for the same reason: a consumer iterating the value and reading
`entry.get("id")` crashes on the first `str`. The `sources` flow-mapping
carrier added in 1.3.0 is unchanged and pinned against regression.
**Measured, with the denominator: this takes the pinned OKF corpus from 0/53 to
6/53** (`_okf-upstream/okf` @ `3fcbb9f`, 53 non-reserved documents; the six are
all in `acme_retail`). It does not close the corpus. **44/53 still stop on
`generated` written as a top-level block mapping** — a form refused on the
no-nesting-past-depth-1 rule, which is a security decision and out of scope
here. Two neighbouring predicates were measured and deliberately not built: a
flush-left block sequence and a folded plain scalar release **0/53** each on
their own and leave the corpus at 6/53 when stacked on this one. Detection
behaviour is not frozen under semver; the exported surface is unchanged.
### Changed
- `docs/LIMITATIONS.md`'s `tags`/`description` entry is rewritten against the
measurement. Three of its claims were wrong: the parser *does* have a
sequence-value type (since 1.3.0 — what it lacks is the indentation the
corpus omits), the figure is 6/53 and not the 1.2.0-era 4/53, and
`tags`/`description` are *not* the residual that blocks the corpus.
- Self-safety: the new predicate is character-level with no regex, so
`docs/redos-sweep.py` cannot see it (it collects compiled patterns). It is
measured instead — linear in both element length (exponent 0.86-0.99) and
element count (0.97-1.05) over four doublings to 800_000, and pinned by two
CPU-clock bounds in `tests/test_okf.py`.
## [1.3.0] — 2026-09-02
### Added — the `sources` provenance family becomes expressible in both spec carriers
`okf.parse_frontmatter` rejected `sources` in every form the OKF spec and its
producers actually use. Measured 2026-09-02 by two consumers independently:
`sources: [{ id: a, resource: x }]` raised on the `[` indicator (with one entry
as with two), and the block sequence of block mappings — SPEC.md §5.1's *own*
example — raised `"nested mappings are not supported"`. `resource` is REQUIRED
within a `sources` entry (§5.1), so the whole provenance family was unwritable,
and a bundle written the way the spec documents it was refused.
Admitted now, both parsing to the same value (a list of dicts):
```yaml
sources: [{ id: a, resource: https://e.com/a }]
sources:
- id: a
resource: https://e.com/a
```
**`resource` is allowlisted inside a `sources` entry and nowhere else.** 1.2.0
left it off the allowlist on the argument that the parser could not tell
`sources[].resource` (§5.1, a citation) from `executor.resource` /
`attester.resource` (§10, a pointer to code to be run — the door-C route closed
in 1.1.0). That premise was false: the owning key is in scope at every call
site and was simply never threaded through. It is threaded now, so the
discrimination is structural rather than a judgement about the value, and
`executor: [{ resource: skills/run.md }]` and `attester:\n - resource: …` are
refused on the allowlist through *every* carrier, including the two this adds.
Refusal stays the default everywhere else. A flow sequence of plain scalars
(`tags: [a, b]`) still raises — the sequence carrier is opened for the flow
mapping element and nothing else. A `sources` entry still admits scalar leaves
only, so SPEC §5.1's optional *per-entry* `usage_window` (a mapping inside a
mapping) is refused: no nesting past depth 1 is a security property, and it was
not spent here. Registered as a conformance gap in `docs/LIMITATIONS.md`. A
block list may not mix scalar items and mappings. Off-allowlist keys, anchors,
aliases, tags, duplicate keys and unclosed collections raise as before, and a
refused mapping still raises rather than degrading into a string.
**New residual, registered:** `sources[].resource` is scanned as text (T1) but
never validated as a URL. T3's https allowlist cannot be extended to it without
over-blocking conformant bundles — §5.1 permits a bundle-relative path or a
scope descriptor a consumer cannot follow at all. A consumer that dereferences
it must call `okf.validate_resource_url` itself.
No exported surface changed and no detector behaviour or calibration changed.
Suite 834 → 868 (34 new rows, plus four pre-existing rows updated where this
release changed the behaviour they pinned); 130/130 classes, 6/6 documented
gaps hold, 45 limitations, ReDoS sweep 0/152 candidates flagged.
## [1.2.0] — 2026-08-23
### Added — OKF frontmatter can express one mapping form: typed and allowlisted

View file

@ -11,40 +11,18 @@ framework-agnostisk kode.
Referanse-implementasjon: `claude-code-llm-wiki` Stage B (`tools/wiki_ingest/`).
Lexikon-seed: `injection-patterns.mjs` fra `llm-security`-pluginen.
Repoet er på **v1.4.0 i koden, UUTGITT** (`pyproject.toml` + `__init__.py` er
bumpet; README-badge, install-pinnen, ADOPTION-BRIEF og BRIEF står med vilje
igjen på `1.3.0`, som er den siste taggen som FINNES — en install-pin må peke på
en ekte tag). Release-commiten (CHANGELOG-overskrift datert, de fire
dokumentflatene bumpet, tag) er ikke tatt. Den eksporterte Python-surfacen er frosset under semver
Repoet er på **v1.2.0** — den eksporterte Python-surfacen er frosset under semver
(deteksjonsatferd er det IKKE; kalibrering flytter seg i 1.x). Stdlib-kjernen er
bygget og testet (15 moduler +
topp-nivå wiring, showcase + korpus), inkl. OKF-adapter og aktivt-innhold-
detektor (EchoLeak-klassen) i output-gaten. OKF-frontmatterens mapping-klasse
har **fire** uttrykkbare bærere (G3 21.08, G30 02.09): flow-mapping som verdi
og som blokkliste-element, flow-sekvens av flow-mappinger, og blokk-sekvens av
blokk-mappinger (SPEC §5.1s egen form). HVER nøkkel i alle fire står på
allowlisten og hvert blad er en ren skalar. Formen er trygg fordi allowlisten
inspiserer hver nøkkel; det blanke avslaget var håndhevelsen, ikke poenget.
**`resource` er allowlistet KUN inne i en `sources`-oppføring** — foreldre-
nøkkelen avgjør, så `executor`/`attester` sin `resource` (§10, dør C) avvises
gjennom hver eneste bærer. 1.2.0s begrunnelse for å utelate den (parseren
manglet foreldre-kontekst) var målt feil: konteksten var der, den var bare
aldri tredd gjennom. Topp-nivå blokk-mapping, dotted- og inline-kolon-rutene
raiser fortsatt, en blokkliste kan ikke blande skalarer og mappinger, og en
avvist mapping raiser — den degraderer aldri til en streng (1.1.0-defekten).
**Flow-sekvens av rene skalarer (`tags: [a, b]`) PARSER fra 1.4.0** (P1,
operatørbeslutning 08.09) — SPEC §4.1s eget skjelett. Et element er en ren
skalar kun hvis det er ikke-tomt og uten `{ } [ ] : , " ' #`, og så gjelder den
uendrete indikator-regelen; sitert element, kolon/komma i elementet, sekvens i
sekvens, tomt element, anker og alias raiser fortsatt, og en flow-sekvens kan
ikke blande skalarer og mappinger. **Målt med nevner: 0/53 → 6/53** på pinnet
OKF-korpus (`3fcbb9f`). Den bindende skranken er IKKE tags/description, men
`generated` som topp-nivå blokk-mapping (44/53) — den bruker opp dybde-1 og er
en sikkerhetsbeslutning. P2 (blokksekvens uten innrykk) og P3 (foldet plain
scalar) er MÅLT til 0/53 hver og bevisst IKKE bygget. `sources[].resource`
URL-valideres ALDRI (T3 ser kun topp-nivå `resource`) — §5.1 tillater
bundle-relative stier og scope-beskrivelser, så en https-gate ville over-blokkert
konforme bundles; konsumenten må selv kalle `validate_resource_url`. Mode-b `import_bundle` skanner
har **én** uttrykkbar form (G3, 21.08): en flow-mapping (`generated: { by: x, at: y }`) — som verdi eller som blokkliste-
element — der HVER nøkkel står på en ni-navns allowlist og hvert blad er en ren
skalar. Formen er trygg fordi allowlisten inspiserer hver nøkkel; det blanke
avslaget var håndhevelsen, ikke poenget. `resource` er bevisst UTE av
allowlisten (peker, ikke etikett — den ene nøkkelen T3 finnes for). Blokk-,
dotted- og inline-kolon-rutene raiser fortsatt, og en avvist mapping raiser —
den degraderer aldri til en streng (1.1.0-defekten). Mode-b `import_bundle` skanner
reserverte strukturfiler (`index.md`/`log.md`) i mottatte bundles i stedet for å
path-avvise dem; upload-front-end beholder shadow-reject (`allow_reserved=False`).
Output-gatens decode-and-rescan mater dekodet base64-klartekst gjennom BÅDE lexicon

View file

@ -2,7 +2,7 @@
Write-time defensive layer for Python pipelines that persist LLM output: sanitize, fence, tool-less quarantined transform, capability isolation, scan before persist, fail-secure.
![Version](https://img.shields.io/badge/version-1.3.0-blue)
![Version](https://img.shields.io/badge/version-1.2.0-blue)
![Status](https://img.shields.io/badge/status-stable-brightgreen)
![Python](https://img.shields.io/badge/python-3.10%2B-purple)
![License](https://img.shields.io/badge/license-MIT-lightgrey)
@ -33,7 +33,7 @@ at write time, never assumed from the format. Any pipeline ingesting external da
into an agent-read store has this shape; an OKF wiki is its canonical form — which
is why the guard ships a first-class OKF adapter (below).
**Status:** `v1.3.0`. The stdlib-only core — its detector, contract, and
**Status:** `v1.2.0`. The stdlib-only core — its detector, contract, and
OKF-adapter modules plus the top-level wiring — is built and tested, exercised by
an end-to-end showcase and adversarial + false-positive corpora. The exported
Python surface is now frozen under semver: nothing exported is removed, renamed or
@ -58,7 +58,7 @@ are real limitations, stated plainly below; read them.
Not on PyPI. The guard is distributed from its Forgejo origin — pin a release tag:
```bash
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v1.3.0"
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v1.2.0"
```
The `open/` mirror is anonymously readable, so CI needs no deploy key, token, or
@ -171,11 +171,7 @@ mapping — `generated: { by: x, at: y }`, `verified: { … }` bare or listed,
allowlist (`by`, `at`, `from`, `to`, `id`, `title`, `author`, `usage_count`,
`last_modified`) with plain-scalar leaves only. A key off that list, a nested
collection or a duplicate key is refused, and `resource` is deliberately not on
it; the block, dotted and inline-colon routes to a mapping still raise. A
*sequence* value has two carriers — the block list, and (as of `1.4.0`) the flow
sequence `tags: [a, b, c]`, which is SPEC §4.1's own skeleton — whose elements
are either all plain scalars or all flow mappings, never a mix. A scalar element
carrying any of `{ } [ ] : , " ' #` is refused rather than guessed at. See
it; the block, dotted and inline-colon routes to a mapping still raise. See
[LIMITATIONS](docs/LIMITATIONS.md) for what that admits and what it still walls
off (a `sources` block list of mappings is still refused); **`resource` https-allowlist** (hard-rejects
`data:`/`javascript:`/`file:` before commit — a reject-gate, not defang);
@ -273,7 +269,7 @@ a green scan means safe content. The highest-impact items:
two of the three corpora are living, so the cells are not rewritten in place.
Method and before/after: [`docs/rawhtml-census.py`](docs/rawhtml-census.py).
**Full list — 45 items, each with the mechanism, plus the out-of-scope boundary:**
**Full list — 43 items, each with the mechanism, plus the out-of-scope boundary:**
[`docs/LIMITATIONS.md`](docs/LIMITATIONS.md). Several carry field measurements from
consumer corpora, including the false positives the URL-shape rule actually produces.

View file

@ -1,317 +0,0 @@
# Beslutningsgrunnlag — LIMITATIONS «punkt 44» (tags/description-gapet)
**Målt:** 2026-09-07, mot `HEAD` = `44e2b31` (v1.3.0, urørt) og pinnet SPEC
`_okf-canonical` @ `ad30107`. **Korpus:** `_okf-upstream` @ `3fcbb9f`.
**Hva dette er:** underlaget for én operatørbeslutning — hvilket predikat, om
noe, skal slippe inn den formen `tags` faktisk har i korpuset.
**Hva dette ikke er:** beslutningen. Ingen fil i `src/` er endret, ingen
versjon er bumpet, ingenting er pushet (0 upushede commits ved øktstart).
---
## 1. Referenten: «punkt 44» er ikke grepbart
Ordren og STATE peker på «LIMITATIONS punkt 44». Målt:
```
$ grep -c '^- \*\*' docs/LIMITATIONS.md
45
$ grep -n '^- \*\*' docs/LIMITATIONS.md | awk -F: 'NR==44 {print $1}'
756 # -> ZWJ-oppføringen, ikke tags/description
```
Tallet stammer fra commit `0184df9` (25.08), som skrev «43 -> 44 items» i
commit-meldingen: **44 var antallet oppføringer etter innsettingen, ikke
oppføringens posisjon.** Oppføringen ble satt inn midt i dokumentet og er i dag
den **12. av 45**, på `docs/LIMITATIONS.md:126`. Etiketten «punkt 44» løser seg
altså ikke opp verken ved posisjon eller ved `grep`, og den vil peke feil igjen
neste gang listen vokser. Referenten er entydig i prosa («tags/description»),
men nummeret bør ikke brukes videre.
## 2. Hva punkt 44 sier i dag, ordrett
`docs/LIMITATIONS.md:126-144`:
> **`tags` and `description` block the OKF import corpus universally, before the
> trust layer is even reached.** The line-flat frontmatter parser has no
> sequence-value type at all: `tags` is present in 53/53 upstream concept
> documents — 9/53 as a flow sequence (`[a, b, c]`, rejected on the `[`
> indicator) and 44/53 as a block sequence (`- a` / `- b`, rejected as
> `"malformed frontmatter line"`) — 100% rejection regardless of form.
> `description` is present in 53/53; 29/53 is a folded plain scalar continuing
> on an indented second line, which the parser has no continuation-line model
> for and misreads as `"nested mappings are not supported"` (the remaining
> 24/53 are single-line and parse fine). Measured directly on the upstream
> reference bundles (`_okf-upstream/okf` @ `3fcbb9f`): removing `tags` alone
> lets 4/53 documents pass; removing both `tags` and `description` together
> (trust layer untouched) lets the same 4/53 pass, and all four then parse
> `generated` correctly as a mapping. **Independent of the mapping-form work
> above:** neither `1.2.0`'s flow mapping nor `1.3.0`'s `sources` carriers move
> anything on this corpus, because `tags`/`description` reject before `sources`
> is ever read. No sequence-value type or continuation-line model exists in the
> stdlib-only parser to close this with.
## 3. SPEC, ordrett fra pinnet kopi (`ad30107`)
**§2 Terminology** (`SPEC.md:81`):
> - **Frontmatter**: A YAML metadata block delimited by `---` at the top of
> a markdown file.
**§4 Concept documents** (`SPEC.md:157-159`):
> 1. A **YAML frontmatter block**, delimited by `---` on its own line at the
> start of the file and a closing `---` on its own line.
**§4.1 Frontmatter**, skjelettet (`SPEC.md:163-173`):
> ```yaml
> ---
> type: <Type name> # REQUIRED
> title: <Optional display name>
> description: <Optional one-line summary>
> resource: <Optional canonical URI for the underlying asset>
> tags: [<tag>, <tag>, ...] # Optional
> ```
**§4.1, de to bærende definisjonene** (`SPEC.md:194-199`):
> - `description`: A single sentence summarizing the concept. Used by
> `index.md` generators, search snippets, and previews.
> - `tags`: A YAML list of short strings for cross-cutting categorization.
**§11 Conformance** (`SPEC.md:738-742`):
> A bundle is **conformant** with OKF v0.2 if:
>
> 1. Every non-reserved `.md` file in the tree contains a parseable YAML
> frontmatter block.
### 3.1 Det spec-en ikke sier
`grep -n -i 'nest\|depth\|indent' SPEC.md` gir **null** treff som uttrykker en
dybde- eller innrykksregel (13 treff, alle i andre betydninger — «flat list of»
i §9 og §13, «indent» ingen). **SPEC-en har ingen dybderegel.** «Ingen nesting
forbi dybde 1» er utelukkende vår egen sikkerhetsegenskap. Konformans-gulvet er
«a parseable YAML frontmatter block» — altså *hele* YAML.
To presiseringer som følger av ordlyden:
- `tags` er definert som «A YAML list» — **ikke** som flow-formen. Skjelettets
`[<tag>, ...]` er ett eksempel, ikke formkravet. Blokkformen er like konform.
- `description` er «A single sentence». En setning brutt over to linjer som
foldet plain scalar er samme setning; spec-en stiller ingen linjekrav.
## 4. Måleoppsett
Korpuset er hentet ut ved pinnen, ikke fra arbeidstreet — `_okf-upstream` står
i dag på `9a15b13`, og `3fcbb9f` er en ekte forgjenger (`git merge-base
--is-ancestor` → 0), med 23 filer endret i `okf/` mellom dem.
```
$ git -C _okf-upstream archive 3fcbb9f okf/bundles | tar -x -C scratchpad/corpus
$ find scratchpad/corpus -name '*.md' ! -name index.md ! -name log.md | wc -l
53
```
**Nevner = 53.** Definisjonen er «hver `.md` under `okf/bundles/` som ikke er et
reservert strukturnavn» — som er nøyaktig §11.1s «every non-reserved `.md` file».
**Positiv kontroll.** Et null-resultat må sjekkes mot et kjent-positivt tilfelle
før det konsumeres. Følgende dokument parser i dag, mot samme `HEAD`:
```yaml
type: Metric
description: Recognized revenue for a period.
tags:
- finance # INNRYKKET blokksekvens
- revenue
generated: { by: reference_agent/gemini-2.5-pro, at: 2026-06-30T14:00:00Z }
sources:
- id: revenue-policy
resource: policies/revenue-recognition.md
```
`tags` blir `['finance', 'revenue']`, `sources` blir en liste av dicter.
Instrumentet avviser altså ikke alt; `tags` som blokksekvens **fungerer allerede
i dag**, forutsatt innrykk.
## 5. Målingen
### 5.1 Formsensus over de 53 (`scratchpad/shapes.py`, exit 0)
| nøkkel | form | antall |
|---|---|---|
| `tags` | blokksekvens **uten innrykk** (`- a` i kolonne 0) | **36/53** |
| `tags` | flow-sekvens `[a, b, c]` | 9/53 |
| `tags` | enkeltlinje-skalar | 8/53 |
| `description` | skalar + innrykket fortsettelseslinje | 29/53 |
| `description` | enkeltlinje-skalar | 24/53 |
| `generated` | **topp-nivå blokk-mapping** (` by:` på neste linje) | **44/53** |
| `generated` | flow-mapping `{ ... }` | 9/53 |
| `sources` | blokksekvens uten innrykk | 44/53 |
| `sources` | blokksekvens med innrykk | 5/53 |
### 5.2 Baseline og kandidater (`scratchpad/candidates.py`, exit 0)
Kandidatene er målt ved å **normalisere overflateformen** inn i en form
parseren allerede godtar, og så importere `parse_frontmatter`. Predikatet er
aldri re-implementert; bare stavemåten på inputen er skrevet om.
| variant | passerer | dominerende residual |
|---|---|---|
| baseline (v1.3.0 som utgitt) | **0/53** | 32× nested mappings, 12× malformed line, 9× flow sequence |
| **P1** skalar-flow-sekvens | **6/53** | 32× nested mappings, 12× malformed line |
| **P2** blokksekvens uten innrykk | 0/53 | 44× nested mappings |
| **P3** foldet plain scalar | 0/53 | 36× malformed line |
| P1+P3 | 6/53 | 36× malformed line |
| P1+P2 | 6/53 | 44× nested mappings |
| **P1+P2+P3** | **6/53** | **44× nested mappings, 3× allowlist** |
De 6 som passerer med P1 er alle i `acme_retail`; de er 6 av de 9 med
flow-sekvens-`tags`, og de tre siste stoppes av allowlisten
(`parameters.name` ×2, `not.term` ×1), ikke av `tags`.
### 5.3 Taket
Med alle tre predikatene er taket **6/53**. Residualet er 44× topp-nivå
blokk-mapping på `generated` og 3× allowlist. En diagnostisk kjøring som også
normaliserte topp-nivå blokk-mapping til flow-mapping traff neste vegg med én
gang: 44× «a quoted scalar inside a flow mapping is not a supported form» —
korpuset skriver `at: '2026-07-10T23:16:06+00:00'` med enkeltfnutter.
## 6. Tre påstander i punkt 44 er målt feil
1. **«The line-flat frontmatter parser has no sequence-value type at all.»**
Usant siden 1.3.0. `_consume_block_list` (`okf.py:662`) parser blokklister av
rene skalarer, og den positive kontrollen i §4 beviser det. Det som mangler
er ikke sekvenstypen, men **innrykkskravet**: `_consume_block_list` krever
`raw[:1] in (" ", "\t")`, og korpuset skriver `- a` i kolonne 0.
Feilklassen «44/53 rejected as malformed frontmatter line» er i dag 12/53,
fordi de øvrige treffer `description`-fortsettelsen først.
2. **«removing `tags` alone lets 4/53 documents pass … removing both … the same
4/53.»** Målt i dag: **6/53** med `tags` fjernet, og **6/53** med begge
fjernet. Tallet 4 var riktig for 1.2.0-parseren; 1.3.0s `sources`-bærere
flyttet to dokumenter til.
3. **«`tags`/`description` reject before `sources` is ever read» → derfor er
dette «det ENESTE residualet som blokkerer hele korpuset».** Den slutningen
holder ikke. `description` alene løsner **0/53** — fjerner man bare
`description`, passerer ingenting. Og lukker man *alle tre* formene, står
**44/53** fortsatt på `generated` som topp-nivå blokk-mapping. Den bindende
skranken på dette korpuset er altså **ikke** tags/description, men den
topp-nivå blokk-mappingen vi bevisst avviser. Punkt 44 overselger sin egen
betydning med en faktor på over sju (6 mot 53).
Punkt 44 bør skrives om etter at operatøren har bestemt seg — det er en
dokumentasjonsendring som hører sammen med predikatvalget, ikke før det.
## 7. Kandidatpredikatene
Ordren spurte etter predikatet som slipper inn en ren skalar-flow-sekvens.
Det er P1. P2 og P3 tas med fordi målingen viser at P1 alene er en liten
gevinst, og beslutningen bør se hva naboene koster.
### P1 — flow-sekvens av rene skalarer
Predikat: i `_parse_flow_sequence`, når første ikke-blanke tegn i et element
ikke er `{`, les elementet som en plain scalar dersom det ikke inneholder noen
av `{ } [ ] : , " ' #` og ikke er tomt. Blandet sekvens (skalar + mapping)
avvises, slik blokklisten allerede gjør.
- **(a) Slipper inn:** `tags: [finance, revenue, headline-metric]` — 9/53 i
korpuset. Passeringen går fra 0/53 til **6/53**.
- **(b) Avviser fortsatt:** siterte elementer (`['a', 'b']`), elementer med
kolon eller komma i seg, tom sekvens `[]`, uavsluttet `[a, b`, blandet
`[a, {b: c}]`, og nestet `[[a]]` — alle på tegn-nivå, uten YAML-semantikk.
- **(c) Dybde-1:** **bruker den ikke opp.** Elementene er blad; ingen ny
nestingsgrad oppstår. Det er samme dybde blokklisten av skalarer allerede har.
- **(d) Testen som pinner den:** `tags: [a, b]``["a", "b"]`; `tags: ['a']`
raiser; `tags: [a, {b: c}]` raiser med blandingsfeilen; `tags: []` raiser;
`sources: [{ id: x }]` parser uendret (ingen regresjon på G30-bæreren).
### P2 — blokksekvens uten innrykk
Predikat: i `_consume_block_list`, godta også `raw[:1] == "-"` når linjen
starter med `- ` og forrige toppnøkkel hadde tom verdi.
- **(a) Slipper inn:** `tags:` + `- a` i kolonne 0 — 36/53. Men også `sources:`
i samme form — 44/53.
- **(b) Avviser fortsatt:** alt innholdet i elementet avviser i dag; allowlisten
og `_reject_mapping_construct` er uendret.
- **(c) Dybde-1:** **bruker den ikke opp** for skalarelementer, men den er ikke
gratis: den åpner samtidig den ikke-innrykkede blokk-mapping-bæreren for
`sources`, hvor elementenes fortsettelseslinjer *er* innrykket. Det er en
større flate enn `tags`, og den bør vurderes for seg.
- **(d) Testen:** `tags:\n- a\n- b``["a", "b"]`; `sources:\n- id: x\n title: y`
→ én dict; en linje `- a` uten forutgående tom toppnøkkel raiser fortsatt;
`tags:\n- a\n- {b: c}` raiser med blandingsfeilen.
- **Målt effekt alene: 0/53.** Den løsner ingenting uten P1 eller uten at
`generated` også åpnes.
### P3 — foldet plain scalar (fortsettelseslinje)
Predikat: etter en toppnøkkel med ikke-tom, ikke-`[`/`{` verdi, slå sammen
etterfølgende innrykkede linjer som *ikke* starter med `- ` og *ikke* inneholder
en uquotet `": "`, med ett mellomrom som skjøt.
- **(a) Slipper inn:** `description` brutt over to linjer — 29/53.
- **(b) Avviser fortsatt:** en innrykket linje som ser ut som `k: v` treffer
fremdeles nested-mapping-avvisningen; `- ` treffer fremdeles listeruten.
`_reject_dangerous_value` kjører på den sammenslåtte verdien, ikke på
fragmentene.
- **(c) Dybde-1:** **bruker den ikke opp** — resultatet er én skalar.
Men den svekker et vern: i dag er *enhver* innrykket linje uten aktiv listenøkkel
et avvist nestet uttrykk. Etter P3 er den regelen betinget av at linjen ikke
inneholder `": "` — altså samme heuristikk som `_reject_mapping_construct`,
gjenbrukt til å *slippe gjennom* i stedet for til å avvise.
- **(d) Testen:** `description: en setning\n som fortsetter` → én streng med
ett mellomrom; `description: x\n y: z` raiser fortsatt som nested mapping;
`description: x\n - a` raiser fortsatt.
- **Målt effekt alene: 0/53.**
## 8. Anbefaling
**P1 alene. Ikke P2, ikke P3, ikke nå.**
Begrunnelsen er tallene, ikke smaken:
- P1 er den eneste av de tre som flytter passeringstallet i det hele tatt
(0 → 6). P2 og P3 gir hver for seg **0/53**, og lagt oppå P1 gir de fortsatt
**6/53**. De koster parserflate og kjøper null målt konformans.
- P1 er den minste flaten: den er et tegn-nivå-predikat inne i en funksjon som
allerede eksisterer, og den bruker ikke opp dybde-1-regelen.
- P1 lukker et gap STATE allerede fører som bevisst («`tags: [a, b]`
konformansgap»), og den bringer parseren i linje med §4.1s eget skjelett —
den ene formen spec-en faktisk skriver ut.
- P2 og P3 bør ikke besluttes på dette korpuset, fordi korpuset ikke kan skille
dem: **44/53 stopper på `generated` som topp-nivå blokk-mapping uansett.**
Å bygge P2 og P3 nå ville være å betale for to predikater og måle null.
**Den ærlige konsekvensen, som må sies høyt:** P1 tar korpuset fra 0/53 til
6/53. Det lukker ikke «hele korpuset». Skal 53/53 nås, er den neste
beslutningen en helt annen og mye tyngre en — topp-nivå blokk-mapping (44/53)
pluss siterte skalarer (44/53) — og *den* bruker opp dybde-1-regelen. Det er en
sikkerhetsbeslutning, ikke en parserdetalj, og den hører ikke i denne ordren.
## 9. Verifiseringslogg
| Påstand | Kommando | Exit | Resultat |
|---|---|---|---|
| Nevner = 53 | `find scratchpad/corpus -name '*.md' ! -name index.md ! -name log.md \| wc -l` | 0 | 53 |
| Pinnen er ekte forgjenger | `git -C _okf-upstream merge-base --is-ancestor 3fcbb9f HEAD` | 0 | ja (HEAD = `9a15b13`) |
| SPEC-pinnen er ren | `git -C _okf-canonical rev-parse --short HEAD; git status --porcelain` | 0 | `ad30107`, rent tre |
| SPEC har ingen dybderegel | `grep -n -i 'nest\|depth\|indent' SPEC.md` | 0 | 13 treff, 0 relevante |
| Baseline 0/53 | `PYTHONPATH=src .venv/bin/python scratchpad/measure.py` | 0 | 0 passerer |
| Formsensus | `PYTHONPATH=src .venv/bin/python scratchpad/shapes.py` | 0 | tabell §5.1 |
| Kandidattall | `PYTHONPATH=src .venv/bin/python scratchpad/candidates.py` | 0 | tabell §5.2 |
| Positiv kontroll | inline, se §4 | 0 | parser, `tags == ['finance','revenue']` |
| LIMITATIONS-antall | `grep -c '^- \*\*' docs/LIMITATIONS.md` | 0 | 45 |
| Upushet ved øktstart | `git rev-list --count origin/main..HEAD` | 0 | **0** |
**Ikke målt:** om P1 påvirker ytelse eller ReDoS-marginen — predikatet er
tegn-for-tegn uten regex, men ingen sveip er kjørt, siden ingen kode er skrevet.
Kandidatene er målt ved overflatenormalisering, ikke ved en patchet parser: det
er en trofast simulering av *hva* som slippes inn, men den beviser ikke at en
implementasjon av P1 avviser nøyaktig (b)-listen. Testene i (d) er det som ville
pinne det.

View file

@ -4,7 +4,7 @@
especially one converging on Google's Open Knowledge Format (OKF v0.1) — and needs
to decide **when** and **where** to add a write-time ingestion guard.
**Status of the guard:** `v1.3.0`. Stdlib-only core, framework-agnostic. The
**Status of the guard:** `v1.2.0`. Stdlib-only core, framework-agnostic. The
exported Python surface is frozen under semver — nothing exported is removed,
renamed or given a different meaning without a `2.0.0`. Detection behaviour is
*not* frozen: severities, thresholds and lexicon entries are calibration and move
@ -142,9 +142,9 @@ live payload:
python -m llm_ingestion_guard.coverage # exit 0 = all as documented
```
As of `v1.3.0`: **130 / 130 defended classes demonstrated (recall 100%)** and **6 /
As of `v1.2.0`: **130 / 130 defended classes demonstrated (recall 100%)** and **6 /
6 documented gaps still hold** (a *closed* gap fails the test, forcing a doc
update). The matrix is the single source of truth for the test suite (**868
update). The matrix is the single source of truth for the test suite (**834
passing**), which also asserts total recall, that every lexicon pattern has a
case (so the matrix cannot fall behind the lexicon), the full LLM02 secret-egress
set, and the container-layer front-end (CSV formula-injection, zip-slip/bomb,

View file

@ -3,7 +3,7 @@
**A reusable, minimal, dependency-light defensive layer for LLM *ingestion*
pipelines — the write-time siblings of query-time chatbot guardrails.**
Status: implemented — v1.3.0, exported surface frozen under semver. This document defines what the repo contains
Status: implemented — v1.2.0, exported surface frozen under semver. This document defines what the repo contains
and why; the stdlib-only core is built and tested (see `README.md` for usage and
`docs/PLAN.md` for the build order).

View file

@ -40,31 +40,27 @@ items; this is the full list, each with the mechanism.
(an injection in a directory listing is caught) rather than path-rejecting the
conformant bundle. A front-end materialising individual uploads keeps the opposite
rule (`allow_reserved=False`): a reserved basename is a listing-shadow and refused.
- **OKF frontmatter is a restricted grammar: a mapping is expressible through
four carriers, all of them key-allowlisted.** Gate T2 accepts a line-oriented
subset deliberately — full YAML is a larger parse-attack surface than a
write-time gate needs. The admitted carriers are the flow mapping as a value
(`generated: { by: x, at: y }`), the flow mapping as a block-list item, the
flow *sequence* of flow mappings (`sources: [{ id: a, resource: x }]`) and the
block sequence of block mappings (SPEC.md §5.1's own form). **The routes that
still fail, each on a different rule:** a top-level block *mapping*
(`k:\n sub: v`) on the nested-mapping check, dotted keys (`k.sub: v`) on the
key pattern, the inline second colon (`k: sub: v`) on the mapping-construct
check, and a flow sequence of plain *scalars* (`tags: [a, b]`) on the `[`
indicator — the sequence carrier is opened for the mapping element and nothing
else. **Every carrier is admitted only when every key is on an allowlist**
- **OKF frontmatter is a restricted grammar: the mapping class has exactly one
expressible form.** Gate T2 accepts a line-oriented subset deliberately — full YAML
is a larger parse-attack surface than a write-time gate needs. Flow sequences
(`[a, b]`) and nested mappings are *rejected outright*, which fails secure.
**Three of the four routes to a mapping fail, each on a different rule** — block
(`k:\n sub: v`) on the nested-mapping check, dotted keys (`k.sub: v`) on the key
pattern, and the inline second colon (`k: sub: v`) on the mapping-construct check.
**The fourth, the flow form, is admitted only when every key is on an allowlist**
(`by`, `at`, `from`, `to`, `id`, `title`, `author`, `usage_count`,
`last_modified` — the keys SPEC.md @ `62432a09` §5.1/§5.2 names inside a mapping,
plus `resource` and `usage_window` *under `sources` only*, see below)
`last_modified` — the keys SPEC.md @ `62432a09` §5.1/§5.2 names inside a mapping)
and every leaf is a plain scalar, itself run through the same value predicates as a
top-level scalar. Nested collections, quoted leaves, duplicate keys, an empty or
unclosed mapping, and `{a:b}` (which PyYAML 6.0.3 reads as the *key* `a:b`, not as
a scalar) all raise. The form is expressible, never trusted: the allowlist
inspects every key, which is the property that carried the security when the
blanket refusal was doing the enforcing. **A block list may not mix scalar items
and mappings** — YAML permits it, but a consumer iterating `sources` and reading
`entry.get("id")` gets an `AttributeError` off the first `str`. What else
survives is scalars and flat lists of strings. **Two routes used to
blanket refusal was doing the enforcing. **`resource` is deliberately off the
allowlist** although §5.1 names it inside a `sources` entry — it is a pointer
rather than a label and the only key T3 exists for, so admitting it would let
`executor: { resource: skills/run.md }` carry an executable-code pointer through a
key the https allowlist never inspects. What else survives is scalars and flat
lists of strings. **Two routes used to
degrade into a string instead of failing, and that defect is closed in `1.1.0`**:
a block-sequence item carrying exactly one key (`sources:\n - uri: https://e.com/a`
yielded the *string* `'uri: https://e.com/a'`) and the inline second colon
@ -89,76 +85,19 @@ items; this is the full list, each with the mechanism.
door A/B persist path, so frontmatter that fails secure on import passes
`screen_output` unremarked. The grammar therefore bounds what a consumer can *receive*,
never what a producer can *emit*. Verified identical on 0.2.0 and 0.3.1.
- **`sources` passes in both of its spec carriers; the per-entry `usage_window`
does not.** The wall used to be total: both of v0.2's backward-breaking migration
targets are mappings — `timestamp``generated.at`, and body `# Citations` → a
`sources` block list of mappings — and a consumer measured **0 of 53** upstream
concepts through the gate. `generated.at` stopped being a wall in `1.2.0`;
`sources` stopped being one in `1.3.0`, which admits both the block sequence of
block mappings (SPEC.md §5.1's own example) and the flow sequence of flow
mappings (the form the OKF producers emit, measured 2026-09-02 by two consumers
independently). **`resource` is allowlisted inside a `sources` entry and nowhere
else.** `1.2.0` left it off on the argument that the parser could not tell
`sources[].resource` (§5.1, a citation) from `executor.resource` / `attester.resource`
(§10, a pointer to code to be run — door C). That premise was measured false: the
owning key is in scope at every call site and was simply never threaded through.
It is threaded now, so `executor: [{ resource: skills/run.md }]` and
`attester:\n - resource: …` are refused on the allowlist through *every* carrier,
including the two this opened. **What stays refused: a `usage_window` inside a
`sources` entry.** SPEC §5.1 permits it per entry ("A single entry MAY carry its
own `usage_window`"), and it is a mapping inside a mapping — depth 2, which this
parser admits at no key. A bundle using the per-entry override is refused; the
shared sibling `usage_window` (the §5.1 example's own form) passes. This is a
registered conformance gap, not an oversight: no-nesting-past-depth-1 is a
security property, and spending it was not what the fix was for.
- **`sources[].resource` is scanned as text but never validated as a URL.** T3's
https allowlist inspects the *top-level* `resource` and nothing else. It cannot
be extended to `sources` entries without over-blocking conformant bundles: SPEC
§5.1 explicitly permits a bundle-relative path, a path into `references/`, or a
scope descriptor a consumer cannot follow at all (the OKF producers' own golden
bundle emits `resource: fixture`). So a `sources` entry may carry
`file://`, `javascript:` or any other string; it goes through T1's scan like any
other frontmatter value, and nothing else. **A consumer that dereferences
`sources[].resource` must validate it itself** — `okf.validate_resource_url` is
exported for exactly that. The dangling-or-substituted `executor`/`attester`
pointer question stays out of reach separately: both are top-level block
mappings, a carrier that is still refused.
- **`tags` parses in two of the three forms the OKF corpus writes it in;
`description`'s folded form still does not — and neither is what caps the
corpus.** Re-measured 2026-09-07/08 against the pinned upstream reference
bundles (`_okf-upstream/okf` @ `3fcbb9f`, denominator **53** — every
non-reserved `.md`, which is SPEC §11.1's own unit) and the pinned SPEC
(`_okf-canonical` @ `ad30107`). This entry carried three claims that the
measurement showed to be wrong, all three inherited from `1.2.0` and left
standing for twelve days after `1.3.0` moved them; they are corrected here.
- **A sequence-value type exists**`_consume_block_list` has parsed block
lists of plain scalars since `1.3.0`, and `1.4.0` adds the flow sequence of
plain scalars (`tags: [a, b, c]`, SPEC §4.1's own skeleton). What is missing
is not the type but the **indentation requirement**: `_consume_block_list`
demands a leading space, and the corpus writes `- a` flush at column 0
(36/53 of `tags`, 44/53 of `sources`). Shape census of `tags` over the 53:
36/53 flush block sequence, 9/53 flow sequence, 8/53 single-line scalar.
- **The number is 6/53, not 4/53.** `4` was the `1.2.0` parser's figure.
Measured on `1.4.0`, the corpus goes from **0/53 to 6/53** — all six in
`acme_retail`, being 6 of the 9 documents whose `tags` is a flow sequence;
the other three stop on the mapping-key allowlist (`parameters.name` ×2,
`not.term` ×1), not on `tags`.
- **`tags`/`description` are not the binding constraint.** `description`
alone releases **0/53** — 29/53 write it as a folded plain scalar over an
indented second line, which the parser has no continuation-line model for,
but removing that obstacle by itself lets nothing through. With all three
surface forms closed the corpus still measures **6/53**, because **44/53
stop on `generated` written as a top-level block mapping**, and behind that
wall sit 44/53 single-quoted scalars inside the mapping. That constraint is
refused deliberately: it spends the no-nesting-past-depth-1 rule, which is
a security decision and not a parser detail.
**Said plainly: `1.4.0` takes this corpus to 6/53, not to 53/53.** The two
neighbouring predicates were measured and deliberately **not** built — a
flush-left block sequence and a folded plain scalar each release **0/53** on
their own, and stacked on top of `1.4.0` they still measure 6/53. They would
cost parser surface and buy no measured conformance. The full measurement,
including the SPEC quotations and the per-candidate table, is
`docs/2026-09-07-limitations-44-maaling.md`.
- **An OKF v0.2 concept traverses the external-import path only if its `sources` are
flat.** The wall used to be total: both of v0.2's backward-breaking migration targets
are mappings — `timestamp``generated.at`, and body `# Citations` → a `sources`
block list of mappings — and a consumer measured **0 of 53** upstream concepts
through the gate. The trust and provenance layer now passes in its spec form
(`generated`, `verified` bare or listed, `usage_window`), so `generated.at` is no
longer a wall. **`sources` still is**: SPEC.md writes each entry as a block mapping
under a block sequence (`- id: …\n resource: …`), and that carrier stays refused —
it is the shape whose one-key degradation smuggled a pointer before `1.1.0`, and
reopening it is a separate parse-safety decision, not a corollary of the flow form.
A concept whose `sources` are flat strings, or absent, imports. The
dangling-or-substituted `executor`/`attester` pointer question stays out of reach
for the same reason: both are mappings whose payload key is `resource`.
- **A persist gate cannot cover execution risk.** OKF v0.2 introduces concepts whose
purpose is to *name code to be run* (`runtime`, `executor.resource`,
`attester.resource`). This library answers "is this safe to **store**"; executable

View file

@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "llm-ingestion-guard"
version = "1.4.0"
version = "1.2.0"
description = "Write-time defensive layer for Python pipelines that persist LLM output: sanitize, fence, tool-less quarantined transform, capability isolation, scan before persist, fail-secure."
readme = "README.md"
requires-python = ">=3.10"

View file

@ -1,155 +0,0 @@
"""What each candidate predicate would admit, measured by normalizing the SURFACE
form onto a shape the CURRENT parser already accepts. The predicate is imported,
never re-implemented: only the input's spelling is rewritten."""
import os, re, sys, collections
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src"))
from llm_ingestion_guard import okf
ROOT = os.path.join(os.path.dirname(__file__), "corpus", "okf", "bundles")
def docs():
out = []
for dirpath, _d, fns in os.walk(ROOT):
for fn in sorted(fns):
if fn.endswith(".md") and fn not in ("index.md", "log.md"):
out.append(os.path.join(dirpath, fn))
return sorted(out)
def split_fm(text):
lines = text.split("\n")
if not lines or lines[0].strip() != "---":
return None, None
for i in range(1, len(lines)):
if lines[i].strip() == "---":
return lines[1:i], lines[i + 1:]
return None, None
TOPKEY = re.compile(r"^([A-Za-z0-9_]+):(.*)$")
def n_flow_scalar_seq(fm):
"""P1: `k: [a, b]` where every element is a plain scalar -> indented block list."""
out = []
for raw in fm:
m = TOPKEY.match(raw)
if m and m.group(2).strip().startswith("[") and m.group(2).strip().endswith("]"):
inner = m.group(2).strip()[1:-1]
elems = [e.strip() for e in inner.split(",")]
if inner and all(e and "{" not in e and "}" not in e and "[" not in e for e in elems):
out.append("%s:" % m.group(1))
out.extend(" - %s" % e for e in elems)
continue
out.append(raw)
return out
def n_flush_block_seq(fm):
"""P2: a flush-left `- item` run under a bare `k:` -> the same run, indented."""
out, i, n = [], 0, len(fm)
while i < n:
raw = fm[i]
m = TOPKEY.match(raw)
if m and m.group(2).strip() == "" and i + 1 < n and fm[i + 1][:1] == "-":
out.append(raw)
i += 1
while i < n and (fm[i][:1] == "-" or fm[i][:1] in (" ", "\t")):
out.append(" " + fm[i])
i += 1
continue
out.append(raw)
i += 1
return out
def n_folded_scalar(fm):
"""P3: `k: text` continued on more-indented plain lines -> one joined line."""
out, i, n = [], 0, len(fm)
while i < n:
raw = fm[i]
m = TOPKEY.match(raw)
if m and m.group(2).strip() and not m.group(2).strip()[0] in "[{":
acc = raw
i += 1
while i < n and fm[i][:1] in (" ", "\t") and not fm[i].strip().startswith("- "):
acc = acc.rstrip() + " " + fm[i].strip()
i += 1
out.append(acc)
continue
out.append(raw)
i += 1
return out
def parses(fm, body):
text = "---\n" + "\n".join(fm) + "\n---\n" + "\n".join(body)
try:
okf.parse_frontmatter(text)
return None
except okf.OKFFrontmatterError as exc:
return str(exc)
COMBOS = [
("baseline (v1.3.0 as shipped)", []),
("P1 scalar flow sequence", [n_flow_scalar_seq]),
("P2 flush block sequence", [n_flush_block_seq]),
("P3 folded plain scalar", [n_folded_scalar]),
("P1+P3", [n_flow_scalar_seq, n_folded_scalar]),
("P2+P3", [n_flush_block_seq, n_folded_scalar]),
("P1+P2", [n_flow_scalar_seq, n_flush_block_seq]),
("P1+P2+P3", [n_flow_scalar_seq, n_flush_block_seq, n_folded_scalar]),
]
def main():
paths = docs()
print("denominator: %d documents (pin 3fcbb9f, parser HEAD)" % len(paths))
for name, fns in COMBOS:
ok = 0
residual = collections.Counter()
for p in paths:
fm, body = split_fm(open(p, encoding="utf-8").read())
for fn in fns:
fm = fn(fm)
err = parses(fm, body)
if err is None:
ok += 1
else:
residual[err.split(":")[0]] += 1
top = "; ".join("%dx %s" % (c, r) for r, c in residual.most_common(3))
print(" %-30s %2d/%d residual: %s" % (name, ok, len(paths), top or "-"))
main()
# --- diagnostic only (NOT a proposal): size the real binding constraint ---
def n_block_mapping(fm):
"""D4: a top-level `k:` followed by indented `a: b` lines -> one flow mapping."""
out, i, n = [], 0, len(fm)
while i < n:
raw = fm[i]
m = TOPKEY.match(raw)
if m and m.group(2).strip() == "" and i + 1 < n and fm[i + 1][:1] in (" ", "\t") \
and not fm[i + 1].strip().startswith("- "):
pairs = []
i += 1
while i < n and fm[i][:1] in (" ", "\t") and not fm[i].strip().startswith("- "):
pairs.append(fm[i].strip())
i += 1
out.append("%s: { %s }" % (m.group(1), ", ".join(pairs)))
continue
out.append(raw)
i += 1
return out
print("\n--- diagnostic: what caps the corpus ABOVE the tags/description gap ---")
for name, fns in [
("D4 top-level block mapping", [n_block_mapping]),
("P1+P2+P3+D4", [n_flow_scalar_seq, n_flush_block_seq, n_folded_scalar, n_block_mapping]),
]:
ok = 0
residual = collections.Counter()
for p in docs():
fm, body = split_fm(open(p, encoding="utf-8").read())
for fn in fns:
fm = fn(fm)
err = parses(fm, body)
if err is None:
ok += 1
else:
residual[err.split(":")[0]] += 1
top = "; ".join("%dx %s" % (c, r) for r, c in residual.most_common(3))
print(" %-30s %2d/53 residual: %s" % (name, ok, top or "-"))

View file

@ -1,45 +0,0 @@
"""Measure the pinned OKF corpus against the CURRENT parser. No src/ changes."""
import os, sys, collections
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src"))
from llm_ingestion_guard import okf
ROOT = os.path.join(os.path.dirname(__file__), "corpus", "okf", "bundles")
def docs():
out = []
for dirpath, _dirnames, filenames in os.walk(ROOT):
for fn in sorted(filenames):
if not fn.endswith(".md"):
continue
if fn in ("index.md", "log.md"):
continue
out.append(os.path.join(dirpath, fn))
return sorted(out)
def main():
paths = docs()
ok, fail = [], []
reasons = collections.Counter()
for p in paths:
with open(p, encoding="utf-8") as fh:
text = fh.read()
try:
okf.parse_frontmatter(text)
ok.append(p)
except okf.OKFFrontmatterError as exc:
fail.append((p, str(exc)))
reasons[str(exc).split(":")[0]] += 1
print("denominator: %d documents" % len(paths))
print("parse OK : %d" % len(ok))
print("parse FAIL : %d" % len(fail))
print("--- failure classes ---")
for r, c in reasons.most_common():
print("%4d %s" % (c, r))
print("--- passing documents (positive control) ---")
for p in ok:
print(" " + os.path.relpath(p, ROOT))
print("--- first 8 failures verbatim ---")
for p, e in fail[:8]:
print(" %s\n %s" % (os.path.relpath(p, ROOT), e))
main()

View file

@ -1,59 +0,0 @@
"""Shape census of the pinned OKF corpus: which surface form each key is written in."""
import os, re, sys, collections
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src"))
from llm_ingestion_guard import okf
ROOT = os.path.join(os.path.dirname(__file__), "corpus", "okf", "bundles")
def docs():
out = []
for dirpath, _d, fns in os.walk(ROOT):
for fn in sorted(fns):
if fn.endswith(".md") and fn not in ("index.md", "log.md"):
out.append(os.path.join(dirpath, fn))
return sorted(out)
def split_fm(text):
lines = text.split("\n")
if not lines or lines[0].strip() != "---":
return None, None
for i in range(1, len(lines)):
if lines[i].strip() == "---":
return lines[1:i], lines[i + 1:]
return None, None
def key_shape(fm, key):
for i, raw in enumerate(fm):
m = re.match(r"^(%s):(.*)$" % re.escape(key), raw)
if not m:
continue
val = m.group(2).strip()
follow = fm[i + 1] if i + 1 < len(fm) else ""
if val.startswith("["):
return "flow-sequence"
if val.startswith("{"):
return "flow-mapping"
if val:
if follow[:1] in (" ", "\t") and not follow.strip().startswith("- "):
return "scalar+continuation"
return "single-line-scalar"
if follow.strip().startswith("- "):
return "block-seq-indented" if follow[:1] in (" ", "\t") else "block-seq-flush"
if follow[:1] in (" ", "\t"):
return "block-mapping"
return "empty"
return "ABSENT"
def main():
paths = docs()
print("denominator: %d" % len(paths))
for key in ("tags", "description", "generated", "verified", "sources"):
c = collections.Counter()
for p in paths:
fm, _ = split_fm(open(p, encoding="utf-8").read())
c[key_shape(fm, key)] += 1
print("\n%s:" % key)
for shape, n in c.most_common():
print(" %-22s %d/%d" % (shape, n, len(paths)))
main()

View file

@ -63,7 +63,7 @@ from .grounding import (
)
from . import okf
__version__ = "1.4.0"
__version__ = "1.2.0"
# --- §6 bookends: the two library-side halves around the transform ---------

View file

@ -542,14 +542,8 @@ def _build_cases() -> list[Case]:
lambda: okf.parse_frontmatter("---\nkey:\n nested: x\n---\nbody\n"), owasp="LLM10"),
_raise_case("okf", "T2 frontmatter block scalar", "OKFFrontmatterError",
lambda: okf.parse_frontmatter("---\ndesc: |\n block\n---\nbody\n"), owasp="LLM10"),
# The plain-scalar flow sequence is ADMITTED as of 1.4.0 (P1, SPEC §4.1's
# own skeleton for `tags`); the class this row measures is the element
# shape the parser would have to interpret rather than read -- here a
# quoted one, which it would have to strip a quote from to return.
_raise_case("okf", "T2 frontmatter flow sequence, quoted element",
"OKFFrontmatterError",
lambda: okf.parse_frontmatter("---\ntags: ['a', 'b']\n---\nbody\n"),
owasp="LLM10"),
_raise_case("okf", "T2 frontmatter flow sequence", "OKFFrontmatterError",
lambda: okf.parse_frontmatter("---\ntags: [a, b]\n---\nbody\n"), owasp="LLM10"),
_raise_case("okf", "T2 mapping key off the allowlist", "OKFFrontmatterError",
lambda: okf.parse_frontmatter(
"---\ngenerated: { by: a, tool: shell }\n---\nbody\n"), owasp="LLM10"),

View file

@ -65,57 +65,32 @@ _KEY_RE = re.compile(r"^[A-Za-z0-9_][A-Za-z0-9_-]*$")
# A plain OKF scalar cannot *begin* with a YAML structural indicator. Any value
# starting with one signals an anchor (&), alias (*), explicit tag (!), block
# scalar (|, >), flow collection ([ ] { }), directive (%) or reserved char
# (@ `) — all outside the supported subset and all rejected. `{` and `[` are
# tried as the allowlisted mapping form (G3) and the flow sequence of them (G30)
# FIRST; they reach this predicate only as a leaf inside one, where a nested
# collection is refused before it can be read.
# (@ `) — all outside the supported subset and all rejected. `{` is tried as the
# allowlisted mapping form FIRST (G3); it reaches this predicate only as a leaf
# inside one, where a nested collection is refused before it can be read.
_DANGEROUS_VALUE_STARTS = frozenset("&*!|>[]{}%@`")
# A quoted scalar is a scalar in YAML however many colons it carries, so the
# mapping check steps aside for one. The quotes are retained rather than
# stripped — a pre-existing divergence, pinned in tests/test_okf.py.
_QUOTE_STARTS = frozenset("\"'")
# P1 - what disqualifies a flow-sequence element from being a plain scalar
# (operator decision, 2026-09-08). Each character is one this parser would have
# to interpret rather than read: the two quotes (retained, never stripped), the
# two splitters, the two collection openers and their closers, and the comment
# indicator. Refusing them is what lets the element be split on commas at the
# character level without a YAML quote state machine.
_FLOW_SCALAR_REFUSED = "{}[]:,\"'#"
# G3 - the one mapping form T2 can express (operator decision, 2026-08-21).
# G3 — the one mapping form T2 can express (operator decision, 2026-08-21).
# Every key inside a mapping must be on this allowlist: the form is safe because
# the allowlist inspects each key, not because mappings became trusted. The keys
# are the ones OKF v0.2 names inside a mapping - `by`/`at` (SPEC.md @ 62432a09
# §5.2 `generated`/`verified`) and `from`/`to` (§5.1 `usage_window`), plus the
# §5.1 `sources`-entry labels.
# §5.2 `generated`/`verified`), `from`/`to` (§5.1 `usage_window`) and the
# `sources`-entry fields (§5.1). `resource` is the one §5.1 key deliberately
# LEFT OFF: it is a pointer rather than a label, it is the only key T3 exists
# for, and admitting it inside a mapping would re-open the door-C route closed
# in 1.1.0 (`executor: {resource: skills/run.md}` puts an executable-code
# pointer in a key the https allowlist never inspects). It costs nothing today,
# because the conformant carrier for `sources[].resource` is the block-sequence
# of block-mappings, which this form does not admit either way.
_MAPPING_KEY_ALLOWLIST = frozenset({
"by", "at", "from", "to", "id", "title", "author", "usage_count",
"last_modified",
})
# G30 - the two §5.1 keys admitted inside a `sources` entry and NOWHERE else
# (operator decision, 2026-09-02). `resource` is REQUIRED within a `sources`
# entry, so leaving it off left the whole provenance family unwritable; but the
# same field name in §10 (`executor.resource`, `attester.resource`) names run
# instructions and code - the door-C route closed in 1.1.0. 1.2.0 argued the
# parser could not tell the two apart without parent-key context it did not
# have. That premise was false: the owning key is in scope at every call site
# below, it was simply never threaded through. It is threaded now, so the
# discrimination is structural rather than a judgement about the value.
# `usage_window` is allowlisted here for accuracy of refusal - §5.1 permits it
# per entry, and it is then refused on the depth rule (a mapping inside a
# mapping, which this parser admits at no key) rather than refused as if the
# key were unknown.
_SOURCES_ENTRY_KEYS = frozenset({"resource", "usage_window"})
def _allowed_mapping_keys(parent_key):
"""The mapping-key allowlist for a mapping owned by ``parent_key``."""
if parent_key == "sources":
return _MAPPING_KEY_ALLOWLIST | _SOURCES_ENTRY_KEYS
return _MAPPING_KEY_ALLOWLIST
class OKFError(Exception):
"""Base class for OKF adapter rejections."""
@ -209,10 +184,8 @@ def _value_regions(value):
A mapping value (G3) is a new *shape* on this surface, not a new exemption:
its leaves are scanned exactly like a scalar or a list item, so an injection
parked in ``generated: { by: ... }`` reaches ``scan_output`` like any other
frontmatter text. The same holds for a *list* of mappings (G30, ``sources``),
which this function already flattens through its list branch. Mapping *keys*
are not scanned because they cannot carry attacker text - the allowlist
admits a fixed, per-parent name set and nothing else.
frontmatter text. Mapping *keys* are not scanned because they cannot carry
attacker text - the allowlist admits nine fixed names and nothing else.
"""
if isinstance(value, dict):
return [leaf for leaf in value.values() if leaf]
@ -643,22 +616,16 @@ def _parse_flat(fm_lines):
raise OKFFrontmatterError("invalid frontmatter key: %r" % key)
if value == "":
items, i = _consume_block_list(fm_lines, i + 1, key)
items, i = _consume_block_list(fm_lines, i + 1)
result[key] = items if items is not None else ""
continue
mapping = _parse_flow_mapping(value, key)
mapping = _parse_flow_mapping(value)
if mapping is not None:
result[key] = mapping
i += 1
continue
sequence = _parse_flow_sequence(value, key)
if sequence is not None:
result[key] = sequence
i += 1
continue
_reject_dangerous_value(value)
_reject_mapping_construct(value)
result[key] = value
@ -667,26 +634,14 @@ def _parse_flat(fm_lines):
return result
def _consume_block_list(fm_lines, start, parent_key=None):
def _consume_block_list(fm_lines, start):
"""Consume `` - item`` lines following a bare ``key:``.
Returns ``(items, next_index)`` - ``items`` is ``None`` (and ``next_index``
Returns ``(items, next_index)`` ``items`` is ``None`` (and ``next_index``
unchanged) when no list item follows, so the caller can treat the key as an
empty scalar and let the next line trip the nested-structure guard.
An item is one of three shapes, decided by the item text alone: a flow
mapping (G3), a block mapping (G30 - an unquoted ``key: value`` opening a
run of more-indented sibling entries, which is SPEC.md §5.1's own carrier
for ``sources``), or a plain scalar. ``parent_key`` is the key that owns the
list; it decides the mapping-key allowlist, which is how ``sources[].resource``
is admitted while ``executor``/``attester`` ``resource`` stays refused.
A list may not mix scalars and mappings. YAML permits it, but a consumer
iterating ``sources`` and reading ``entry.get("id")`` gets an
``AttributeError`` off the first ``str`` - refusing is the cheaper failure.
"""
items = []
kinds = set()
i = start
n = len(fm_lines)
while i < n:
@ -695,113 +650,24 @@ def _consume_block_list(fm_lines, start, parent_key=None):
if stripped == "" or stripped.startswith("#"):
i += 1
continue
if not (raw[:1] in (" ", "\t") and stripped.startswith("- ")):
break
if raw[:1] in (" ", "\t") and stripped.startswith("- "):
item = stripped[2:].strip()
mapping = _parse_flow_mapping(item, parent_key)
mapping = _parse_flow_mapping(item)
if mapping is not None:
items.append(mapping)
kinds.add("mapping")
i += 1
continue
entry = _block_mapping_entry(item)
if entry is not None:
mapping, i = _consume_block_mapping(fm_lines, i + 1, entry, parent_key)
items.append(mapping)
kinds.add("mapping")
continue
_reject_dangerous_value(item)
_reject_mapping_construct(item)
items.append(item)
kinds.add("scalar")
i += 1
if len(kinds) > 1:
raise OKFFrontmatterError(
"a block list may not mix scalar items and mappings: %r" % (parent_key,)
)
continue
break
if not items:
return None, start
return items, i
def _block_mapping_entry(text):
"""Read ``text`` as one ``key: value`` block-mapping entry, or return ``None``.
The trigger is deliberately the same shape ``_reject_mapping_construct``
uses to *refuse* a scalar: an unquoted ``": "``. What changes in 1.3.0 is
only what happens next - the entry is admitted key-by-key against the
allowlist instead of refused wholesale. Every shape that is a scalar to
PyYAML stays one here: a quoted item, a colon with no space
(``domain:security``, ``https://e.com:8443/a``) and a trailing colon all
return ``None`` and fall through to the unchanged scalar rules.
"""
if not text or text[0] in _QUOTE_STARTS:
return None
key, sep, leaf = text.partition(": ")
if not sep:
return None
key = key.strip()
if not _KEY_RE.match(key):
return None
return key, leaf.strip()
def _consume_block_mapping(fm_lines, start, first_entry, parent_key):
"""Consume the sibling entries of a block mapping opened by a ``- `` item.
Returns ``(mapping, next_index)``. A sibling is an indented line that does
not open a new list item; the run ends at a blank line, a comment, a new
``- `` item, or a line at column zero. Depth is capped at one by giving the
leaves the *unchanged* scalar predicates: a nested collection opens with
``{`` or ``[`` and is refused by ``_reject_dangerous_value``, and a further
block level is refused by ``_reject_mapping_construct``.
"""
allowed = _allowed_mapping_keys(parent_key)
mapping = {}
_admit_mapping_entry(mapping, first_entry[0], first_entry[1], allowed, parent_key)
i = start
n = len(fm_lines)
while i < n:
raw = fm_lines[i]
stripped = raw.strip()
if stripped == "" or stripped.startswith("#"):
break
if raw[:1] not in (" ", "\t") or stripped.startswith("- "):
break
entry = _block_mapping_entry(stripped)
if entry is None:
_reject_dangerous_value(stripped)
_reject_mapping_construct(stripped)
raise OKFFrontmatterError(
"a block-mapping entry must be 'key: value': %r" % (raw,)
)
_admit_mapping_entry(mapping, entry[0], entry[1], allowed, parent_key)
i += 1
return mapping, i
def _admit_mapping_entry(mapping, key, leaf, allowed, parent_key):
"""Admit one mapping entry, or raise. The single gate both carriers pass."""
if not _KEY_RE.match(key):
raise OKFFrontmatterError("invalid mapping key: %r" % (key,))
if key not in allowed:
raise OKFFrontmatterError(
"mapping key %r is not on the OKF mapping allowlist under %r"
% (key, parent_key)
)
if key in mapping:
raise OKFFrontmatterError("duplicate mapping key %r" % (key,))
_reject_dangerous_value(leaf)
_reject_mapping_construct(leaf)
mapping[key] = leaf
def _reject_dangerous_value(value):
if value and value[0] in _DANGEROUS_VALUE_STARTS:
raise OKFFrontmatterError(
@ -838,7 +704,7 @@ def _reject_mapping_construct(value):
)
def _parse_flow_mapping(value, parent_key=None):
def _parse_flow_mapping(value):
"""Parse ``{ key: value, ... }`` into a typed dict, or refuse it (G3).
Returns ``None`` when ``value`` does not open a flow mapping, so the caller
@ -907,7 +773,6 @@ def _parse_flow_mapping(value, parent_key=None):
% (value,)
)
allowed = _allowed_mapping_keys(parent_key)
mapping = {}
for entry in inner.split(","):
entry = entry.strip()
@ -916,122 +781,20 @@ def _parse_flow_mapping(value, parent_key=None):
raise OKFFrontmatterError(
"a flow-mapping entry must be 'key: value': %r" % (entry,)
)
_admit_mapping_entry(mapping, key.strip(), leaf.strip(), allowed, parent_key)
key = key.strip()
leaf = leaf.strip()
if not _KEY_RE.match(key):
raise OKFFrontmatterError("invalid flow-mapping key: %r" % (key,))
if key not in _MAPPING_KEY_ALLOWLIST:
raise OKFFrontmatterError(
"flow-mapping key %r is not on the OKF mapping allowlist: %r"
% (key, value)
)
if key in mapping:
raise OKFFrontmatterError(
"duplicate flow-mapping key %r: %r" % (key, value)
)
_reject_dangerous_value(leaf)
_reject_mapping_construct(leaf)
mapping[key] = leaf
return mapping
def _parse_flow_sequence(value, parent_key=None):
"""Parse ``[{ ... }, { ... }]`` into a list of typed dicts, or refuse it (G30).
Returns ``None`` when ``value`` does not open a flow sequence, so the caller
falls through to the unchanged scalar rules - where ``[`` is still a
disallowed indicator. This carrier is opened for the flow-mapping element
and nothing else: it is the form the OKF producers emit for ``sources``
(measured 02.09 against llm-ingestion-okf's golden bundle, where a
one-element sequence raised on the ``[`` just as a two-element one did).
As of 1.4.0 it also carries a sequence of plain *scalars*
(``tags: [a, b, c]``) - SPEC.md §4.1's own skeleton for ``tags``, and the
one candidate form measured to move the upstream corpus at all (0/53 ->
6/53 against ``_okf-upstream`` @ 3fcbb9f, denominator 53; see
docs/2026-09-07-limitations-44-maaling.md). The quoting and comma-splitting
problem that kept it refused is answered by refusing the characters that
create it rather than by parsing them: an element is a plain scalar only if
it is non-empty and carries none of ``{ } [ ] : , " ' #``, and it then
passes the unchanged scalar indicator rule. Everything needing YAML
semantics to split or unquote correctly still raises.
A mapping element is split on ``}`` rather than on commas, which is sound
precisely because ``_parse_flow_mapping`` admits no nested collection: a
``}`` inside an element cannot occur, so the first ``}`` after ``{`` always
closes it. A scalar element runs to the next comma, which cannot occur
inside one. Anything between elements that is not a separating comma is
refused, which is what makes trailing junk fail rather than parse.
A sequence may not mix the two, for the reason the block list may not: a
consumer iterating the value and reading ``entry.get("id")`` gets an
``AttributeError`` off the first ``str``. The mixing verdict is reached
before the element is parsed, so the caller is told about the mix rather
than about a key the allowlist would have complained of instead.
"""
if not value or value[0] != "[":
return None
if not value.endswith("]"):
raise OKFFrontmatterError(
"a flow sequence must be closed by ']' on the same line: %r" % (value,)
)
inner = value[1:-1].strip()
if not inner:
raise OKFFrontmatterError("an empty flow sequence carries nothing: %r" % (value,))
items = []
kinds = set()
i = 0
n = len(inner)
while True:
while i < n and inner[i] in " \t":
i += 1
if i >= n:
break
if inner[i] == "{":
_refuse_mixed_flow_sequence(kinds, "mapping", value)
close = inner.find("}", i)
if close == -1:
raise OKFFrontmatterError(
"an unclosed flow mapping inside a flow sequence: %r" % (value,)
)
items.append(_parse_flow_mapping(inner[i:close + 1], parent_key))
i = close + 1
else:
_refuse_mixed_flow_sequence(kinds, "scalar", value)
end = i
while end < n and inner[end] != ",":
end += 1
items.append(_flow_sequence_scalar(inner[i:end].strip(), value))
i = end
while i < n and inner[i] in " \t":
i += 1
if i >= n:
break
if inner[i] != ",":
raise OKFFrontmatterError(
"trailing junk after a flow-sequence element: %r" % (value,)
)
i += 1
return items
def _refuse_mixed_flow_sequence(kinds, kind, value):
kinds.add(kind)
if len(kinds) > 1:
raise OKFFrontmatterError(
"a flow sequence may not mix scalar items and mappings: %r" % (value,)
)
def _flow_sequence_scalar(element, value):
"""Read one flow-sequence element as a plain scalar, or refuse it (P1).
Character-level, with no YAML semantics: the element must be non-empty and
carry none of :data:`_FLOW_SCALAR_REFUSED`. That set is not a style rule -
each member is a character whose meaning this parser would have to guess at.
A quote would have to be stripped (this parser retains quotes, so it would
hand back a different value than YAML reads); a comma or a colon would have
to be split on; a bracket or a brace would open a second collection level,
which no carrier here admits; a ``#`` opens a comment. The unchanged
indicator rule then applies to what is left, exactly as it does to a
block-list item, so an anchor or an alias is no more a scalar here.
"""
if not element:
raise OKFFrontmatterError(
"an empty element in a flow sequence carries nothing: %r" % (value,)
)
for char in _FLOW_SCALAR_REFUSED:
if char in element:
raise OKFFrontmatterError(
"a flow-sequence scalar admits plain scalars only, not %r: %r"
% (char, value)
)
_reject_dangerous_value(element)
return element

View file

@ -116,13 +116,9 @@ def test_rejects_unterminated_frontmatter():
def test_rejects_flow_collection():
# An inline flow collection this parser cannot read without guessing ->
# reject, don't silently mis-parse the bracket string as a scalar. The
# QUOTED elements are what makes this row stay red: the plain-scalar form
# (`tags: [pii, customers]`) is admitted as of 1.4.0 -- see the P1 block at
# the foot of this file -- while a quoted element would have to be stripped
# to be read, and this parser retains quotes.
doc = "---\ntype: table\ntags: ['pii', 'customers']\n---\nbody\n"
# inline flow collections are outside the supported subset -> reject, don't
# silently mis-parse the bracket string as a scalar.
doc = "---\ntype: table\ntags: [pii, customers]\n---\nbody\n"
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(doc)
@ -552,15 +548,9 @@ _V02_REJECTED = [
("generated (nested)", "generated:\n at: 2026-07-26T10:00:00Z\n"),
("executor (nested)", "executor:\n resource: skills/run-on-bq.md\n"),
("attester (nested)", "attester:\n resource: attesters/sql_equality.py\n"),
# The carrier is admitted as of 1.3.0 (G30); this row now measures the KEY
# SET - `uri`/`kind` are producer-invented, not SPEC §5.1 - and stays red for
# that reason. SPEC's own §5.1 keys parse; see the G30 block at the foot.
("sources (block list, off-allowlist keys)",
("sources (block list of mappings)",
"sources:\n - uri: https://e.com/a\n kind: doc\n"),
# The plain-scalar flow sequence is admitted as of 1.4.0 (P1); this row now
# measures the QUOTED element, which stays refused for the same reason it
# always did -- reading it would mean stripping a quote this parser retains.
("flow sequence, quoted elements", "tags: ['a', 'b']\n"),
("flow sequence", "tags: [a, b, c]\n"),
("flow mapping", "executor: {resource: skills/run.md}\n"),
]
@ -579,8 +569,6 @@ _V02_ADMITTED = [
("status/stale_after", "status: active\nstale_after: 2026-12-01\n"),
("verified bool", "verified: true\n"),
("block sequence of scalars", "tags:\n - alpha\n - beta\n"),
# SPEC §4.1's own skeleton for `tags`, admitted as of 1.4.0 (P1).
("flow sequence of scalars", "tags: [alpha, beta]\n"),
]
@ -598,8 +586,6 @@ def test_v02_flat_frontmatter_still_parses(cid, fm):
_DEGRADED_TO_STRING = [
# (id, frontmatter, what PyYAML 6.0.3 makes of it)
# Still red after G30 opened the block-mapping carrier: `uri` is not a §5.1
# key, so the item is refused by the allowlist instead of by the grammar.
("one key per item", "sources:\n - uri: https://e.com/a\n", "[{'uri': ...}]"),
("item, trailing colon", "sources:\n - uri:\n", "[{'uri': None}]"),
("inline double colon", "attester: resource: attesters/sql_equality.py\n", "parse error"),
@ -667,16 +653,13 @@ def test_pointer_in_a_degraded_mapping_no_longer_reaches_the_consumer_tree(cid,
assert result.disposition is Disposition.FAIL_SECURE, "hole reopened — see LIMITATIONS.md"
def test_the_expressible_mapping_routes_are_the_ones_the_allowlist_inspects():
# Was `test_exactly_one_route_to_a_mapping_is_expressible` (1.2.0), and before
# that ALL FOUR routes failed so the mapping *class* had no expressible form.
# There are four expressible carriers as of 1.3.0 - flow mapping as a value,
# flow mapping as a list item, flow sequence of flow mappings, block sequence
# of block mappings - and the criterion that admits them is unchanged: each
# hands the allowlist every key. The routes below stay shut for the same
# reason, each on its own rule: a top-level block MAPPING (not a sequence),
# a dotted key, and an inline second colon give the allowlist nothing to
# inspect.
def test_exactly_one_route_to_a_mapping_is_expressible():
# Was: ALL FOUR routes failed, each on its own rule, so the mapping *class* had
# no expressible form (and v0.2's `generated` could not be written at all). G3
# opens exactly ONE of them - the allowlisted flow form - and the other three
# still fail, each on its own rule. That the openable route is the one whose
# every key the allowlist inspects is the whole design: block, dotted and inline
# give the allowlist nothing to inspect, so they stay shut.
assert parse_frontmatter("---\nid: x\ngenerated: { by: x, at: y }\n---\n\nbody\n")[0][
"generated"] == {"by": "x", "at": "y"}
@ -712,35 +695,19 @@ def test_block_lists_admitted_by_item_shape(cid, fm, expected):
assert parse_frontmatter(f"---\nid: x\n{fm}---\n\nbody\n")[0][key] == expected
def test_the_block_list_rejects_on_the_key_set_not_on_arity():
# Was `test_two_keys_per_item_is_where_the_block_list_hard_rejects`: a
# two-key item was refused because the block mapping had no expressible form
# at all. G30 gives it one (SPEC.md §5.1's own carrier), so arity is no
# longer the boundary - the key set is. The same two-key item parses when its
# keys are §5.1's, and still raises when one of them is not.
parsed = parse_frontmatter(
"---\nid: x\nsources:\n - id: a\n resource: file://x\n---\n\nbody\n"
)[0]
assert parsed["sources"] == [{"id": "a", "resource": "file://x"}]
def test_two_keys_per_item_is_where_the_block_list_hard_rejects():
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(
"---\nid: x\nsources:\n - id: a\n uri: file://x\n---\n\nbody\n"
"---\nid: x\nsources:\n - id: a\n resource: file://x\n---\n\nbody\n"
)
@pytest.mark.parametrize("fm", [
# The flow row carries a key OFF the G3 allowlist: the shape is admitted, the
# key is not, so this stays a T2 rejection and the door A/B half still holds.
# The two `sources` rows that lived here until 1.3.0 now PARSE - that is the
# G30 fix, not a weakening of this property. Their replacements are the same
# carriers with a key off the allowlist, so the row still measures what it
# says: the shape is admitted, the key set is not.
# The `tags` row moved the same way in 1.4.0: `[a, b]` now PARSES (P1), so
# the row carries the quoted form, which is still a T2 rejection.
"generated: { by: x, tool: y }\n", "sources: [{ id: a, uri: u }]\n", "tags: ['a', 'b']\n",
"generated: { by: x, tool: y }\n", "sources: [{ id: a }]\n", "tags: [a, b]\n",
"generated:\n by: x\n", "generated.by: x\n",
"sources:\n - id: a\n uri: file://x\n",
"executor: [{ resource: skills/run.md }]\n",
"sources:\n - id: a\n resource: file://x\n",
])
def test_t2_constrains_import_not_emission(fm):
# T2 runs on door C only. The same frontmatter that FAIL_SECUREs through
@ -936,331 +903,3 @@ def test_a_conformant_v02_trust_layer_now_reaches_the_gate():
result = import_bundle({"tables/users.md": doc})
assert result.disposition is Disposition.WARN
assert result.concepts[0].error is None
# --- G30: the `sources` provenance layer becomes reachable (2026-09-02) ------
# Door 2. G3 gave the mapping *class* one expressible form but left `sources`
# unreachable: SPEC.md §5.1 writes an entry as a MAPPING carrying a REQUIRED
# `resource`, so neither of the two carriers the spec and the producers actually
# use could parse. Measured 02.09 by two consumers independently -- a flow
# sequence of flow mappings raised on the `[` indicator, a block sequence of
# block mappings raised "nested mappings are not supported".
#
# Why `resource` is admissible now when 1.2.0 argued it was not: the old
# argument was that the parser could not tell `sources[].resource` (§5.1, a
# citation) from `executor.resource` (§10, a code pointer). That premise was
# false -- the owning key is in scope at every call site, it was simply never
# threaded through. `resource` is allowlisted for `sources` entries ONLY, so
# the door-C routes above stay shut on the same input.
_SPEC_51_BLOCK = (
"sources:\n"
" - id: ga4-schema\n"
" resource: https://developers.google.com/analytics/bigquery/export-schema\n"
" title: GA4 BigQuery Export schema\n"
" author: team:ga4-docs\n"
" usage_count: 5000\n"
" last_modified: 2026-05-30T00:00:00Z\n"
)
def test_spec_sources_block_sequence_of_block_mappings_parses():
# SPEC.md §5.1's own example block, verbatim. It is the spec's canonical
# carrier for a REQUIRED field, so §11.1 ("parseable YAML frontmatter") makes
# a bundle written this way conformant -- refusing it refuses a conformant
# bundle, which is the failure mode G3 was opened to end.
fm, _ = parse_frontmatter(f"---\ntype: table\n{_SPEC_51_BLOCK}---\n\nbody\n")
assert fm["sources"] == [{
"id": "ga4-schema",
"resource": "https://developers.google.com/analytics/bigquery/export-schema",
"title": "GA4 BigQuery Export schema",
"author": "team:ga4-docs",
"usage_count": "5000",
"last_modified": "2026-05-30T00:00:00Z",
}]
@pytest.mark.parametrize("cid,fm,expected", [
("one entry",
"sources: [{ id: golden-v0-2-sales, resource: fixture }]\n",
[{"id": "golden-v0-2-sales", "resource": "fixture"}]),
("two entries",
"sources: [{ id: a, resource: https://e.com/a }, { id: b, resource: https://e.com/b }]\n",
[{"id": "a", "resource": "https://e.com/a"},
{"id": "b", "resource": "https://e.com/b"}]),
])
def test_sources_flow_sequence_of_flow_mappings_parses(cid, fm, expected):
# The form the producer emits today (llm-ingestion-okf's golden
# expected-bundle/ingest-sales.md, measured 02.09). One entry raised too, so
# this is not an arity bug: the `[` indicator refused the carrier outright.
assert parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")[0]["sources"] == expected
def test_the_two_sources_carriers_parse_to_the_same_value():
block = parse_frontmatter(
"---\ntype: t\nsources:\n - id: a\n resource: https://e.com/a\n---\n\nb\n")[0]
flow = parse_frontmatter(
"---\ntype: t\nsources: [{ id: a, resource: https://e.com/a }]\n---\n\nb\n")[0]
assert block["sources"] == flow["sources"] == [{"id": "a", "resource": "https://e.com/a"}]
@pytest.mark.parametrize("cid,fm", [
("unknown key, flow", "sources: [{ id: a, uri: https://e.com/a }]\n"),
("unknown key, block", "sources:\n - id: a\n uri: https://e.com/a\n"),
("unknown key, flow value", "sources: { id: a, kind: doc }\n"),
])
def test_an_unknown_key_in_a_sources_entry_is_still_rejected(cid, fm):
# The negative control. The carrier is admitted; the key set is not. A
# producer-invented key gets no free ride on the new shape.
with pytest.raises(OKFFrontmatterError) as exc:
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
assert "allowlist" in str(exc.value)
@pytest.mark.parametrize("cid,fm", [
("executor, block sequence", "executor:\n - resource: skills/run-on-bq.md\n"),
("executor, flow sequence", "executor: [{ resource: skills/run-on-bq.md }]\n"),
("attester, flow sequence", "attester: [{ resource: attesters/sql_equality.py }]\n"),
("attester, block sequence", "attester:\n - resource: attesters/sql_equality.py\n"),
])
def test_resource_is_allowlisted_for_sources_entries_only(cid, fm):
# The whole reason `resource` can be admitted at all: the owning key decides.
# §10's `executor.resource` / `attester.resource` name run instructions and
# code -- door C -- and stay refused through EVERY carrier, including the two
# this change opens for `sources`.
doc = f"---\nid: x\ntype: Attested Computation\n{fm}---\n\nbody\n"
with pytest.raises(OKFFrontmatterError) as exc:
parse_frontmatter(doc)
assert "allowlist" in str(exc.value)
assert import_bundle({"computations/x.md": doc}).disposition is Disposition.FAIL_SECURE
@pytest.mark.parametrize("cid,fm", [
# SPEC §5.1: "A single entry MAY carry its own `usage_window`". That is a
# mapping inside a mapping -- depth 2 -- and stays refused: no nesting deeper
# than one is a security property this change does not spend. Registered as a
# conformance gap in docs/LIMITATIONS.md, not as an oversight.
("per-entry usage_window, flow",
"sources: [{ id: a, usage_window: { from: x, to: y } }]\n"),
("per-entry usage_window, block",
"sources:\n - id: a\n usage_window: { from: x, to: y }\n"),
# A sequence inside an entry is the same depth violation.
("nested sequence", "sources: [{ id: a, title: [x, y] }]\n"),
])
def test_a_sources_entry_admits_scalar_leaves_only(cid, fm):
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
@pytest.mark.parametrize("cid,fm", [
("scalar then mapping", "sources:\n - https://e.com/a\n - id: b\n title: B\n"),
("mapping then scalar", "sources:\n - id: a\n title: A\n - https://e.com/b\n"),
])
def test_a_block_list_may_not_mix_scalars_and_mappings(cid, fm):
# A consumer that reads `entry.get("id")` over the list crashes on the str.
# One list, one item type -- refuse rather than hand back a mixed tree.
with pytest.raises(OKFFrontmatterError) as exc:
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
assert "mix" in str(exc.value)
@pytest.mark.parametrize("cid,fm", [
("empty flow sequence", "sources: []\n"),
("flow sequence, unclosed", "sources: [{ id: a }\n"),
("flow sequence, trailing junk", "sources: [{ id: a }] x\n"),
("flow sequence, nested sequence", "sources: [[ id ]]\n"),
])
def test_the_flow_sequence_refuses_a_carrier_it_cannot_read(cid, fm):
# What no element shape rescues: nothing at all, an unclosed mapping, junk
# between elements, and a sequence inside a sequence. The plain-scalar
# element admitted in 1.4.0 (P1) is a THIRD shape, not a loosening of these
# -- each row here still raises through it.
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
# --- P1: the flow sequence of plain scalars (operator decision, 2026-09-08) --
# `tags: [a, b, c]` is the ONE form SPEC.md §4.1's own skeleton writes out for
# `tags`, and the only candidate measured to move the upstream corpus at all:
# 0/53 -> 6/53 against the pinned reference bundles (`_okf-upstream` @ 3fcbb9f,
# denominator 53). The two neighbours measured 0/53 EACH and are deliberately
# NOT built -- P2 (a flush-left block sequence) and P3 (a folded plain scalar)
# would cost parser surface and buy no measured conformance on this corpus;
# see docs/2026-09-07-limitations-44-maaling.md §§7-8.
#
# It does not spend the depth-1 rule: the elements are leaves, the same depth
# the block list of scalars already carries. The binding constraint on the
# corpus is elsewhere and stays refused -- 44/53 stop on `generated` written as
# a top-level block mapping, which is a security decision, not this one.
@pytest.mark.parametrize("cid,fm,expected", [
("SPEC §4.1 skeleton", "tags: [finance, revenue, headline-metric]\n",
["finance", "revenue", "headline-metric"]),
("two elements", "tags: [a, b]\n", ["a", "b"]),
("one element", "tags: [solo]\n", ["solo"]),
("uneven spacing", "tags: [ a ,b ]\n", ["a", "b"]),
("one trailing comma, as the flow mapping already allows", "tags: [a, b,]\n", ["a", "b"]),
])
def test_a_flow_sequence_of_plain_scalars_parses(cid, fm, expected):
# The red test for P1: `[a, b]` must come back as a real list of strings,
# never a degraded string (the 1.1.0 defect) and never a refusal.
assert parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")[0]["tags"] == expected
def test_the_two_scalar_sequence_carriers_parse_to_the_same_value():
flow = parse_frontmatter("---\ntype: t\ntags: [a, b]\n---\n\nb\n")[0]
block = parse_frontmatter("---\ntype: t\ntags:\n - a\n - b\n---\n\nb\n")[0]
assert flow["tags"] == block["tags"] == ["a", "b"]
@pytest.mark.parametrize("cid,fm", [
# Quoting is the failure mode this shape is refused for elsewhere: the
# parser retains quotes rather than stripping them, so admitting a quoted
# element would hand back a value YAML reads differently.
("single-quoted element", "tags: ['a']\n"),
("double-quoted element", 'tags: ["a", "b"]\n'),
# Comma splitting is character-level, so anything that would need YAML
# semantics to split correctly is refused rather than guessed at.
("colon inside an element", "tags: [a: b]\n"),
("comment indicator", "tags: [a #b]\n"),
("brace inside an element", "tags: [a{b}]\n"),
("bracket inside an element", "tags: [a[b]]\n"),
# Depth: a sequence inside a sequence opens a second level and is refused
# on the same character rule, with no YAML semantics involved.
("nested flow sequence", "tags: [[a]]\n"),
("empty element", "tags: [a, , b]\n"),
("leading empty element", "tags: [, a]\n"),
("empty sequence", "tags: []\n"),
("unterminated sequence", "tags: [a, b\n"),
# The unchanged scalar indicators still apply to an element, exactly as they
# do to a block-list item: an anchor or an alias is not a plain scalar.
("anchor element", "tags: [&anchor]\n"),
("alias element", "tags: [*alias]\n"),
("explicit tag element", "tags: [!!python/object]\n"),
])
def test_a_flow_sequence_scalar_element_must_be_a_plain_scalar(cid, fm):
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
@pytest.mark.parametrize("cid,fm", [
("scalar then mapping", "tags: [a, {b: c}]\n"),
("mapping then scalar", "sources: [{ id: a }, plain]\n"),
])
def test_a_flow_sequence_may_not_mix_scalars_and_mappings(cid, fm):
# Same rule, same reason as the block list: a consumer iterating the value
# and reading `entry.get("id")` crashes on the first str. The mixing verdict
# is reached BEFORE the element is parsed, so it is what the caller sees --
# not an allowlist complaint about a key that was never the problem.
with pytest.raises(OKFFrontmatterError) as exc:
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
assert "mix" in str(exc.value)
@pytest.mark.parametrize("cid,fm,expected", [
("one entry", "sources: [{ id: x }]\n", [{"id": "x"}]),
("two entries", "sources: [{ id: a, title: A }, { id: b, title: B }]\n",
[{"id": "a", "title": "A"}, {"id": "b", "title": "B"}]),
])
def test_the_g30_flow_mapping_carrier_is_unchanged_by_the_scalar_element(cid, fm, expected):
# The no-regression pin. P1 adds a branch to the same function that carries
# `sources`; the mapping element must parse exactly as it did in 1.3.0.
assert parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")[0]["sources"] == expected
# The self-safety row for P1 (OWASP LLM10). The predicate is character-level
# with no regex, so there is no backtracking engine to blow up -- but "no regex"
# is an argument, and the sweep in docs/redos-sweep.py cannot check it, because
# it collects compiled patterns and this predicate compiles none. So it is
# measured instead, on the same CPU clock every other bound here uses. Over four
# doublings (50k -> 800k, 2026-09-08) the exponent is 0.86-0.99 in the element
# LENGTH and 0.97-1.05 in the element COUNT: linear in both, 0.66s at 800_000
# elements and 0.03s at an 800_000-character element. The refusal path is the
# same shape -- a forbidden character at the very end of an 800_000-character
# element is found in 0.04s -- because the element is scanned once per rule, not
# rescanned per start position.
_FLOW_SCALAR_PERF_N = 400_000
def test_a_long_flow_sequence_of_scalars_stays_bounded():
many = "---\ntype: t\ntags: [" + ", ".join(["ab"] * _FLOW_SCALAR_PERF_N) + "]\n---\nbody\n"
assert scan_seconds(parse_frontmatter, many) < 2.0
one_long = "---\ntype: t\ntags: [" + "a" * _FLOW_SCALAR_PERF_N + "]\n---\nbody\n"
assert scan_seconds(parse_frontmatter, one_long) < 2.0
def test_a_long_flow_sequence_element_is_refused_without_a_rescan():
# The refusal is the half an attacker controls: a forbidden character parked
# at the END of a long element is the worst case for any per-start rescan.
doc = "---\ntype: t\ntags: [" + "a" * _FLOW_SCALAR_PERF_N + ":]\n---\nbody\n"
def refuse(payload):
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(payload)
assert scan_seconds(refuse, doc) < 2.0
def test_injection_in_a_sources_entry_leaf_is_caught_by_the_scan():
# T1 over the new shape: every leaf of every entry reaches scan_output.
doc = f"---\ntype: table\nsources: [{{ id: a, title: {_INJECTION} }}]\n---\nclean\n"
assert scan_concept(doc).found is True
doc_block = f"---\ntype: table\nsources:\n - id: a\n title: {_INJECTION}\n---\nclean\n"
assert scan_concept(doc_block).found is True
def test_the_producer_golden_now_passes_the_gate():
# llm-ingestion-okf's expected-bundle/ingest-sales.md, the K5 blocker.
doc = (
"---\n"
"type: dataset\n"
"title: Regional Sales\n"
"source_system: golden-v0-2-sales\n"
"ingested_at: 2026-07-16T12:00:00Z\n"
"generated: { by: process:okf-ingest, at: 2026-07-16T12:00:00Z }\n"
"sources: [{ id: golden-v0-2-sales, resource: fixture }]\n"
"---\n\n| region | units |\n| --- | --- |\n| nord | 412 |\n"
)
result = import_bundle({"datasets/sales.md": doc})
assert result.disposition is Disposition.WARN
assert result.concepts[0].error is None
@pytest.mark.parametrize("cid,fm", [
# A `- ` item whose text begins with a YAML indicator AND carries a `": "`
# must fall through to the scalar predicates, not into the block-mapping
# route with a half-validated key. Each of these fails `_KEY_RE` on the key
# side, so `_reject_dangerous_value` gets the item intact.
("anchor", "sources:\n - &anchor id: a\n"),
("alias", "sources:\n - *anchor id: a\n"),
("tag", "sources:\n - !!str id: a\n"),
("directive", "sources:\n - %YAML id: a\n"),
("reserved", "sources:\n - `x id: a\n"),
("merge key", "sources:\n - id: a\n <<: *base\n"),
])
def test_an_indicator_in_a_block_item_is_refused_before_the_mapping_route(cid, fm):
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(f"---\ntype: t\n{fm}---\n\nbody\n")
@pytest.mark.parametrize("cid,fm", [
# The block mapping ends at a blank line and at a line in column zero. Both
# hand control back with an index that must not skip or re-read a line: a
# dangling `resource:` line left over from a mapping that closed early must
# RAISE, never be silently dropped -- a pointer that vanishes rather than
# failing is exactly this repo's failure class.
("blank line inside the mapping", "sources:\n - id: a\n\n resource: b\n"),
("top-level key interleaved", "sources:\n - id: a\ntags: x\n resource: b\n"),
])
def test_a_line_orphaned_by_the_mapping_boundary_raises_rather_than_vanishing(cid, fm):
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(f"---\ntype: t\n{fm}---\n\nbody\n")
def test_the_block_list_hands_back_an_index_that_resumes_at_the_next_key():
# The return-index contract: a top-level key following a multi-entry block
# list is neither swallowed by the list nor re-read as a list item.
fm, _ = parse_frontmatter(
"---\ntype: t\nsources:\n - id: a\n resource: b\n - id: c\ntitle: T\n---\n\nbody\n")
assert fm == {"type": "t", "sources": [{"id": "a", "resource": "b"}, {"id": "c"}],
"title": "T"}