1
0
Fork 0

Compare commits

...

6 commits

Author SHA1 Message Date
d19de8cb38 chore(release): 1.4.0 2026-09-08 06:10:53 +02:00
3e324a1f86 feat(okf): a flow sequence of plain scalars parses, and the corpus number is 6/53
`tags: [a, b, c]` is the form SPEC.md 4.1's own frontmatter skeleton writes out,
and 9/53 upstream reference concepts use it. It raised on the `[` indicator.
It parses now, to the same list its block-sequence sibling already produced.

The predicate is character-level, inside `_parse_flow_sequence`: an element is a
plain scalar only if it is non-empty and carries none of `{ } [ ] : , " ' #`,
and it then passes the unchanged scalar-indicator rule. Everything that would
need YAML semantics to split or unquote still raises - a quoted element (quotes
are retained here, never stripped), a colon or comma inside an element, a
sequence inside a sequence, an empty element, an anchor, an alias. A sequence
may not mix scalars and mappings, the rule the block list already carries, and
the mixing verdict is reached before the element is parsed so the caller is told
about the mix rather than about a key the allowlist would have named instead.
The 1.3.0 `sources` flow-mapping carrier is unchanged and pinned against
regression. Depth 1 is not spent: the elements are leaves.

Measured with the denominator, against the pinned corpus (`_okf-upstream` @
3fcbb9f, 53 non-reserved documents) and the pinned SPEC (`_okf-canonical` @
ad30107). Baseline reproduced first, with a known-positive control, at 0/53.
After: 6/53, all six in acme_retail. It does not close the corpus - 44/53 still
stop on `generated` written as a top-level block mapping, which spends the
no-nesting-past-depth-1 rule and is a security decision, out of scope here.

P1 alone, per the operator decision of 08.09. The two neighbouring predicates
were measured and deliberately not built: a flush-left block sequence and a
folded plain scalar release 0/53 each on their own, and stacked on this one they
still measure 6/53. `_consume_block_list`, the `description` continuation and
the allowlist are untouched.

docs/LIMITATIONS.md's tags/description entry is rewritten against the
measurement: three of its claims were wrong. The parser does have a
sequence-value type (since 1.3.0 - what it lacks is the indentation the corpus
omits); the figure is 6/53, not the 1.2.0-era 4/53; and tags/description are
not the residual that blocks the corpus. README gains the sequence carrier in
the paragraph that already describes the mapping one.

Self-safety: the predicate compiles no regex, so docs/redos-sweep.py cannot see
it. Measured instead on the CPU clock - linear in element length (exponent
0.86-0.99) and in element count (0.97-1.05) over four doublings to 800_000 -
and pinned by two bounds in tests/test_okf.py.

Six rows that pinned the old refusal are re-aimed at the class that still
holds - the quoted element - the way the 1.3.0 rows were when the carrier
opened. One of them lives in src/llm_ingestion_guard/coverage.py, which is why
the src diff is three files rather than one.

Version 1.4.0 in the code only. The CHANGELOG entry stays under Unreleased and
no tag is cut: README's badge and install pin must keep naming a tag that
exists.

Gates after `git add`: 893 passed (was 868), coverage 130/130 + 6/6 gaps,
redos-sweep exit 0, LIMITATIONS still 45 entries.
2026-09-08 05:34:55 +02:00
6e7c8d2b98 docs(okf): measure the tags/description gap against the pinned SPEC and corpus
Order 20260906T213322Z: measure and propose, no src, no bump. Measured against
SPEC _okf-canonical @ ad30107 and the corpus _okf-upstream @ 3fcbb9f (denominator
53, extracted at the pin because the work tree had moved on to 9a15b13).

Three claims in the tags/description entry are stale against the 1.3.0 parser.
The parser does have a sequence type -- _consume_block_list parses block lists of
scalars -- what it lacks is tolerance for the flush-left indentation 36/53 of the
corpus uses. Removing tags alone now lets 6/53 pass, not the 4/53 recorded, and
the entry's headline claim does not hold at all: description alone unblocks 0/53,
and with all three surface forms closed 44/53 still stop on generated as a
top-level block mapping. The entry oversells its own reach by a factor of seven.

The SPEC has no depth rule. no-nesting-past-depth-1 is entirely ours; the
conformance floor is only "a parseable YAML frontmatter block" (SS11.1).

Candidates were measured by normalizing the surface form onto a shape the parser
already accepts, then importing parse_frontmatter -- the predicate is never
re-implemented, only the input's spelling is rewritten. P1 (scalar flow sequence)
takes the corpus from 0/53 to 6/53 without spending depth-1; P2 and P3 buy 0/53
each and 6/53 stacked on P1. Recommendation is P1 alone, and the note says out
loud that this does not open the corpus.

The label "punkt 44" is not grep-able: it came from a count-after-insertion, and
the entry is today the 12th of 45 at docs/LIMITATIONS.md:126.

No src change, no version bump, no push. 868 passed, coverage 130/130 + 6/6 gaps,
redos sweep exit 0 -- all after git add.
2026-09-06 23:58:02 +02:00
44e2b31afd test(okf): pin the two boundary classes the new carriers introduced
Probed after 1.3.0 landed, on advisor challenge. Both came back clean --
these pin them so they cannot regress silently.

1. A `- ` item whose text begins with a YAML indicator AND carries a `": "`
   could in principle route into the block-mapping path with a
   half-validated key instead of into `_reject_dangerous_value`. It does
   not: `&anchor id` fails `_KEY_RE` on the key side, so the item falls
   through intact and raises on the indicator. One row each for & * ! %
   backtick, plus the merge key inside an entry.

2. `_consume_block_mapping` ends at a blank line and at column zero, and
   hands an index back to `_consume_block_list`, which hands one back to
   `_parse_flat`. A line orphaned by that boundary -- a `resource:` left
   over from a mapping that closed early -- must RAISE, not be dropped. A
   pointer that vanishes rather than failing is this repo's failure class,
   so it gets a test rather than a probe. Plus the return-index contract:
   a top-level key after a multi-entry list is neither swallowed nor
   re-read as an item.

868 green (was 859), 130/130 classes. Test-only; no source change.
2026-09-02 17:26:08 +02:00
a965e8ac5b feat(okf): sources becomes expressible, and the parent key is what admits resource
Order 20260902T150716Z from .claude -- a K5 blocker in the OKF programme.
`parse_frontmatter` rejected `sources` in every form the spec and its
producers actually use. Measured 02.09 by two consumers independently:
`sources: [{ id: a, resource: x }]` raised on the `[` indicator (one entry
as well as two), and the block sequence of block mappings -- SPEC.md 5.1's
OWN example -- raised "nested mappings are not supported". `resource` is
REQUIRED within a `sources` entry, so the whole provenance family was
unwritable and a bundle written the way the spec documents it was refused.

Measured against the spec before coding, not reasoned: 5.1's example block
is the canonical carrier for a REQUIRED field and 11.1 defines conformance
as parseable frontmatter, so refusing it refuses a conformant bundle. Both
carriers now parse to the same list of dicts.

The load-bearing change is not the carrier, it is WHO admits `resource`.
1.2.0 left it off the allowlist arguing the parser could not tell
`sources[].resource` (5.1, a citation) from `executor.resource` /
`attester.resource` (10, a pointer to code to be run -- the door-C route
closed in 1.1.0). That premise was false: the owning key is in scope at
every call site and was simply never threaded through. It is threaded now,
so the discrimination is structural, and door C stays shut through EVERY
carrier including the two this adds -- pinned by a new test that drives
`executor`/`attester` through all four.

Refusal stays the default elsewhere. A flow sequence of plain scalars
(`tags: [a, b]`) still raises: the sequence carrier is opened for the flow
mapping element and nothing else. A `sources` entry admits scalar leaves
only, so 5.1's optional PER-ENTRY `usage_window` is refused -- no nesting
past depth 1 is a security property and it was not spent here; registered
as a conformance gap rather than left as an oversight. A block list may not
mix scalars and mappings, because a consumer reading `entry.get("id")` over
one gets an AttributeError off the first str.

New residual registered: `sources[].resource` is scanned as text (T1) but
never URL-validated. T3's https allowlist cannot reach it without
over-blocking conformant bundles -- 5.1 permits bundle-relative paths and
scope descriptors, and the producers' own golden emits `resource: fixture`.
A consumer that dereferences it must call `validate_resource_url` itself.

Suite 834 -> 859 green. 25 new rows; four pre-existing rows changed because
this release changed the behaviour they pinned, two of them renamed since
their names asserted the old invariant (`exactly_one_route_to_a_mapping`,
`two_keys_per_item_is_where_the_block_list_hard_rejects`). Not "unchanged".
130/130 classes, 6/6 gaps hold, 44 -> 45 limitations, ReDoS 0/152 (the
sweep adds no evidence here -- this change adds no regex and the splitting
is linear). Six version surfaces bumped by hand, no sed. Re-measured alone
after the bump.

No exported surface changed; no detector behaviour and no calibration
changed.
2026-09-02 17:21:26 +02:00
0184df9ed9 docs(limits): tags and description block the whole OKF corpus before sources is even read
Measured 08-23 (order 20260823T161935Z): the line-flat frontmatter parser has
no sequence-value type at all, so tags rejects 53/53 upstream concepts
regardless of flow or block form, and description's folded-scalar continuation
misreads as a nested mapping in 29/53. Independent of both the mapping-form
gap and the sources block-form gap already documented here -- closing either
moves nothing on this corpus. 43 -> 44 items; README count moved with it. No
code change, no release.
2026-08-25 08:27:22 +02:00
16 changed files with 1477 additions and 108 deletions

3
.gitignore vendored
View file

@ -24,3 +24,6 @@ coverage/
*.local.md
*.local.sh
.DS_Store
# --- scratchpad: measurement scripts are tracked, the extracted corpus copy is not ---
/scratchpad/corpus/

View file

@ -5,6 +5,105 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [1.4.0] — 2026-09-08
### Added — `tags: [a, b, c]`, the one flow-sequence form SPEC §4.1 writes out
`okf.parse_frontmatter` refused a flow sequence of plain scalars. That is the
form SPEC.md §4.1's own frontmatter skeleton uses for `tags`, and `tags` is
present in 53/53 upstream reference concepts, 9/53 of them in exactly this form.
Admitted now, parsing to the same value its block-sequence sibling already did:
```yaml
tags: [finance, revenue, headline-metric]
```
An element is a plain scalar only if it is non-empty and carries none of
`{ } [ ] : , " ' #`, and it then passes the unchanged scalar-indicator rule.
The refusals that make that a rule rather than a preference: a quoted element
(this parser retains quotes rather than stripping them, so reading one would
hand back a value YAML does not), an element carrying a colon or a comma, a
sequence inside a sequence, an empty element, an anchor or an alias. A sequence
may not **mix** scalars and mappings — the same rule the block list already
carries, for the same reason: a consumer iterating the value and reading
`entry.get("id")` crashes on the first `str`. The `sources` flow-mapping
carrier added in 1.3.0 is unchanged and pinned against regression.
**Measured, with the denominator: this takes the pinned OKF corpus from 0/53 to
6/53** (`_okf-upstream/okf` @ `3fcbb9f`, 53 non-reserved documents; the six are
all in `acme_retail`). It does not close the corpus. **44/53 still stop on
`generated` written as a top-level block mapping** — a form refused on the
no-nesting-past-depth-1 rule, which is a security decision and out of scope
here. Two neighbouring predicates were measured and deliberately not built: a
flush-left block sequence and a folded plain scalar release **0/53** each on
their own and leave the corpus at 6/53 when stacked on this one. Detection
behaviour is not frozen under semver; the exported surface is unchanged.
### Changed
- `docs/LIMITATIONS.md`'s `tags`/`description` entry is rewritten against the
measurement. Three of its claims were wrong: the parser *does* have a
sequence-value type (since 1.3.0 — what it lacks is the indentation the
corpus omits), the figure is 6/53 and not the 1.2.0-era 4/53, and
`tags`/`description` are *not* the residual that blocks the corpus.
- Self-safety: the new predicate is character-level with no regex, so
`docs/redos-sweep.py` cannot see it (it collects compiled patterns). It is
measured instead — linear in both element length (exponent 0.86-0.99) and
element count (0.97-1.05) over four doublings to 800_000, and pinned by two
CPU-clock bounds in `tests/test_okf.py`.
## [1.3.0] — 2026-09-02
### Added — the `sources` provenance family becomes expressible in both spec carriers
`okf.parse_frontmatter` rejected `sources` in every form the OKF spec and its
producers actually use. Measured 2026-09-02 by two consumers independently:
`sources: [{ id: a, resource: x }]` raised on the `[` indicator (with one entry
as with two), and the block sequence of block mappings — SPEC.md §5.1's *own*
example — raised `"nested mappings are not supported"`. `resource` is REQUIRED
within a `sources` entry (§5.1), so the whole provenance family was unwritable,
and a bundle written the way the spec documents it was refused.
Admitted now, both parsing to the same value (a list of dicts):
```yaml
sources: [{ id: a, resource: https://e.com/a }]
sources:
- id: a
resource: https://e.com/a
```
**`resource` is allowlisted inside a `sources` entry and nowhere else.** 1.2.0
left it off the allowlist on the argument that the parser could not tell
`sources[].resource` (§5.1, a citation) from `executor.resource` /
`attester.resource` (§10, a pointer to code to be run — the door-C route closed
in 1.1.0). That premise was false: the owning key is in scope at every call
site and was simply never threaded through. It is threaded now, so the
discrimination is structural rather than a judgement about the value, and
`executor: [{ resource: skills/run.md }]` and `attester:\n - resource: …` are
refused on the allowlist through *every* carrier, including the two this adds.
Refusal stays the default everywhere else. A flow sequence of plain scalars
(`tags: [a, b]`) still raises — the sequence carrier is opened for the flow
mapping element and nothing else. A `sources` entry still admits scalar leaves
only, so SPEC §5.1's optional *per-entry* `usage_window` (a mapping inside a
mapping) is refused: no nesting past depth 1 is a security property, and it was
not spent here. Registered as a conformance gap in `docs/LIMITATIONS.md`. A
block list may not mix scalar items and mappings. Off-allowlist keys, anchors,
aliases, tags, duplicate keys and unclosed collections raise as before, and a
refused mapping still raises rather than degrading into a string.
**New residual, registered:** `sources[].resource` is scanned as text (T1) but
never validated as a URL. T3's https allowlist cannot be extended to it without
over-blocking conformant bundles — §5.1 permits a bundle-relative path or a
scope descriptor a consumer cannot follow at all. A consumer that dereferences
it must call `okf.validate_resource_url` itself.
No exported surface changed and no detector behaviour or calibration changed.
Suite 834 → 868 (34 new rows, plus four pre-existing rows updated where this
release changed the behaviour they pinned); 130/130 classes, 6/6 documented
gaps hold, 45 limitations, ReDoS sweep 0/152 candidates flagged.
## [1.2.0] — 2026-08-23
### Added — OKF frontmatter can express one mapping form: typed and allowlisted

View file

@ -11,18 +11,40 @@ framework-agnostisk kode.
Referanse-implementasjon: `claude-code-llm-wiki` Stage B (`tools/wiki_ingest/`).
Lexikon-seed: `injection-patterns.mjs` fra `llm-security`-pluginen.
Repoet er på **v1.2.0** — den eksporterte Python-surfacen er frosset under semver
Repoet er på **v1.4.0 i koden, UUTGITT** (`pyproject.toml` + `__init__.py` er
bumpet; README-badge, install-pinnen, ADOPTION-BRIEF og BRIEF står med vilje
igjen på `1.3.0`, som er den siste taggen som FINNES — en install-pin må peke på
en ekte tag). Release-commiten (CHANGELOG-overskrift datert, de fire
dokumentflatene bumpet, tag) er ikke tatt. Den eksporterte Python-surfacen er frosset under semver
(deteksjonsatferd er det IKKE; kalibrering flytter seg i 1.x). Stdlib-kjernen er
bygget og testet (15 moduler +
topp-nivå wiring, showcase + korpus), inkl. OKF-adapter og aktivt-innhold-
detektor (EchoLeak-klassen) i output-gaten. OKF-frontmatterens mapping-klasse
har **én** uttrykkbar form (G3, 21.08): en flow-mapping (`generated: { by: x, at: y }`) — som verdi eller som blokkliste-
element — der HVER nøkkel står på en ni-navns allowlist og hvert blad er en ren
skalar. Formen er trygg fordi allowlisten inspiserer hver nøkkel; det blanke
avslaget var håndhevelsen, ikke poenget. `resource` er bevisst UTE av
allowlisten (peker, ikke etikett — den ene nøkkelen T3 finnes for). Blokk-,
dotted- og inline-kolon-rutene raiser fortsatt, og en avvist mapping raiser —
den degraderer aldri til en streng (1.1.0-defekten). Mode-b `import_bundle` skanner
har **fire** uttrykkbare bærere (G3 21.08, G30 02.09): flow-mapping som verdi
og som blokkliste-element, flow-sekvens av flow-mappinger, og blokk-sekvens av
blokk-mappinger (SPEC §5.1s egen form). HVER nøkkel i alle fire står på
allowlisten og hvert blad er en ren skalar. Formen er trygg fordi allowlisten
inspiserer hver nøkkel; det blanke avslaget var håndhevelsen, ikke poenget.
**`resource` er allowlistet KUN inne i en `sources`-oppføring** — foreldre-
nøkkelen avgjør, så `executor`/`attester` sin `resource` (§10, dør C) avvises
gjennom hver eneste bærer. 1.2.0s begrunnelse for å utelate den (parseren
manglet foreldre-kontekst) var målt feil: konteksten var der, den var bare
aldri tredd gjennom. Topp-nivå blokk-mapping, dotted- og inline-kolon-rutene
raiser fortsatt, en blokkliste kan ikke blande skalarer og mappinger, og en
avvist mapping raiser — den degraderer aldri til en streng (1.1.0-defekten).
**Flow-sekvens av rene skalarer (`tags: [a, b]`) PARSER fra 1.4.0** (P1,
operatørbeslutning 08.09) — SPEC §4.1s eget skjelett. Et element er en ren
skalar kun hvis det er ikke-tomt og uten `{ } [ ] : , " ' #`, og så gjelder den
uendrete indikator-regelen; sitert element, kolon/komma i elementet, sekvens i
sekvens, tomt element, anker og alias raiser fortsatt, og en flow-sekvens kan
ikke blande skalarer og mappinger. **Målt med nevner: 0/53 → 6/53** på pinnet
OKF-korpus (`3fcbb9f`). Den bindende skranken er IKKE tags/description, men
`generated` som topp-nivå blokk-mapping (44/53) — den bruker opp dybde-1 og er
en sikkerhetsbeslutning. P2 (blokksekvens uten innrykk) og P3 (foldet plain
scalar) er MÅLT til 0/53 hver og bevisst IKKE bygget. `sources[].resource`
URL-valideres ALDRI (T3 ser kun topp-nivå `resource`) — §5.1 tillater
bundle-relative stier og scope-beskrivelser, så en https-gate ville over-blokkert
konforme bundles; konsumenten må selv kalle `validate_resource_url`. Mode-b `import_bundle` skanner
reserverte strukturfiler (`index.md`/`log.md`) i mottatte bundles i stedet for å
path-avvise dem; upload-front-end beholder shadow-reject (`allow_reserved=False`).
Output-gatens decode-and-rescan mater dekodet base64-klartekst gjennom BÅDE lexicon

View file

@ -2,7 +2,7 @@
Write-time defensive layer for Python pipelines that persist LLM output: sanitize, fence, tool-less quarantined transform, capability isolation, scan before persist, fail-secure.
![Version](https://img.shields.io/badge/version-1.2.0-blue)
![Version](https://img.shields.io/badge/version-1.3.0-blue)
![Status](https://img.shields.io/badge/status-stable-brightgreen)
![Python](https://img.shields.io/badge/python-3.10%2B-purple)
![License](https://img.shields.io/badge/license-MIT-lightgrey)
@ -33,7 +33,7 @@ at write time, never assumed from the format. Any pipeline ingesting external da
into an agent-read store has this shape; an OKF wiki is its canonical form — which
is why the guard ships a first-class OKF adapter (below).
**Status:** `v1.2.0`. The stdlib-only core — its detector, contract, and
**Status:** `v1.3.0`. The stdlib-only core — its detector, contract, and
OKF-adapter modules plus the top-level wiring — is built and tested, exercised by
an end-to-end showcase and adversarial + false-positive corpora. The exported
Python surface is now frozen under semver: nothing exported is removed, renamed or
@ -58,7 +58,7 @@ are real limitations, stated plainly below; read them.
Not on PyPI. The guard is distributed from its Forgejo origin — pin a release tag:
```bash
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v1.2.0"
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v1.3.0"
```
The `open/` mirror is anonymously readable, so CI needs no deploy key, token, or
@ -171,7 +171,11 @@ mapping — `generated: { by: x, at: y }`, `verified: { … }` bare or listed,
allowlist (`by`, `at`, `from`, `to`, `id`, `title`, `author`, `usage_count`,
`last_modified`) with plain-scalar leaves only. A key off that list, a nested
collection or a duplicate key is refused, and `resource` is deliberately not on
it; the block, dotted and inline-colon routes to a mapping still raise. See
it; the block, dotted and inline-colon routes to a mapping still raise. A
*sequence* value has two carriers — the block list, and (as of `1.4.0`) the flow
sequence `tags: [a, b, c]`, which is SPEC §4.1's own skeleton — whose elements
are either all plain scalars or all flow mappings, never a mix. A scalar element
carrying any of `{ } [ ] : , " ' #` is refused rather than guessed at. See
[LIMITATIONS](docs/LIMITATIONS.md) for what that admits and what it still walls
off (a `sources` block list of mappings is still refused); **`resource` https-allowlist** (hard-rejects
`data:`/`javascript:`/`file:` before commit — a reject-gate, not defang);
@ -269,7 +273,7 @@ a green scan means safe content. The highest-impact items:
two of the three corpora are living, so the cells are not rewritten in place.
Method and before/after: [`docs/rawhtml-census.py`](docs/rawhtml-census.py).
**Full list — 43 items, each with the mechanism, plus the out-of-scope boundary:**
**Full list — 45 items, each with the mechanism, plus the out-of-scope boundary:**
[`docs/LIMITATIONS.md`](docs/LIMITATIONS.md). Several carry field measurements from
consumer corpora, including the false positives the URL-shape rule actually produces.

View file

@ -0,0 +1,317 @@
# Beslutningsgrunnlag — LIMITATIONS «punkt 44» (tags/description-gapet)
**Målt:** 2026-09-07, mot `HEAD` = `44e2b31` (v1.3.0, urørt) og pinnet SPEC
`_okf-canonical` @ `ad30107`. **Korpus:** `_okf-upstream` @ `3fcbb9f`.
**Hva dette er:** underlaget for én operatørbeslutning — hvilket predikat, om
noe, skal slippe inn den formen `tags` faktisk har i korpuset.
**Hva dette ikke er:** beslutningen. Ingen fil i `src/` er endret, ingen
versjon er bumpet, ingenting er pushet (0 upushede commits ved øktstart).
---
## 1. Referenten: «punkt 44» er ikke grepbart
Ordren og STATE peker på «LIMITATIONS punkt 44». Målt:
```
$ grep -c '^- \*\*' docs/LIMITATIONS.md
45
$ grep -n '^- \*\*' docs/LIMITATIONS.md | awk -F: 'NR==44 {print $1}'
756 # -> ZWJ-oppføringen, ikke tags/description
```
Tallet stammer fra commit `0184df9` (25.08), som skrev «43 -> 44 items» i
commit-meldingen: **44 var antallet oppføringer etter innsettingen, ikke
oppføringens posisjon.** Oppføringen ble satt inn midt i dokumentet og er i dag
den **12. av 45**, på `docs/LIMITATIONS.md:126`. Etiketten «punkt 44» løser seg
altså ikke opp verken ved posisjon eller ved `grep`, og den vil peke feil igjen
neste gang listen vokser. Referenten er entydig i prosa («tags/description»),
men nummeret bør ikke brukes videre.
## 2. Hva punkt 44 sier i dag, ordrett
`docs/LIMITATIONS.md:126-144`:
> **`tags` and `description` block the OKF import corpus universally, before the
> trust layer is even reached.** The line-flat frontmatter parser has no
> sequence-value type at all: `tags` is present in 53/53 upstream concept
> documents — 9/53 as a flow sequence (`[a, b, c]`, rejected on the `[`
> indicator) and 44/53 as a block sequence (`- a` / `- b`, rejected as
> `"malformed frontmatter line"`) — 100% rejection regardless of form.
> `description` is present in 53/53; 29/53 is a folded plain scalar continuing
> on an indented second line, which the parser has no continuation-line model
> for and misreads as `"nested mappings are not supported"` (the remaining
> 24/53 are single-line and parse fine). Measured directly on the upstream
> reference bundles (`_okf-upstream/okf` @ `3fcbb9f`): removing `tags` alone
> lets 4/53 documents pass; removing both `tags` and `description` together
> (trust layer untouched) lets the same 4/53 pass, and all four then parse
> `generated` correctly as a mapping. **Independent of the mapping-form work
> above:** neither `1.2.0`'s flow mapping nor `1.3.0`'s `sources` carriers move
> anything on this corpus, because `tags`/`description` reject before `sources`
> is ever read. No sequence-value type or continuation-line model exists in the
> stdlib-only parser to close this with.
## 3. SPEC, ordrett fra pinnet kopi (`ad30107`)
**§2 Terminology** (`SPEC.md:81`):
> - **Frontmatter**: A YAML metadata block delimited by `---` at the top of
> a markdown file.
**§4 Concept documents** (`SPEC.md:157-159`):
> 1. A **YAML frontmatter block**, delimited by `---` on its own line at the
> start of the file and a closing `---` on its own line.
**§4.1 Frontmatter**, skjelettet (`SPEC.md:163-173`):
> ```yaml
> ---
> type: <Type name> # REQUIRED
> title: <Optional display name>
> description: <Optional one-line summary>
> resource: <Optional canonical URI for the underlying asset>
> tags: [<tag>, <tag>, ...] # Optional
> ```
**§4.1, de to bærende definisjonene** (`SPEC.md:194-199`):
> - `description`: A single sentence summarizing the concept. Used by
> `index.md` generators, search snippets, and previews.
> - `tags`: A YAML list of short strings for cross-cutting categorization.
**§11 Conformance** (`SPEC.md:738-742`):
> A bundle is **conformant** with OKF v0.2 if:
>
> 1. Every non-reserved `.md` file in the tree contains a parseable YAML
> frontmatter block.
### 3.1 Det spec-en ikke sier
`grep -n -i 'nest\|depth\|indent' SPEC.md` gir **null** treff som uttrykker en
dybde- eller innrykksregel (13 treff, alle i andre betydninger — «flat list of»
i §9 og §13, «indent» ingen). **SPEC-en har ingen dybderegel.** «Ingen nesting
forbi dybde 1» er utelukkende vår egen sikkerhetsegenskap. Konformans-gulvet er
«a parseable YAML frontmatter block» — altså *hele* YAML.
To presiseringer som følger av ordlyden:
- `tags` er definert som «A YAML list» — **ikke** som flow-formen. Skjelettets
`[<tag>, ...]` er ett eksempel, ikke formkravet. Blokkformen er like konform.
- `description` er «A single sentence». En setning brutt over to linjer som
foldet plain scalar er samme setning; spec-en stiller ingen linjekrav.
## 4. Måleoppsett
Korpuset er hentet ut ved pinnen, ikke fra arbeidstreet — `_okf-upstream` står
i dag på `9a15b13`, og `3fcbb9f` er en ekte forgjenger (`git merge-base
--is-ancestor` → 0), med 23 filer endret i `okf/` mellom dem.
```
$ git -C _okf-upstream archive 3fcbb9f okf/bundles | tar -x -C scratchpad/corpus
$ find scratchpad/corpus -name '*.md' ! -name index.md ! -name log.md | wc -l
53
```
**Nevner = 53.** Definisjonen er «hver `.md` under `okf/bundles/` som ikke er et
reservert strukturnavn» — som er nøyaktig §11.1s «every non-reserved `.md` file».
**Positiv kontroll.** Et null-resultat må sjekkes mot et kjent-positivt tilfelle
før det konsumeres. Følgende dokument parser i dag, mot samme `HEAD`:
```yaml
type: Metric
description: Recognized revenue for a period.
tags:
- finance # INNRYKKET blokksekvens
- revenue
generated: { by: reference_agent/gemini-2.5-pro, at: 2026-06-30T14:00:00Z }
sources:
- id: revenue-policy
resource: policies/revenue-recognition.md
```
`tags` blir `['finance', 'revenue']`, `sources` blir en liste av dicter.
Instrumentet avviser altså ikke alt; `tags` som blokksekvens **fungerer allerede
i dag**, forutsatt innrykk.
## 5. Målingen
### 5.1 Formsensus over de 53 (`scratchpad/shapes.py`, exit 0)
| nøkkel | form | antall |
|---|---|---|
| `tags` | blokksekvens **uten innrykk** (`- a` i kolonne 0) | **36/53** |
| `tags` | flow-sekvens `[a, b, c]` | 9/53 |
| `tags` | enkeltlinje-skalar | 8/53 |
| `description` | skalar + innrykket fortsettelseslinje | 29/53 |
| `description` | enkeltlinje-skalar | 24/53 |
| `generated` | **topp-nivå blokk-mapping** (` by:` på neste linje) | **44/53** |
| `generated` | flow-mapping `{ ... }` | 9/53 |
| `sources` | blokksekvens uten innrykk | 44/53 |
| `sources` | blokksekvens med innrykk | 5/53 |
### 5.2 Baseline og kandidater (`scratchpad/candidates.py`, exit 0)
Kandidatene er målt ved å **normalisere overflateformen** inn i en form
parseren allerede godtar, og så importere `parse_frontmatter`. Predikatet er
aldri re-implementert; bare stavemåten på inputen er skrevet om.
| variant | passerer | dominerende residual |
|---|---|---|
| baseline (v1.3.0 som utgitt) | **0/53** | 32× nested mappings, 12× malformed line, 9× flow sequence |
| **P1** skalar-flow-sekvens | **6/53** | 32× nested mappings, 12× malformed line |
| **P2** blokksekvens uten innrykk | 0/53 | 44× nested mappings |
| **P3** foldet plain scalar | 0/53 | 36× malformed line |
| P1+P3 | 6/53 | 36× malformed line |
| P1+P2 | 6/53 | 44× nested mappings |
| **P1+P2+P3** | **6/53** | **44× nested mappings, 3× allowlist** |
De 6 som passerer med P1 er alle i `acme_retail`; de er 6 av de 9 med
flow-sekvens-`tags`, og de tre siste stoppes av allowlisten
(`parameters.name` ×2, `not.term` ×1), ikke av `tags`.
### 5.3 Taket
Med alle tre predikatene er taket **6/53**. Residualet er 44× topp-nivå
blokk-mapping på `generated` og 3× allowlist. En diagnostisk kjøring som også
normaliserte topp-nivå blokk-mapping til flow-mapping traff neste vegg med én
gang: 44× «a quoted scalar inside a flow mapping is not a supported form» —
korpuset skriver `at: '2026-07-10T23:16:06+00:00'` med enkeltfnutter.
## 6. Tre påstander i punkt 44 er målt feil
1. **«The line-flat frontmatter parser has no sequence-value type at all.»**
Usant siden 1.3.0. `_consume_block_list` (`okf.py:662`) parser blokklister av
rene skalarer, og den positive kontrollen i §4 beviser det. Det som mangler
er ikke sekvenstypen, men **innrykkskravet**: `_consume_block_list` krever
`raw[:1] in (" ", "\t")`, og korpuset skriver `- a` i kolonne 0.
Feilklassen «44/53 rejected as malformed frontmatter line» er i dag 12/53,
fordi de øvrige treffer `description`-fortsettelsen først.
2. **«removing `tags` alone lets 4/53 documents pass … removing both … the same
4/53.»** Målt i dag: **6/53** med `tags` fjernet, og **6/53** med begge
fjernet. Tallet 4 var riktig for 1.2.0-parseren; 1.3.0s `sources`-bærere
flyttet to dokumenter til.
3. **«`tags`/`description` reject before `sources` is ever read» → derfor er
dette «det ENESTE residualet som blokkerer hele korpuset».** Den slutningen
holder ikke. `description` alene løsner **0/53** — fjerner man bare
`description`, passerer ingenting. Og lukker man *alle tre* formene, står
**44/53** fortsatt på `generated` som topp-nivå blokk-mapping. Den bindende
skranken på dette korpuset er altså **ikke** tags/description, men den
topp-nivå blokk-mappingen vi bevisst avviser. Punkt 44 overselger sin egen
betydning med en faktor på over sju (6 mot 53).
Punkt 44 bør skrives om etter at operatøren har bestemt seg — det er en
dokumentasjonsendring som hører sammen med predikatvalget, ikke før det.
## 7. Kandidatpredikatene
Ordren spurte etter predikatet som slipper inn en ren skalar-flow-sekvens.
Det er P1. P2 og P3 tas med fordi målingen viser at P1 alene er en liten
gevinst, og beslutningen bør se hva naboene koster.
### P1 — flow-sekvens av rene skalarer
Predikat: i `_parse_flow_sequence`, når første ikke-blanke tegn i et element
ikke er `{`, les elementet som en plain scalar dersom det ikke inneholder noen
av `{ } [ ] : , " ' #` og ikke er tomt. Blandet sekvens (skalar + mapping)
avvises, slik blokklisten allerede gjør.
- **(a) Slipper inn:** `tags: [finance, revenue, headline-metric]` — 9/53 i
korpuset. Passeringen går fra 0/53 til **6/53**.
- **(b) Avviser fortsatt:** siterte elementer (`['a', 'b']`), elementer med
kolon eller komma i seg, tom sekvens `[]`, uavsluttet `[a, b`, blandet
`[a, {b: c}]`, og nestet `[[a]]` — alle på tegn-nivå, uten YAML-semantikk.
- **(c) Dybde-1:** **bruker den ikke opp.** Elementene er blad; ingen ny
nestingsgrad oppstår. Det er samme dybde blokklisten av skalarer allerede har.
- **(d) Testen som pinner den:** `tags: [a, b]``["a", "b"]`; `tags: ['a']`
raiser; `tags: [a, {b: c}]` raiser med blandingsfeilen; `tags: []` raiser;
`sources: [{ id: x }]` parser uendret (ingen regresjon på G30-bæreren).
### P2 — blokksekvens uten innrykk
Predikat: i `_consume_block_list`, godta også `raw[:1] == "-"` når linjen
starter med `- ` og forrige toppnøkkel hadde tom verdi.
- **(a) Slipper inn:** `tags:` + `- a` i kolonne 0 — 36/53. Men også `sources:`
i samme form — 44/53.
- **(b) Avviser fortsatt:** alt innholdet i elementet avviser i dag; allowlisten
og `_reject_mapping_construct` er uendret.
- **(c) Dybde-1:** **bruker den ikke opp** for skalarelementer, men den er ikke
gratis: den åpner samtidig den ikke-innrykkede blokk-mapping-bæreren for
`sources`, hvor elementenes fortsettelseslinjer *er* innrykket. Det er en
større flate enn `tags`, og den bør vurderes for seg.
- **(d) Testen:** `tags:\n- a\n- b``["a", "b"]`; `sources:\n- id: x\n title: y`
→ én dict; en linje `- a` uten forutgående tom toppnøkkel raiser fortsatt;
`tags:\n- a\n- {b: c}` raiser med blandingsfeilen.
- **Målt effekt alene: 0/53.** Den løsner ingenting uten P1 eller uten at
`generated` også åpnes.
### P3 — foldet plain scalar (fortsettelseslinje)
Predikat: etter en toppnøkkel med ikke-tom, ikke-`[`/`{` verdi, slå sammen
etterfølgende innrykkede linjer som *ikke* starter med `- ` og *ikke* inneholder
en uquotet `": "`, med ett mellomrom som skjøt.
- **(a) Slipper inn:** `description` brutt over to linjer — 29/53.
- **(b) Avviser fortsatt:** en innrykket linje som ser ut som `k: v` treffer
fremdeles nested-mapping-avvisningen; `- ` treffer fremdeles listeruten.
`_reject_dangerous_value` kjører på den sammenslåtte verdien, ikke på
fragmentene.
- **(c) Dybde-1:** **bruker den ikke opp** — resultatet er én skalar.
Men den svekker et vern: i dag er *enhver* innrykket linje uten aktiv listenøkkel
et avvist nestet uttrykk. Etter P3 er den regelen betinget av at linjen ikke
inneholder `": "` — altså samme heuristikk som `_reject_mapping_construct`,
gjenbrukt til å *slippe gjennom* i stedet for til å avvise.
- **(d) Testen:** `description: en setning\n som fortsetter` → én streng med
ett mellomrom; `description: x\n y: z` raiser fortsatt som nested mapping;
`description: x\n - a` raiser fortsatt.
- **Målt effekt alene: 0/53.**
## 8. Anbefaling
**P1 alene. Ikke P2, ikke P3, ikke nå.**
Begrunnelsen er tallene, ikke smaken:
- P1 er den eneste av de tre som flytter passeringstallet i det hele tatt
(0 → 6). P2 og P3 gir hver for seg **0/53**, og lagt oppå P1 gir de fortsatt
**6/53**. De koster parserflate og kjøper null målt konformans.
- P1 er den minste flaten: den er et tegn-nivå-predikat inne i en funksjon som
allerede eksisterer, og den bruker ikke opp dybde-1-regelen.
- P1 lukker et gap STATE allerede fører som bevisst («`tags: [a, b]`
konformansgap»), og den bringer parseren i linje med §4.1s eget skjelett —
den ene formen spec-en faktisk skriver ut.
- P2 og P3 bør ikke besluttes på dette korpuset, fordi korpuset ikke kan skille
dem: **44/53 stopper på `generated` som topp-nivå blokk-mapping uansett.**
Å bygge P2 og P3 nå ville være å betale for to predikater og måle null.
**Den ærlige konsekvensen, som må sies høyt:** P1 tar korpuset fra 0/53 til
6/53. Det lukker ikke «hele korpuset». Skal 53/53 nås, er den neste
beslutningen en helt annen og mye tyngre en — topp-nivå blokk-mapping (44/53)
pluss siterte skalarer (44/53) — og *den* bruker opp dybde-1-regelen. Det er en
sikkerhetsbeslutning, ikke en parserdetalj, og den hører ikke i denne ordren.
## 9. Verifiseringslogg
| Påstand | Kommando | Exit | Resultat |
|---|---|---|---|
| Nevner = 53 | `find scratchpad/corpus -name '*.md' ! -name index.md ! -name log.md \| wc -l` | 0 | 53 |
| Pinnen er ekte forgjenger | `git -C _okf-upstream merge-base --is-ancestor 3fcbb9f HEAD` | 0 | ja (HEAD = `9a15b13`) |
| SPEC-pinnen er ren | `git -C _okf-canonical rev-parse --short HEAD; git status --porcelain` | 0 | `ad30107`, rent tre |
| SPEC har ingen dybderegel | `grep -n -i 'nest\|depth\|indent' SPEC.md` | 0 | 13 treff, 0 relevante |
| Baseline 0/53 | `PYTHONPATH=src .venv/bin/python scratchpad/measure.py` | 0 | 0 passerer |
| Formsensus | `PYTHONPATH=src .venv/bin/python scratchpad/shapes.py` | 0 | tabell §5.1 |
| Kandidattall | `PYTHONPATH=src .venv/bin/python scratchpad/candidates.py` | 0 | tabell §5.2 |
| Positiv kontroll | inline, se §4 | 0 | parser, `tags == ['finance','revenue']` |
| LIMITATIONS-antall | `grep -c '^- \*\*' docs/LIMITATIONS.md` | 0 | 45 |
| Upushet ved øktstart | `git rev-list --count origin/main..HEAD` | 0 | **0** |
**Ikke målt:** om P1 påvirker ytelse eller ReDoS-marginen — predikatet er
tegn-for-tegn uten regex, men ingen sveip er kjørt, siden ingen kode er skrevet.
Kandidatene er målt ved overflatenormalisering, ikke ved en patchet parser: det
er en trofast simulering av *hva* som slippes inn, men den beviser ikke at en
implementasjon av P1 avviser nøyaktig (b)-listen. Testene i (d) er det som ville
pinne det.

View file

@ -4,7 +4,7 @@
especially one converging on Google's Open Knowledge Format (OKF v0.1) — and needs
to decide **when** and **where** to add a write-time ingestion guard.
**Status of the guard:** `v1.2.0`. Stdlib-only core, framework-agnostic. The
**Status of the guard:** `v1.3.0`. Stdlib-only core, framework-agnostic. The
exported Python surface is frozen under semver — nothing exported is removed,
renamed or given a different meaning without a `2.0.0`. Detection behaviour is
*not* frozen: severities, thresholds and lexicon entries are calibration and move
@ -142,9 +142,9 @@ live payload:
python -m llm_ingestion_guard.coverage # exit 0 = all as documented
```
As of `v1.2.0`: **130 / 130 defended classes demonstrated (recall 100%)** and **6 /
As of `v1.3.0`: **130 / 130 defended classes demonstrated (recall 100%)** and **6 /
6 documented gaps still hold** (a *closed* gap fails the test, forcing a doc
update). The matrix is the single source of truth for the test suite (**834
update). The matrix is the single source of truth for the test suite (**868
passing**), which also asserts total recall, that every lexicon pattern has a
case (so the matrix cannot fall behind the lexicon), the full LLM02 secret-egress
set, and the container-layer front-end (CSV formula-injection, zip-slip/bomb,

View file

@ -3,7 +3,7 @@
**A reusable, minimal, dependency-light defensive layer for LLM *ingestion*
pipelines — the write-time siblings of query-time chatbot guardrails.**
Status: implemented — v1.2.0, exported surface frozen under semver. This document defines what the repo contains
Status: implemented — v1.3.0, exported surface frozen under semver. This document defines what the repo contains
and why; the stdlib-only core is built and tested (see `README.md` for usage and
`docs/PLAN.md` for the build order).

View file

@ -40,27 +40,31 @@ items; this is the full list, each with the mechanism.
(an injection in a directory listing is caught) rather than path-rejecting the
conformant bundle. A front-end materialising individual uploads keeps the opposite
rule (`allow_reserved=False`): a reserved basename is a listing-shadow and refused.
- **OKF frontmatter is a restricted grammar: the mapping class has exactly one
expressible form.** Gate T2 accepts a line-oriented subset deliberately — full YAML
is a larger parse-attack surface than a write-time gate needs. Flow sequences
(`[a, b]`) and nested mappings are *rejected outright*, which fails secure.
**Three of the four routes to a mapping fail, each on a different rule** — block
(`k:\n sub: v`) on the nested-mapping check, dotted keys (`k.sub: v`) on the key
pattern, and the inline second colon (`k: sub: v`) on the mapping-construct check.
**The fourth, the flow form, is admitted only when every key is on an allowlist**
- **OKF frontmatter is a restricted grammar: a mapping is expressible through
four carriers, all of them key-allowlisted.** Gate T2 accepts a line-oriented
subset deliberately — full YAML is a larger parse-attack surface than a
write-time gate needs. The admitted carriers are the flow mapping as a value
(`generated: { by: x, at: y }`), the flow mapping as a block-list item, the
flow *sequence* of flow mappings (`sources: [{ id: a, resource: x }]`) and the
block sequence of block mappings (SPEC.md §5.1's own form). **The routes that
still fail, each on a different rule:** a top-level block *mapping*
(`k:\n sub: v`) on the nested-mapping check, dotted keys (`k.sub: v`) on the
key pattern, the inline second colon (`k: sub: v`) on the mapping-construct
check, and a flow sequence of plain *scalars* (`tags: [a, b]`) on the `[`
indicator — the sequence carrier is opened for the mapping element and nothing
else. **Every carrier is admitted only when every key is on an allowlist**
(`by`, `at`, `from`, `to`, `id`, `title`, `author`, `usage_count`,
`last_modified` — the keys SPEC.md @ `62432a09` §5.1/§5.2 names inside a mapping)
`last_modified` — the keys SPEC.md @ `62432a09` §5.1/§5.2 names inside a mapping,
plus `resource` and `usage_window` *under `sources` only*, see below)
and every leaf is a plain scalar, itself run through the same value predicates as a
top-level scalar. Nested collections, quoted leaves, duplicate keys, an empty or
unclosed mapping, and `{a:b}` (which PyYAML 6.0.3 reads as the *key* `a:b`, not as
a scalar) all raise. The form is expressible, never trusted: the allowlist
inspects every key, which is the property that carried the security when the
blanket refusal was doing the enforcing. **`resource` is deliberately off the
allowlist** although §5.1 names it inside a `sources` entry — it is a pointer
rather than a label and the only key T3 exists for, so admitting it would let
`executor: { resource: skills/run.md }` carry an executable-code pointer through a
key the https allowlist never inspects. What else survives is scalars and flat
lists of strings. **Two routes used to
blanket refusal was doing the enforcing. **A block list may not mix scalar items
and mappings** — YAML permits it, but a consumer iterating `sources` and reading
`entry.get("id")` gets an `AttributeError` off the first `str`. What else
survives is scalars and flat lists of strings. **Two routes used to
degrade into a string instead of failing, and that defect is closed in `1.1.0`**:
a block-sequence item carrying exactly one key (`sources:\n - uri: https://e.com/a`
yielded the *string* `'uri: https://e.com/a'`) and the inline second colon
@ -85,19 +89,76 @@ items; this is the full list, each with the mechanism.
door A/B persist path, so frontmatter that fails secure on import passes
`screen_output` unremarked. The grammar therefore bounds what a consumer can *receive*,
never what a producer can *emit*. Verified identical on 0.2.0 and 0.3.1.
- **An OKF v0.2 concept traverses the external-import path only if its `sources` are
flat.** The wall used to be total: both of v0.2's backward-breaking migration targets
are mappings — `timestamp``generated.at`, and body `# Citations` → a `sources`
block list of mappings — and a consumer measured **0 of 53** upstream concepts
through the gate. The trust and provenance layer now passes in its spec form
(`generated`, `verified` bare or listed, `usage_window`), so `generated.at` is no
longer a wall. **`sources` still is**: SPEC.md writes each entry as a block mapping
under a block sequence (`- id: …\n resource: …`), and that carrier stays refused —
it is the shape whose one-key degradation smuggled a pointer before `1.1.0`, and
reopening it is a separate parse-safety decision, not a corollary of the flow form.
A concept whose `sources` are flat strings, or absent, imports. The
dangling-or-substituted `executor`/`attester` pointer question stays out of reach
for the same reason: both are mappings whose payload key is `resource`.
- **`sources` passes in both of its spec carriers; the per-entry `usage_window`
does not.** The wall used to be total: both of v0.2's backward-breaking migration
targets are mappings — `timestamp``generated.at`, and body `# Citations` → a
`sources` block list of mappings — and a consumer measured **0 of 53** upstream
concepts through the gate. `generated.at` stopped being a wall in `1.2.0`;
`sources` stopped being one in `1.3.0`, which admits both the block sequence of
block mappings (SPEC.md §5.1's own example) and the flow sequence of flow
mappings (the form the OKF producers emit, measured 2026-09-02 by two consumers
independently). **`resource` is allowlisted inside a `sources` entry and nowhere
else.** `1.2.0` left it off on the argument that the parser could not tell
`sources[].resource` (§5.1, a citation) from `executor.resource` / `attester.resource`
(§10, a pointer to code to be run — door C). That premise was measured false: the
owning key is in scope at every call site and was simply never threaded through.
It is threaded now, so `executor: [{ resource: skills/run.md }]` and
`attester:\n - resource: …` are refused on the allowlist through *every* carrier,
including the two this opened. **What stays refused: a `usage_window` inside a
`sources` entry.** SPEC §5.1 permits it per entry ("A single entry MAY carry its
own `usage_window`"), and it is a mapping inside a mapping — depth 2, which this
parser admits at no key. A bundle using the per-entry override is refused; the
shared sibling `usage_window` (the §5.1 example's own form) passes. This is a
registered conformance gap, not an oversight: no-nesting-past-depth-1 is a
security property, and spending it was not what the fix was for.
- **`sources[].resource` is scanned as text but never validated as a URL.** T3's
https allowlist inspects the *top-level* `resource` and nothing else. It cannot
be extended to `sources` entries without over-blocking conformant bundles: SPEC
§5.1 explicitly permits a bundle-relative path, a path into `references/`, or a
scope descriptor a consumer cannot follow at all (the OKF producers' own golden
bundle emits `resource: fixture`). So a `sources` entry may carry
`file://`, `javascript:` or any other string; it goes through T1's scan like any
other frontmatter value, and nothing else. **A consumer that dereferences
`sources[].resource` must validate it itself** — `okf.validate_resource_url` is
exported for exactly that. The dangling-or-substituted `executor`/`attester`
pointer question stays out of reach separately: both are top-level block
mappings, a carrier that is still refused.
- **`tags` parses in two of the three forms the OKF corpus writes it in;
`description`'s folded form still does not — and neither is what caps the
corpus.** Re-measured 2026-09-07/08 against the pinned upstream reference
bundles (`_okf-upstream/okf` @ `3fcbb9f`, denominator **53** — every
non-reserved `.md`, which is SPEC §11.1's own unit) and the pinned SPEC
(`_okf-canonical` @ `ad30107`). This entry carried three claims that the
measurement showed to be wrong, all three inherited from `1.2.0` and left
standing for twelve days after `1.3.0` moved them; they are corrected here.
- **A sequence-value type exists**`_consume_block_list` has parsed block
lists of plain scalars since `1.3.0`, and `1.4.0` adds the flow sequence of
plain scalars (`tags: [a, b, c]`, SPEC §4.1's own skeleton). What is missing
is not the type but the **indentation requirement**: `_consume_block_list`
demands a leading space, and the corpus writes `- a` flush at column 0
(36/53 of `tags`, 44/53 of `sources`). Shape census of `tags` over the 53:
36/53 flush block sequence, 9/53 flow sequence, 8/53 single-line scalar.
- **The number is 6/53, not 4/53.** `4` was the `1.2.0` parser's figure.
Measured on `1.4.0`, the corpus goes from **0/53 to 6/53** — all six in
`acme_retail`, being 6 of the 9 documents whose `tags` is a flow sequence;
the other three stop on the mapping-key allowlist (`parameters.name` ×2,
`not.term` ×1), not on `tags`.
- **`tags`/`description` are not the binding constraint.** `description`
alone releases **0/53** — 29/53 write it as a folded plain scalar over an
indented second line, which the parser has no continuation-line model for,
but removing that obstacle by itself lets nothing through. With all three
surface forms closed the corpus still measures **6/53**, because **44/53
stop on `generated` written as a top-level block mapping**, and behind that
wall sit 44/53 single-quoted scalars inside the mapping. That constraint is
refused deliberately: it spends the no-nesting-past-depth-1 rule, which is
a security decision and not a parser detail.
**Said plainly: `1.4.0` takes this corpus to 6/53, not to 53/53.** The two
neighbouring predicates were measured and deliberately **not** built — a
flush-left block sequence and a folded plain scalar each release **0/53** on
their own, and stacked on top of `1.4.0` they still measure 6/53. They would
cost parser surface and buy no measured conformance. The full measurement,
including the SPEC quotations and the per-candidate table, is
`docs/2026-09-07-limitations-44-maaling.md`.
- **A persist gate cannot cover execution risk.** OKF v0.2 introduces concepts whose
purpose is to *name code to be run* (`runtime`, `executor.resource`,
`attester.resource`). This library answers "is this safe to **store**"; executable

View file

@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "llm-ingestion-guard"
version = "1.2.0"
version = "1.4.0"
description = "Write-time defensive layer for Python pipelines that persist LLM output: sanitize, fence, tool-less quarantined transform, capability isolation, scan before persist, fail-secure."
readme = "README.md"
requires-python = ">=3.10"

155
scratchpad/candidates.py Normal file
View file

@ -0,0 +1,155 @@
"""What each candidate predicate would admit, measured by normalizing the SURFACE
form onto a shape the CURRENT parser already accepts. The predicate is imported,
never re-implemented: only the input's spelling is rewritten."""
import os, re, sys, collections
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src"))
from llm_ingestion_guard import okf
ROOT = os.path.join(os.path.dirname(__file__), "corpus", "okf", "bundles")
def docs():
out = []
for dirpath, _d, fns in os.walk(ROOT):
for fn in sorted(fns):
if fn.endswith(".md") and fn not in ("index.md", "log.md"):
out.append(os.path.join(dirpath, fn))
return sorted(out)
def split_fm(text):
lines = text.split("\n")
if not lines or lines[0].strip() != "---":
return None, None
for i in range(1, len(lines)):
if lines[i].strip() == "---":
return lines[1:i], lines[i + 1:]
return None, None
TOPKEY = re.compile(r"^([A-Za-z0-9_]+):(.*)$")
def n_flow_scalar_seq(fm):
"""P1: `k: [a, b]` where every element is a plain scalar -> indented block list."""
out = []
for raw in fm:
m = TOPKEY.match(raw)
if m and m.group(2).strip().startswith("[") and m.group(2).strip().endswith("]"):
inner = m.group(2).strip()[1:-1]
elems = [e.strip() for e in inner.split(",")]
if inner and all(e and "{" not in e and "}" not in e and "[" not in e for e in elems):
out.append("%s:" % m.group(1))
out.extend(" - %s" % e for e in elems)
continue
out.append(raw)
return out
def n_flush_block_seq(fm):
"""P2: a flush-left `- item` run under a bare `k:` -> the same run, indented."""
out, i, n = [], 0, len(fm)
while i < n:
raw = fm[i]
m = TOPKEY.match(raw)
if m and m.group(2).strip() == "" and i + 1 < n and fm[i + 1][:1] == "-":
out.append(raw)
i += 1
while i < n and (fm[i][:1] == "-" or fm[i][:1] in (" ", "\t")):
out.append(" " + fm[i])
i += 1
continue
out.append(raw)
i += 1
return out
def n_folded_scalar(fm):
"""P3: `k: text` continued on more-indented plain lines -> one joined line."""
out, i, n = [], 0, len(fm)
while i < n:
raw = fm[i]
m = TOPKEY.match(raw)
if m and m.group(2).strip() and not m.group(2).strip()[0] in "[{":
acc = raw
i += 1
while i < n and fm[i][:1] in (" ", "\t") and not fm[i].strip().startswith("- "):
acc = acc.rstrip() + " " + fm[i].strip()
i += 1
out.append(acc)
continue
out.append(raw)
i += 1
return out
def parses(fm, body):
text = "---\n" + "\n".join(fm) + "\n---\n" + "\n".join(body)
try:
okf.parse_frontmatter(text)
return None
except okf.OKFFrontmatterError as exc:
return str(exc)
COMBOS = [
("baseline (v1.3.0 as shipped)", []),
("P1 scalar flow sequence", [n_flow_scalar_seq]),
("P2 flush block sequence", [n_flush_block_seq]),
("P3 folded plain scalar", [n_folded_scalar]),
("P1+P3", [n_flow_scalar_seq, n_folded_scalar]),
("P2+P3", [n_flush_block_seq, n_folded_scalar]),
("P1+P2", [n_flow_scalar_seq, n_flush_block_seq]),
("P1+P2+P3", [n_flow_scalar_seq, n_flush_block_seq, n_folded_scalar]),
]
def main():
paths = docs()
print("denominator: %d documents (pin 3fcbb9f, parser HEAD)" % len(paths))
for name, fns in COMBOS:
ok = 0
residual = collections.Counter()
for p in paths:
fm, body = split_fm(open(p, encoding="utf-8").read())
for fn in fns:
fm = fn(fm)
err = parses(fm, body)
if err is None:
ok += 1
else:
residual[err.split(":")[0]] += 1
top = "; ".join("%dx %s" % (c, r) for r, c in residual.most_common(3))
print(" %-30s %2d/%d residual: %s" % (name, ok, len(paths), top or "-"))
main()
# --- diagnostic only (NOT a proposal): size the real binding constraint ---
def n_block_mapping(fm):
"""D4: a top-level `k:` followed by indented `a: b` lines -> one flow mapping."""
out, i, n = [], 0, len(fm)
while i < n:
raw = fm[i]
m = TOPKEY.match(raw)
if m and m.group(2).strip() == "" and i + 1 < n and fm[i + 1][:1] in (" ", "\t") \
and not fm[i + 1].strip().startswith("- "):
pairs = []
i += 1
while i < n and fm[i][:1] in (" ", "\t") and not fm[i].strip().startswith("- "):
pairs.append(fm[i].strip())
i += 1
out.append("%s: { %s }" % (m.group(1), ", ".join(pairs)))
continue
out.append(raw)
i += 1
return out
print("\n--- diagnostic: what caps the corpus ABOVE the tags/description gap ---")
for name, fns in [
("D4 top-level block mapping", [n_block_mapping]),
("P1+P2+P3+D4", [n_flow_scalar_seq, n_flush_block_seq, n_folded_scalar, n_block_mapping]),
]:
ok = 0
residual = collections.Counter()
for p in docs():
fm, body = split_fm(open(p, encoding="utf-8").read())
for fn in fns:
fm = fn(fm)
err = parses(fm, body)
if err is None:
ok += 1
else:
residual[err.split(":")[0]] += 1
top = "; ".join("%dx %s" % (c, r) for r, c in residual.most_common(3))
print(" %-30s %2d/53 residual: %s" % (name, ok, top or "-"))

45
scratchpad/measure.py Normal file
View file

@ -0,0 +1,45 @@
"""Measure the pinned OKF corpus against the CURRENT parser. No src/ changes."""
import os, sys, collections
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src"))
from llm_ingestion_guard import okf
ROOT = os.path.join(os.path.dirname(__file__), "corpus", "okf", "bundles")
def docs():
out = []
for dirpath, _dirnames, filenames in os.walk(ROOT):
for fn in sorted(filenames):
if not fn.endswith(".md"):
continue
if fn in ("index.md", "log.md"):
continue
out.append(os.path.join(dirpath, fn))
return sorted(out)
def main():
paths = docs()
ok, fail = [], []
reasons = collections.Counter()
for p in paths:
with open(p, encoding="utf-8") as fh:
text = fh.read()
try:
okf.parse_frontmatter(text)
ok.append(p)
except okf.OKFFrontmatterError as exc:
fail.append((p, str(exc)))
reasons[str(exc).split(":")[0]] += 1
print("denominator: %d documents" % len(paths))
print("parse OK : %d" % len(ok))
print("parse FAIL : %d" % len(fail))
print("--- failure classes ---")
for r, c in reasons.most_common():
print("%4d %s" % (c, r))
print("--- passing documents (positive control) ---")
for p in ok:
print(" " + os.path.relpath(p, ROOT))
print("--- first 8 failures verbatim ---")
for p, e in fail[:8]:
print(" %s\n %s" % (os.path.relpath(p, ROOT), e))
main()

59
scratchpad/shapes.py Normal file
View file

@ -0,0 +1,59 @@
"""Shape census of the pinned OKF corpus: which surface form each key is written in."""
import os, re, sys, collections
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src"))
from llm_ingestion_guard import okf
ROOT = os.path.join(os.path.dirname(__file__), "corpus", "okf", "bundles")
def docs():
out = []
for dirpath, _d, fns in os.walk(ROOT):
for fn in sorted(fns):
if fn.endswith(".md") and fn not in ("index.md", "log.md"):
out.append(os.path.join(dirpath, fn))
return sorted(out)
def split_fm(text):
lines = text.split("\n")
if not lines or lines[0].strip() != "---":
return None, None
for i in range(1, len(lines)):
if lines[i].strip() == "---":
return lines[1:i], lines[i + 1:]
return None, None
def key_shape(fm, key):
for i, raw in enumerate(fm):
m = re.match(r"^(%s):(.*)$" % re.escape(key), raw)
if not m:
continue
val = m.group(2).strip()
follow = fm[i + 1] if i + 1 < len(fm) else ""
if val.startswith("["):
return "flow-sequence"
if val.startswith("{"):
return "flow-mapping"
if val:
if follow[:1] in (" ", "\t") and not follow.strip().startswith("- "):
return "scalar+continuation"
return "single-line-scalar"
if follow.strip().startswith("- "):
return "block-seq-indented" if follow[:1] in (" ", "\t") else "block-seq-flush"
if follow[:1] in (" ", "\t"):
return "block-mapping"
return "empty"
return "ABSENT"
def main():
paths = docs()
print("denominator: %d" % len(paths))
for key in ("tags", "description", "generated", "verified", "sources"):
c = collections.Counter()
for p in paths:
fm, _ = split_fm(open(p, encoding="utf-8").read())
c[key_shape(fm, key)] += 1
print("\n%s:" % key)
for shape, n in c.most_common():
print(" %-22s %d/%d" % (shape, n, len(paths)))
main()

View file

@ -63,7 +63,7 @@ from .grounding import (
)
from . import okf
__version__ = "1.2.0"
__version__ = "1.4.0"
# --- §6 bookends: the two library-side halves around the transform ---------

View file

@ -542,8 +542,14 @@ def _build_cases() -> list[Case]:
lambda: okf.parse_frontmatter("---\nkey:\n nested: x\n---\nbody\n"), owasp="LLM10"),
_raise_case("okf", "T2 frontmatter block scalar", "OKFFrontmatterError",
lambda: okf.parse_frontmatter("---\ndesc: |\n block\n---\nbody\n"), owasp="LLM10"),
_raise_case("okf", "T2 frontmatter flow sequence", "OKFFrontmatterError",
lambda: okf.parse_frontmatter("---\ntags: [a, b]\n---\nbody\n"), owasp="LLM10"),
# The plain-scalar flow sequence is ADMITTED as of 1.4.0 (P1, SPEC §4.1's
# own skeleton for `tags`); the class this row measures is the element
# shape the parser would have to interpret rather than read -- here a
# quoted one, which it would have to strip a quote from to return.
_raise_case("okf", "T2 frontmatter flow sequence, quoted element",
"OKFFrontmatterError",
lambda: okf.parse_frontmatter("---\ntags: ['a', 'b']\n---\nbody\n"),
owasp="LLM10"),
_raise_case("okf", "T2 mapping key off the allowlist", "OKFFrontmatterError",
lambda: okf.parse_frontmatter(
"---\ngenerated: { by: a, tool: shell }\n---\nbody\n"), owasp="LLM10"),

View file

@ -65,32 +65,57 @@ _KEY_RE = re.compile(r"^[A-Za-z0-9_][A-Za-z0-9_-]*$")
# A plain OKF scalar cannot *begin* with a YAML structural indicator. Any value
# starting with one signals an anchor (&), alias (*), explicit tag (!), block
# scalar (|, >), flow collection ([ ] { }), directive (%) or reserved char
# (@ `) — all outside the supported subset and all rejected. `{` is tried as the
# allowlisted mapping form FIRST (G3); it reaches this predicate only as a leaf
# inside one, where a nested collection is refused before it can be read.
# (@ `) — all outside the supported subset and all rejected. `{` and `[` are
# tried as the allowlisted mapping form (G3) and the flow sequence of them (G30)
# FIRST; they reach this predicate only as a leaf inside one, where a nested
# collection is refused before it can be read.
_DANGEROUS_VALUE_STARTS = frozenset("&*!|>[]{}%@`")
# A quoted scalar is a scalar in YAML however many colons it carries, so the
# mapping check steps aside for one. The quotes are retained rather than
# stripped — a pre-existing divergence, pinned in tests/test_okf.py.
_QUOTE_STARTS = frozenset("\"'")
# G3 — the one mapping form T2 can express (operator decision, 2026-08-21).
# P1 - what disqualifies a flow-sequence element from being a plain scalar
# (operator decision, 2026-09-08). Each character is one this parser would have
# to interpret rather than read: the two quotes (retained, never stripped), the
# two splitters, the two collection openers and their closers, and the comment
# indicator. Refusing them is what lets the element be split on commas at the
# character level without a YAML quote state machine.
_FLOW_SCALAR_REFUSED = "{}[]:,\"'#"
# G3 - the one mapping form T2 can express (operator decision, 2026-08-21).
# Every key inside a mapping must be on this allowlist: the form is safe because
# the allowlist inspects each key, not because mappings became trusted. The keys
# are the ones OKF v0.2 names inside a mapping - `by`/`at` (SPEC.md @ 62432a09
# §5.2 `generated`/`verified`), `from`/`to` (§5.1 `usage_window`) and the
# `sources`-entry fields (§5.1). `resource` is the one §5.1 key deliberately
# LEFT OFF: it is a pointer rather than a label, it is the only key T3 exists
# for, and admitting it inside a mapping would re-open the door-C route closed
# in 1.1.0 (`executor: {resource: skills/run.md}` puts an executable-code
# pointer in a key the https allowlist never inspects). It costs nothing today,
# because the conformant carrier for `sources[].resource` is the block-sequence
# of block-mappings, which this form does not admit either way.
# §5.2 `generated`/`verified`) and `from`/`to` (§5.1 `usage_window`), plus the
# §5.1 `sources`-entry labels.
_MAPPING_KEY_ALLOWLIST = frozenset({
"by", "at", "from", "to", "id", "title", "author", "usage_count",
"last_modified",
})
# G30 - the two §5.1 keys admitted inside a `sources` entry and NOWHERE else
# (operator decision, 2026-09-02). `resource` is REQUIRED within a `sources`
# entry, so leaving it off left the whole provenance family unwritable; but the
# same field name in §10 (`executor.resource`, `attester.resource`) names run
# instructions and code - the door-C route closed in 1.1.0. 1.2.0 argued the
# parser could not tell the two apart without parent-key context it did not
# have. That premise was false: the owning key is in scope at every call site
# below, it was simply never threaded through. It is threaded now, so the
# discrimination is structural rather than a judgement about the value.
# `usage_window` is allowlisted here for accuracy of refusal - §5.1 permits it
# per entry, and it is then refused on the depth rule (a mapping inside a
# mapping, which this parser admits at no key) rather than refused as if the
# key were unknown.
_SOURCES_ENTRY_KEYS = frozenset({"resource", "usage_window"})
def _allowed_mapping_keys(parent_key):
"""The mapping-key allowlist for a mapping owned by ``parent_key``."""
if parent_key == "sources":
return _MAPPING_KEY_ALLOWLIST | _SOURCES_ENTRY_KEYS
return _MAPPING_KEY_ALLOWLIST
class OKFError(Exception):
"""Base class for OKF adapter rejections."""
@ -184,8 +209,10 @@ def _value_regions(value):
A mapping value (G3) is a new *shape* on this surface, not a new exemption:
its leaves are scanned exactly like a scalar or a list item, so an injection
parked in ``generated: { by: ... }`` reaches ``scan_output`` like any other
frontmatter text. Mapping *keys* are not scanned because they cannot carry
attacker text - the allowlist admits nine fixed names and nothing else.
frontmatter text. The same holds for a *list* of mappings (G30, ``sources``),
which this function already flattens through its list branch. Mapping *keys*
are not scanned because they cannot carry attacker text - the allowlist
admits a fixed, per-parent name set and nothing else.
"""
if isinstance(value, dict):
return [leaf for leaf in value.values() if leaf]
@ -616,16 +643,22 @@ def _parse_flat(fm_lines):
raise OKFFrontmatterError("invalid frontmatter key: %r" % key)
if value == "":
items, i = _consume_block_list(fm_lines, i + 1)
items, i = _consume_block_list(fm_lines, i + 1, key)
result[key] = items if items is not None else ""
continue
mapping = _parse_flow_mapping(value)
mapping = _parse_flow_mapping(value, key)
if mapping is not None:
result[key] = mapping
i += 1
continue
sequence = _parse_flow_sequence(value, key)
if sequence is not None:
result[key] = sequence
i += 1
continue
_reject_dangerous_value(value)
_reject_mapping_construct(value)
result[key] = value
@ -634,14 +667,26 @@ def _parse_flat(fm_lines):
return result
def _consume_block_list(fm_lines, start):
def _consume_block_list(fm_lines, start, parent_key=None):
"""Consume `` - item`` lines following a bare ``key:``.
Returns ``(items, next_index)`` ``items`` is ``None`` (and ``next_index``
Returns ``(items, next_index)`` - ``items`` is ``None`` (and ``next_index``
unchanged) when no list item follows, so the caller can treat the key as an
empty scalar and let the next line trip the nested-structure guard.
An item is one of three shapes, decided by the item text alone: a flow
mapping (G3), a block mapping (G30 - an unquoted ``key: value`` opening a
run of more-indented sibling entries, which is SPEC.md §5.1's own carrier
for ``sources``), or a plain scalar. ``parent_key`` is the key that owns the
list; it decides the mapping-key allowlist, which is how ``sources[].resource``
is admitted while ``executor``/``attester`` ``resource`` stays refused.
A list may not mix scalars and mappings. YAML permits it, but a consumer
iterating ``sources`` and reading ``entry.get("id")`` gets an
``AttributeError`` off the first ``str`` - refusing is the cheaper failure.
"""
items = []
kinds = set()
i = start
n = len(fm_lines)
while i < n:
@ -650,24 +695,113 @@ def _consume_block_list(fm_lines, start):
if stripped == "" or stripped.startswith("#"):
i += 1
continue
if raw[:1] in (" ", "\t") and stripped.startswith("- "):
if not (raw[:1] in (" ", "\t") and stripped.startswith("- ")):
break
item = stripped[2:].strip()
mapping = _parse_flow_mapping(item)
mapping = _parse_flow_mapping(item, parent_key)
if mapping is not None:
items.append(mapping)
kinds.add("mapping")
i += 1
continue
entry = _block_mapping_entry(item)
if entry is not None:
mapping, i = _consume_block_mapping(fm_lines, i + 1, entry, parent_key)
items.append(mapping)
kinds.add("mapping")
continue
_reject_dangerous_value(item)
_reject_mapping_construct(item)
items.append(item)
kinds.add("scalar")
i += 1
continue
break
if len(kinds) > 1:
raise OKFFrontmatterError(
"a block list may not mix scalar items and mappings: %r" % (parent_key,)
)
if not items:
return None, start
return items, i
def _block_mapping_entry(text):
"""Read ``text`` as one ``key: value`` block-mapping entry, or return ``None``.
The trigger is deliberately the same shape ``_reject_mapping_construct``
uses to *refuse* a scalar: an unquoted ``": "``. What changes in 1.3.0 is
only what happens next - the entry is admitted key-by-key against the
allowlist instead of refused wholesale. Every shape that is a scalar to
PyYAML stays one here: a quoted item, a colon with no space
(``domain:security``, ``https://e.com:8443/a``) and a trailing colon all
return ``None`` and fall through to the unchanged scalar rules.
"""
if not text or text[0] in _QUOTE_STARTS:
return None
key, sep, leaf = text.partition(": ")
if not sep:
return None
key = key.strip()
if not _KEY_RE.match(key):
return None
return key, leaf.strip()
def _consume_block_mapping(fm_lines, start, first_entry, parent_key):
"""Consume the sibling entries of a block mapping opened by a ``- `` item.
Returns ``(mapping, next_index)``. A sibling is an indented line that does
not open a new list item; the run ends at a blank line, a comment, a new
``- `` item, or a line at column zero. Depth is capped at one by giving the
leaves the *unchanged* scalar predicates: a nested collection opens with
``{`` or ``[`` and is refused by ``_reject_dangerous_value``, and a further
block level is refused by ``_reject_mapping_construct``.
"""
allowed = _allowed_mapping_keys(parent_key)
mapping = {}
_admit_mapping_entry(mapping, first_entry[0], first_entry[1], allowed, parent_key)
i = start
n = len(fm_lines)
while i < n:
raw = fm_lines[i]
stripped = raw.strip()
if stripped == "" or stripped.startswith("#"):
break
if raw[:1] not in (" ", "\t") or stripped.startswith("- "):
break
entry = _block_mapping_entry(stripped)
if entry is None:
_reject_dangerous_value(stripped)
_reject_mapping_construct(stripped)
raise OKFFrontmatterError(
"a block-mapping entry must be 'key: value': %r" % (raw,)
)
_admit_mapping_entry(mapping, entry[0], entry[1], allowed, parent_key)
i += 1
return mapping, i
def _admit_mapping_entry(mapping, key, leaf, allowed, parent_key):
"""Admit one mapping entry, or raise. The single gate both carriers pass."""
if not _KEY_RE.match(key):
raise OKFFrontmatterError("invalid mapping key: %r" % (key,))
if key not in allowed:
raise OKFFrontmatterError(
"mapping key %r is not on the OKF mapping allowlist under %r"
% (key, parent_key)
)
if key in mapping:
raise OKFFrontmatterError("duplicate mapping key %r" % (key,))
_reject_dangerous_value(leaf)
_reject_mapping_construct(leaf)
mapping[key] = leaf
def _reject_dangerous_value(value):
if value and value[0] in _DANGEROUS_VALUE_STARTS:
raise OKFFrontmatterError(
@ -704,7 +838,7 @@ def _reject_mapping_construct(value):
)
def _parse_flow_mapping(value):
def _parse_flow_mapping(value, parent_key=None):
"""Parse ``{ key: value, ... }`` into a typed dict, or refuse it (G3).
Returns ``None`` when ``value`` does not open a flow mapping, so the caller
@ -773,6 +907,7 @@ def _parse_flow_mapping(value):
% (value,)
)
allowed = _allowed_mapping_keys(parent_key)
mapping = {}
for entry in inner.split(","):
entry = entry.strip()
@ -781,20 +916,122 @@ def _parse_flow_mapping(value):
raise OKFFrontmatterError(
"a flow-mapping entry must be 'key: value': %r" % (entry,)
)
key = key.strip()
leaf = leaf.strip()
if not _KEY_RE.match(key):
raise OKFFrontmatterError("invalid flow-mapping key: %r" % (key,))
if key not in _MAPPING_KEY_ALLOWLIST:
raise OKFFrontmatterError(
"flow-mapping key %r is not on the OKF mapping allowlist: %r"
% (key, value)
)
if key in mapping:
raise OKFFrontmatterError(
"duplicate flow-mapping key %r: %r" % (key, value)
)
_reject_dangerous_value(leaf)
_reject_mapping_construct(leaf)
mapping[key] = leaf
_admit_mapping_entry(mapping, key.strip(), leaf.strip(), allowed, parent_key)
return mapping
def _parse_flow_sequence(value, parent_key=None):
"""Parse ``[{ ... }, { ... }]`` into a list of typed dicts, or refuse it (G30).
Returns ``None`` when ``value`` does not open a flow sequence, so the caller
falls through to the unchanged scalar rules - where ``[`` is still a
disallowed indicator. This carrier is opened for the flow-mapping element
and nothing else: it is the form the OKF producers emit for ``sources``
(measured 02.09 against llm-ingestion-okf's golden bundle, where a
one-element sequence raised on the ``[`` just as a two-element one did).
As of 1.4.0 it also carries a sequence of plain *scalars*
(``tags: [a, b, c]``) - SPEC.md §4.1's own skeleton for ``tags``, and the
one candidate form measured to move the upstream corpus at all (0/53 ->
6/53 against ``_okf-upstream`` @ 3fcbb9f, denominator 53; see
docs/2026-09-07-limitations-44-maaling.md). The quoting and comma-splitting
problem that kept it refused is answered by refusing the characters that
create it rather than by parsing them: an element is a plain scalar only if
it is non-empty and carries none of ``{ } [ ] : , " ' #``, and it then
passes the unchanged scalar indicator rule. Everything needing YAML
semantics to split or unquote correctly still raises.
A mapping element is split on ``}`` rather than on commas, which is sound
precisely because ``_parse_flow_mapping`` admits no nested collection: a
``}`` inside an element cannot occur, so the first ``}`` after ``{`` always
closes it. A scalar element runs to the next comma, which cannot occur
inside one. Anything between elements that is not a separating comma is
refused, which is what makes trailing junk fail rather than parse.
A sequence may not mix the two, for the reason the block list may not: a
consumer iterating the value and reading ``entry.get("id")`` gets an
``AttributeError`` off the first ``str``. The mixing verdict is reached
before the element is parsed, so the caller is told about the mix rather
than about a key the allowlist would have complained of instead.
"""
if not value or value[0] != "[":
return None
if not value.endswith("]"):
raise OKFFrontmatterError(
"a flow sequence must be closed by ']' on the same line: %r" % (value,)
)
inner = value[1:-1].strip()
if not inner:
raise OKFFrontmatterError("an empty flow sequence carries nothing: %r" % (value,))
items = []
kinds = set()
i = 0
n = len(inner)
while True:
while i < n and inner[i] in " \t":
i += 1
if i >= n:
break
if inner[i] == "{":
_refuse_mixed_flow_sequence(kinds, "mapping", value)
close = inner.find("}", i)
if close == -1:
raise OKFFrontmatterError(
"an unclosed flow mapping inside a flow sequence: %r" % (value,)
)
items.append(_parse_flow_mapping(inner[i:close + 1], parent_key))
i = close + 1
else:
_refuse_mixed_flow_sequence(kinds, "scalar", value)
end = i
while end < n and inner[end] != ",":
end += 1
items.append(_flow_sequence_scalar(inner[i:end].strip(), value))
i = end
while i < n and inner[i] in " \t":
i += 1
if i >= n:
break
if inner[i] != ",":
raise OKFFrontmatterError(
"trailing junk after a flow-sequence element: %r" % (value,)
)
i += 1
return items
def _refuse_mixed_flow_sequence(kinds, kind, value):
kinds.add(kind)
if len(kinds) > 1:
raise OKFFrontmatterError(
"a flow sequence may not mix scalar items and mappings: %r" % (value,)
)
def _flow_sequence_scalar(element, value):
"""Read one flow-sequence element as a plain scalar, or refuse it (P1).
Character-level, with no YAML semantics: the element must be non-empty and
carry none of :data:`_FLOW_SCALAR_REFUSED`. That set is not a style rule -
each member is a character whose meaning this parser would have to guess at.
A quote would have to be stripped (this parser retains quotes, so it would
hand back a different value than YAML reads); a comma or a colon would have
to be split on; a bracket or a brace would open a second collection level,
which no carrier here admits; a ``#`` opens a comment. The unchanged
indicator rule then applies to what is left, exactly as it does to a
block-list item, so an anchor or an alias is no more a scalar here.
"""
if not element:
raise OKFFrontmatterError(
"an empty element in a flow sequence carries nothing: %r" % (value,)
)
for char in _FLOW_SCALAR_REFUSED:
if char in element:
raise OKFFrontmatterError(
"a flow-sequence scalar admits plain scalars only, not %r: %r"
% (char, value)
)
_reject_dangerous_value(element)
return element

View file

@ -116,9 +116,13 @@ def test_rejects_unterminated_frontmatter():
def test_rejects_flow_collection():
# inline flow collections are outside the supported subset -> reject, don't
# silently mis-parse the bracket string as a scalar.
doc = "---\ntype: table\ntags: [pii, customers]\n---\nbody\n"
# An inline flow collection this parser cannot read without guessing ->
# reject, don't silently mis-parse the bracket string as a scalar. The
# QUOTED elements are what makes this row stay red: the plain-scalar form
# (`tags: [pii, customers]`) is admitted as of 1.4.0 -- see the P1 block at
# the foot of this file -- while a quoted element would have to be stripped
# to be read, and this parser retains quotes.
doc = "---\ntype: table\ntags: ['pii', 'customers']\n---\nbody\n"
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(doc)
@ -548,9 +552,15 @@ _V02_REJECTED = [
("generated (nested)", "generated:\n at: 2026-07-26T10:00:00Z\n"),
("executor (nested)", "executor:\n resource: skills/run-on-bq.md\n"),
("attester (nested)", "attester:\n resource: attesters/sql_equality.py\n"),
("sources (block list of mappings)",
# The carrier is admitted as of 1.3.0 (G30); this row now measures the KEY
# SET - `uri`/`kind` are producer-invented, not SPEC §5.1 - and stays red for
# that reason. SPEC's own §5.1 keys parse; see the G30 block at the foot.
("sources (block list, off-allowlist keys)",
"sources:\n - uri: https://e.com/a\n kind: doc\n"),
("flow sequence", "tags: [a, b, c]\n"),
# The plain-scalar flow sequence is admitted as of 1.4.0 (P1); this row now
# measures the QUOTED element, which stays refused for the same reason it
# always did -- reading it would mean stripping a quote this parser retains.
("flow sequence, quoted elements", "tags: ['a', 'b']\n"),
("flow mapping", "executor: {resource: skills/run.md}\n"),
]
@ -569,6 +579,8 @@ _V02_ADMITTED = [
("status/stale_after", "status: active\nstale_after: 2026-12-01\n"),
("verified bool", "verified: true\n"),
("block sequence of scalars", "tags:\n - alpha\n - beta\n"),
# SPEC §4.1's own skeleton for `tags`, admitted as of 1.4.0 (P1).
("flow sequence of scalars", "tags: [alpha, beta]\n"),
]
@ -586,6 +598,8 @@ def test_v02_flat_frontmatter_still_parses(cid, fm):
_DEGRADED_TO_STRING = [
# (id, frontmatter, what PyYAML 6.0.3 makes of it)
# Still red after G30 opened the block-mapping carrier: `uri` is not a §5.1
# key, so the item is refused by the allowlist instead of by the grammar.
("one key per item", "sources:\n - uri: https://e.com/a\n", "[{'uri': ...}]"),
("item, trailing colon", "sources:\n - uri:\n", "[{'uri': None}]"),
("inline double colon", "attester: resource: attesters/sql_equality.py\n", "parse error"),
@ -653,13 +667,16 @@ def test_pointer_in_a_degraded_mapping_no_longer_reaches_the_consumer_tree(cid,
assert result.disposition is Disposition.FAIL_SECURE, "hole reopened — see LIMITATIONS.md"
def test_exactly_one_route_to_a_mapping_is_expressible():
# Was: ALL FOUR routes failed, each on its own rule, so the mapping *class* had
# no expressible form (and v0.2's `generated` could not be written at all). G3
# opens exactly ONE of them - the allowlisted flow form - and the other three
# still fail, each on its own rule. That the openable route is the one whose
# every key the allowlist inspects is the whole design: block, dotted and inline
# give the allowlist nothing to inspect, so they stay shut.
def test_the_expressible_mapping_routes_are_the_ones_the_allowlist_inspects():
# Was `test_exactly_one_route_to_a_mapping_is_expressible` (1.2.0), and before
# that ALL FOUR routes failed so the mapping *class* had no expressible form.
# There are four expressible carriers as of 1.3.0 - flow mapping as a value,
# flow mapping as a list item, flow sequence of flow mappings, block sequence
# of block mappings - and the criterion that admits them is unchanged: each
# hands the allowlist every key. The routes below stay shut for the same
# reason, each on its own rule: a top-level block MAPPING (not a sequence),
# a dotted key, and an inline second colon give the allowlist nothing to
# inspect.
assert parse_frontmatter("---\nid: x\ngenerated: { by: x, at: y }\n---\n\nbody\n")[0][
"generated"] == {"by": "x", "at": "y"}
@ -695,19 +712,35 @@ def test_block_lists_admitted_by_item_shape(cid, fm, expected):
assert parse_frontmatter(f"---\nid: x\n{fm}---\n\nbody\n")[0][key] == expected
def test_two_keys_per_item_is_where_the_block_list_hard_rejects():
def test_the_block_list_rejects_on_the_key_set_not_on_arity():
# Was `test_two_keys_per_item_is_where_the_block_list_hard_rejects`: a
# two-key item was refused because the block mapping had no expressible form
# at all. G30 gives it one (SPEC.md §5.1's own carrier), so arity is no
# longer the boundary - the key set is. The same two-key item parses when its
# keys are §5.1's, and still raises when one of them is not.
parsed = parse_frontmatter(
"---\nid: x\nsources:\n - id: a\n resource: file://x\n---\n\nbody\n"
)[0]
assert parsed["sources"] == [{"id": "a", "resource": "file://x"}]
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(
"---\nid: x\nsources:\n - id: a\n resource: file://x\n---\n\nbody\n"
"---\nid: x\nsources:\n - id: a\n uri: file://x\n---\n\nbody\n"
)
@pytest.mark.parametrize("fm", [
# The flow row carries a key OFF the G3 allowlist: the shape is admitted, the
# key is not, so this stays a T2 rejection and the door A/B half still holds.
"generated: { by: x, tool: y }\n", "sources: [{ id: a }]\n", "tags: [a, b]\n",
# The two `sources` rows that lived here until 1.3.0 now PARSE - that is the
# G30 fix, not a weakening of this property. Their replacements are the same
# carriers with a key off the allowlist, so the row still measures what it
# says: the shape is admitted, the key set is not.
# The `tags` row moved the same way in 1.4.0: `[a, b]` now PARSES (P1), so
# the row carries the quoted form, which is still a T2 rejection.
"generated: { by: x, tool: y }\n", "sources: [{ id: a, uri: u }]\n", "tags: ['a', 'b']\n",
"generated:\n by: x\n", "generated.by: x\n",
"sources:\n - id: a\n resource: file://x\n",
"sources:\n - id: a\n uri: file://x\n",
"executor: [{ resource: skills/run.md }]\n",
])
def test_t2_constrains_import_not_emission(fm):
# T2 runs on door C only. The same frontmatter that FAIL_SECUREs through
@ -903,3 +936,331 @@ def test_a_conformant_v02_trust_layer_now_reaches_the_gate():
result = import_bundle({"tables/users.md": doc})
assert result.disposition is Disposition.WARN
assert result.concepts[0].error is None
# --- G30: the `sources` provenance layer becomes reachable (2026-09-02) ------
# Door 2. G3 gave the mapping *class* one expressible form but left `sources`
# unreachable: SPEC.md §5.1 writes an entry as a MAPPING carrying a REQUIRED
# `resource`, so neither of the two carriers the spec and the producers actually
# use could parse. Measured 02.09 by two consumers independently -- a flow
# sequence of flow mappings raised on the `[` indicator, a block sequence of
# block mappings raised "nested mappings are not supported".
#
# Why `resource` is admissible now when 1.2.0 argued it was not: the old
# argument was that the parser could not tell `sources[].resource` (§5.1, a
# citation) from `executor.resource` (§10, a code pointer). That premise was
# false -- the owning key is in scope at every call site, it was simply never
# threaded through. `resource` is allowlisted for `sources` entries ONLY, so
# the door-C routes above stay shut on the same input.
_SPEC_51_BLOCK = (
"sources:\n"
" - id: ga4-schema\n"
" resource: https://developers.google.com/analytics/bigquery/export-schema\n"
" title: GA4 BigQuery Export schema\n"
" author: team:ga4-docs\n"
" usage_count: 5000\n"
" last_modified: 2026-05-30T00:00:00Z\n"
)
def test_spec_sources_block_sequence_of_block_mappings_parses():
# SPEC.md §5.1's own example block, verbatim. It is the spec's canonical
# carrier for a REQUIRED field, so §11.1 ("parseable YAML frontmatter") makes
# a bundle written this way conformant -- refusing it refuses a conformant
# bundle, which is the failure mode G3 was opened to end.
fm, _ = parse_frontmatter(f"---\ntype: table\n{_SPEC_51_BLOCK}---\n\nbody\n")
assert fm["sources"] == [{
"id": "ga4-schema",
"resource": "https://developers.google.com/analytics/bigquery/export-schema",
"title": "GA4 BigQuery Export schema",
"author": "team:ga4-docs",
"usage_count": "5000",
"last_modified": "2026-05-30T00:00:00Z",
}]
@pytest.mark.parametrize("cid,fm,expected", [
("one entry",
"sources: [{ id: golden-v0-2-sales, resource: fixture }]\n",
[{"id": "golden-v0-2-sales", "resource": "fixture"}]),
("two entries",
"sources: [{ id: a, resource: https://e.com/a }, { id: b, resource: https://e.com/b }]\n",
[{"id": "a", "resource": "https://e.com/a"},
{"id": "b", "resource": "https://e.com/b"}]),
])
def test_sources_flow_sequence_of_flow_mappings_parses(cid, fm, expected):
# The form the producer emits today (llm-ingestion-okf's golden
# expected-bundle/ingest-sales.md, measured 02.09). One entry raised too, so
# this is not an arity bug: the `[` indicator refused the carrier outright.
assert parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")[0]["sources"] == expected
def test_the_two_sources_carriers_parse_to_the_same_value():
block = parse_frontmatter(
"---\ntype: t\nsources:\n - id: a\n resource: https://e.com/a\n---\n\nb\n")[0]
flow = parse_frontmatter(
"---\ntype: t\nsources: [{ id: a, resource: https://e.com/a }]\n---\n\nb\n")[0]
assert block["sources"] == flow["sources"] == [{"id": "a", "resource": "https://e.com/a"}]
@pytest.mark.parametrize("cid,fm", [
("unknown key, flow", "sources: [{ id: a, uri: https://e.com/a }]\n"),
("unknown key, block", "sources:\n - id: a\n uri: https://e.com/a\n"),
("unknown key, flow value", "sources: { id: a, kind: doc }\n"),
])
def test_an_unknown_key_in_a_sources_entry_is_still_rejected(cid, fm):
# The negative control. The carrier is admitted; the key set is not. A
# producer-invented key gets no free ride on the new shape.
with pytest.raises(OKFFrontmatterError) as exc:
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
assert "allowlist" in str(exc.value)
@pytest.mark.parametrize("cid,fm", [
("executor, block sequence", "executor:\n - resource: skills/run-on-bq.md\n"),
("executor, flow sequence", "executor: [{ resource: skills/run-on-bq.md }]\n"),
("attester, flow sequence", "attester: [{ resource: attesters/sql_equality.py }]\n"),
("attester, block sequence", "attester:\n - resource: attesters/sql_equality.py\n"),
])
def test_resource_is_allowlisted_for_sources_entries_only(cid, fm):
# The whole reason `resource` can be admitted at all: the owning key decides.
# §10's `executor.resource` / `attester.resource` name run instructions and
# code -- door C -- and stay refused through EVERY carrier, including the two
# this change opens for `sources`.
doc = f"---\nid: x\ntype: Attested Computation\n{fm}---\n\nbody\n"
with pytest.raises(OKFFrontmatterError) as exc:
parse_frontmatter(doc)
assert "allowlist" in str(exc.value)
assert import_bundle({"computations/x.md": doc}).disposition is Disposition.FAIL_SECURE
@pytest.mark.parametrize("cid,fm", [
# SPEC §5.1: "A single entry MAY carry its own `usage_window`". That is a
# mapping inside a mapping -- depth 2 -- and stays refused: no nesting deeper
# than one is a security property this change does not spend. Registered as a
# conformance gap in docs/LIMITATIONS.md, not as an oversight.
("per-entry usage_window, flow",
"sources: [{ id: a, usage_window: { from: x, to: y } }]\n"),
("per-entry usage_window, block",
"sources:\n - id: a\n usage_window: { from: x, to: y }\n"),
# A sequence inside an entry is the same depth violation.
("nested sequence", "sources: [{ id: a, title: [x, y] }]\n"),
])
def test_a_sources_entry_admits_scalar_leaves_only(cid, fm):
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
@pytest.mark.parametrize("cid,fm", [
("scalar then mapping", "sources:\n - https://e.com/a\n - id: b\n title: B\n"),
("mapping then scalar", "sources:\n - id: a\n title: A\n - https://e.com/b\n"),
])
def test_a_block_list_may_not_mix_scalars_and_mappings(cid, fm):
# A consumer that reads `entry.get("id")` over the list crashes on the str.
# One list, one item type -- refuse rather than hand back a mixed tree.
with pytest.raises(OKFFrontmatterError) as exc:
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
assert "mix" in str(exc.value)
@pytest.mark.parametrize("cid,fm", [
("empty flow sequence", "sources: []\n"),
("flow sequence, unclosed", "sources: [{ id: a }\n"),
("flow sequence, trailing junk", "sources: [{ id: a }] x\n"),
("flow sequence, nested sequence", "sources: [[ id ]]\n"),
])
def test_the_flow_sequence_refuses_a_carrier_it_cannot_read(cid, fm):
# What no element shape rescues: nothing at all, an unclosed mapping, junk
# between elements, and a sequence inside a sequence. The plain-scalar
# element admitted in 1.4.0 (P1) is a THIRD shape, not a loosening of these
# -- each row here still raises through it.
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
# --- P1: the flow sequence of plain scalars (operator decision, 2026-09-08) --
# `tags: [a, b, c]` is the ONE form SPEC.md §4.1's own skeleton writes out for
# `tags`, and the only candidate measured to move the upstream corpus at all:
# 0/53 -> 6/53 against the pinned reference bundles (`_okf-upstream` @ 3fcbb9f,
# denominator 53). The two neighbours measured 0/53 EACH and are deliberately
# NOT built -- P2 (a flush-left block sequence) and P3 (a folded plain scalar)
# would cost parser surface and buy no measured conformance on this corpus;
# see docs/2026-09-07-limitations-44-maaling.md §§7-8.
#
# It does not spend the depth-1 rule: the elements are leaves, the same depth
# the block list of scalars already carries. The binding constraint on the
# corpus is elsewhere and stays refused -- 44/53 stop on `generated` written as
# a top-level block mapping, which is a security decision, not this one.
@pytest.mark.parametrize("cid,fm,expected", [
("SPEC §4.1 skeleton", "tags: [finance, revenue, headline-metric]\n",
["finance", "revenue", "headline-metric"]),
("two elements", "tags: [a, b]\n", ["a", "b"]),
("one element", "tags: [solo]\n", ["solo"]),
("uneven spacing", "tags: [ a ,b ]\n", ["a", "b"]),
("one trailing comma, as the flow mapping already allows", "tags: [a, b,]\n", ["a", "b"]),
])
def test_a_flow_sequence_of_plain_scalars_parses(cid, fm, expected):
# The red test for P1: `[a, b]` must come back as a real list of strings,
# never a degraded string (the 1.1.0 defect) and never a refusal.
assert parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")[0]["tags"] == expected
def test_the_two_scalar_sequence_carriers_parse_to_the_same_value():
flow = parse_frontmatter("---\ntype: t\ntags: [a, b]\n---\n\nb\n")[0]
block = parse_frontmatter("---\ntype: t\ntags:\n - a\n - b\n---\n\nb\n")[0]
assert flow["tags"] == block["tags"] == ["a", "b"]
@pytest.mark.parametrize("cid,fm", [
# Quoting is the failure mode this shape is refused for elsewhere: the
# parser retains quotes rather than stripping them, so admitting a quoted
# element would hand back a value YAML reads differently.
("single-quoted element", "tags: ['a']\n"),
("double-quoted element", 'tags: ["a", "b"]\n'),
# Comma splitting is character-level, so anything that would need YAML
# semantics to split correctly is refused rather than guessed at.
("colon inside an element", "tags: [a: b]\n"),
("comment indicator", "tags: [a #b]\n"),
("brace inside an element", "tags: [a{b}]\n"),
("bracket inside an element", "tags: [a[b]]\n"),
# Depth: a sequence inside a sequence opens a second level and is refused
# on the same character rule, with no YAML semantics involved.
("nested flow sequence", "tags: [[a]]\n"),
("empty element", "tags: [a, , b]\n"),
("leading empty element", "tags: [, a]\n"),
("empty sequence", "tags: []\n"),
("unterminated sequence", "tags: [a, b\n"),
# The unchanged scalar indicators still apply to an element, exactly as they
# do to a block-list item: an anchor or an alias is not a plain scalar.
("anchor element", "tags: [&anchor]\n"),
("alias element", "tags: [*alias]\n"),
("explicit tag element", "tags: [!!python/object]\n"),
])
def test_a_flow_sequence_scalar_element_must_be_a_plain_scalar(cid, fm):
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
@pytest.mark.parametrize("cid,fm", [
("scalar then mapping", "tags: [a, {b: c}]\n"),
("mapping then scalar", "sources: [{ id: a }, plain]\n"),
])
def test_a_flow_sequence_may_not_mix_scalars_and_mappings(cid, fm):
# Same rule, same reason as the block list: a consumer iterating the value
# and reading `entry.get("id")` crashes on the first str. The mixing verdict
# is reached BEFORE the element is parsed, so it is what the caller sees --
# not an allowlist complaint about a key that was never the problem.
with pytest.raises(OKFFrontmatterError) as exc:
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
assert "mix" in str(exc.value)
@pytest.mark.parametrize("cid,fm,expected", [
("one entry", "sources: [{ id: x }]\n", [{"id": "x"}]),
("two entries", "sources: [{ id: a, title: A }, { id: b, title: B }]\n",
[{"id": "a", "title": "A"}, {"id": "b", "title": "B"}]),
])
def test_the_g30_flow_mapping_carrier_is_unchanged_by_the_scalar_element(cid, fm, expected):
# The no-regression pin. P1 adds a branch to the same function that carries
# `sources`; the mapping element must parse exactly as it did in 1.3.0.
assert parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")[0]["sources"] == expected
# The self-safety row for P1 (OWASP LLM10). The predicate is character-level
# with no regex, so there is no backtracking engine to blow up -- but "no regex"
# is an argument, and the sweep in docs/redos-sweep.py cannot check it, because
# it collects compiled patterns and this predicate compiles none. So it is
# measured instead, on the same CPU clock every other bound here uses. Over four
# doublings (50k -> 800k, 2026-09-08) the exponent is 0.86-0.99 in the element
# LENGTH and 0.97-1.05 in the element COUNT: linear in both, 0.66s at 800_000
# elements and 0.03s at an 800_000-character element. The refusal path is the
# same shape -- a forbidden character at the very end of an 800_000-character
# element is found in 0.04s -- because the element is scanned once per rule, not
# rescanned per start position.
_FLOW_SCALAR_PERF_N = 400_000
def test_a_long_flow_sequence_of_scalars_stays_bounded():
many = "---\ntype: t\ntags: [" + ", ".join(["ab"] * _FLOW_SCALAR_PERF_N) + "]\n---\nbody\n"
assert scan_seconds(parse_frontmatter, many) < 2.0
one_long = "---\ntype: t\ntags: [" + "a" * _FLOW_SCALAR_PERF_N + "]\n---\nbody\n"
assert scan_seconds(parse_frontmatter, one_long) < 2.0
def test_a_long_flow_sequence_element_is_refused_without_a_rescan():
# The refusal is the half an attacker controls: a forbidden character parked
# at the END of a long element is the worst case for any per-start rescan.
doc = "---\ntype: t\ntags: [" + "a" * _FLOW_SCALAR_PERF_N + ":]\n---\nbody\n"
def refuse(payload):
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(payload)
assert scan_seconds(refuse, doc) < 2.0
def test_injection_in_a_sources_entry_leaf_is_caught_by_the_scan():
# T1 over the new shape: every leaf of every entry reaches scan_output.
doc = f"---\ntype: table\nsources: [{{ id: a, title: {_INJECTION} }}]\n---\nclean\n"
assert scan_concept(doc).found is True
doc_block = f"---\ntype: table\nsources:\n - id: a\n title: {_INJECTION}\n---\nclean\n"
assert scan_concept(doc_block).found is True
def test_the_producer_golden_now_passes_the_gate():
# llm-ingestion-okf's expected-bundle/ingest-sales.md, the K5 blocker.
doc = (
"---\n"
"type: dataset\n"
"title: Regional Sales\n"
"source_system: golden-v0-2-sales\n"
"ingested_at: 2026-07-16T12:00:00Z\n"
"generated: { by: process:okf-ingest, at: 2026-07-16T12:00:00Z }\n"
"sources: [{ id: golden-v0-2-sales, resource: fixture }]\n"
"---\n\n| region | units |\n| --- | --- |\n| nord | 412 |\n"
)
result = import_bundle({"datasets/sales.md": doc})
assert result.disposition is Disposition.WARN
assert result.concepts[0].error is None
@pytest.mark.parametrize("cid,fm", [
# A `- ` item whose text begins with a YAML indicator AND carries a `": "`
# must fall through to the scalar predicates, not into the block-mapping
# route with a half-validated key. Each of these fails `_KEY_RE` on the key
# side, so `_reject_dangerous_value` gets the item intact.
("anchor", "sources:\n - &anchor id: a\n"),
("alias", "sources:\n - *anchor id: a\n"),
("tag", "sources:\n - !!str id: a\n"),
("directive", "sources:\n - %YAML id: a\n"),
("reserved", "sources:\n - `x id: a\n"),
("merge key", "sources:\n - id: a\n <<: *base\n"),
])
def test_an_indicator_in_a_block_item_is_refused_before_the_mapping_route(cid, fm):
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(f"---\ntype: t\n{fm}---\n\nbody\n")
@pytest.mark.parametrize("cid,fm", [
# The block mapping ends at a blank line and at a line in column zero. Both
# hand control back with an index that must not skip or re-read a line: a
# dangling `resource:` line left over from a mapping that closed early must
# RAISE, never be silently dropped -- a pointer that vanishes rather than
# failing is exactly this repo's failure class.
("blank line inside the mapping", "sources:\n - id: a\n\n resource: b\n"),
("top-level key interleaved", "sources:\n - id: a\ntags: x\n resource: b\n"),
])
def test_a_line_orphaned_by_the_mapping_boundary_raises_rather_than_vanishing(cid, fm):
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(f"---\ntype: t\n{fm}---\n\nbody\n")
def test_the_block_list_hands_back_an_index_that_resumes_at_the_next_key():
# The return-index contract: a top-level key following a multi-entry block
# list is neither swallowed by the list nor re-read as a list item.
fm, _ = parse_frontmatter(
"---\ntype: t\nsources:\n - id: a\n resource: b\n - id: c\ntitle: T\n---\n\nbody\n")
assert fm == {"type": "t", "sources": [{"id": "a", "resource": "b"}, {"id": "c"}],
"title": "T"}