Compare commits
3 commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 44e2b31afd | |||
| a965e8ac5b | |||
| 0184df9ed9 |
10 changed files with 609 additions and 101 deletions
52
CHANGELOG.md
52
CHANGELOG.md
|
|
@ -5,6 +5,58 @@ All notable changes to this project will be documented in this file.
|
|||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [1.3.0] — 2026-09-02
|
||||
|
||||
### Added — the `sources` provenance family becomes expressible in both spec carriers
|
||||
|
||||
`okf.parse_frontmatter` rejected `sources` in every form the OKF spec and its
|
||||
producers actually use. Measured 2026-09-02 by two consumers independently:
|
||||
`sources: [{ id: a, resource: x }]` raised on the `[` indicator (with one entry
|
||||
as with two), and the block sequence of block mappings — SPEC.md §5.1's *own*
|
||||
example — raised `"nested mappings are not supported"`. `resource` is REQUIRED
|
||||
within a `sources` entry (§5.1), so the whole provenance family was unwritable,
|
||||
and a bundle written the way the spec documents it was refused.
|
||||
|
||||
Admitted now, both parsing to the same value (a list of dicts):
|
||||
|
||||
```yaml
|
||||
sources: [{ id: a, resource: https://e.com/a }]
|
||||
sources:
|
||||
- id: a
|
||||
resource: https://e.com/a
|
||||
```
|
||||
|
||||
**`resource` is allowlisted inside a `sources` entry and nowhere else.** 1.2.0
|
||||
left it off the allowlist on the argument that the parser could not tell
|
||||
`sources[].resource` (§5.1, a citation) from `executor.resource` /
|
||||
`attester.resource` (§10, a pointer to code to be run — the door-C route closed
|
||||
in 1.1.0). That premise was false: the owning key is in scope at every call
|
||||
site and was simply never threaded through. It is threaded now, so the
|
||||
discrimination is structural rather than a judgement about the value, and
|
||||
`executor: [{ resource: skills/run.md }]` and `attester:\n - resource: …` are
|
||||
refused on the allowlist through *every* carrier, including the two this adds.
|
||||
|
||||
Refusal stays the default everywhere else. A flow sequence of plain scalars
|
||||
(`tags: [a, b]`) still raises — the sequence carrier is opened for the flow
|
||||
mapping element and nothing else. A `sources` entry still admits scalar leaves
|
||||
only, so SPEC §5.1's optional *per-entry* `usage_window` (a mapping inside a
|
||||
mapping) is refused: no nesting past depth 1 is a security property, and it was
|
||||
not spent here. Registered as a conformance gap in `docs/LIMITATIONS.md`. A
|
||||
block list may not mix scalar items and mappings. Off-allowlist keys, anchors,
|
||||
aliases, tags, duplicate keys and unclosed collections raise as before, and a
|
||||
refused mapping still raises rather than degrading into a string.
|
||||
|
||||
**New residual, registered:** `sources[].resource` is scanned as text (T1) but
|
||||
never validated as a URL. T3's https allowlist cannot be extended to it without
|
||||
over-blocking conformant bundles — §5.1 permits a bundle-relative path or a
|
||||
scope descriptor a consumer cannot follow at all. A consumer that dereferences
|
||||
it must call `okf.validate_resource_url` itself.
|
||||
|
||||
No exported surface changed and no detector behaviour or calibration changed.
|
||||
Suite 834 → 868 (34 new rows, plus four pre-existing rows updated where this
|
||||
release changed the behaviour they pinned); 130/130 classes, 6/6 documented
|
||||
gaps hold, 45 limitations, ReDoS sweep 0/152 candidates flagged.
|
||||
|
||||
## [1.2.0] — 2026-08-23
|
||||
|
||||
### Added — OKF frontmatter can express one mapping form: typed and allowlisted
|
||||
|
|
|
|||
25
CLAUDE.md
25
CLAUDE.md
|
|
@ -11,18 +11,27 @@ framework-agnostisk kode.
|
|||
Referanse-implementasjon: `claude-code-llm-wiki` Stage B (`tools/wiki_ingest/`).
|
||||
Lexikon-seed: `injection-patterns.mjs` fra `llm-security`-pluginen.
|
||||
|
||||
Repoet er på **v1.2.0** — den eksporterte Python-surfacen er frosset under semver
|
||||
Repoet er på **v1.3.0** — den eksporterte Python-surfacen er frosset under semver
|
||||
(deteksjonsatferd er det IKKE; kalibrering flytter seg i 1.x). Stdlib-kjernen er
|
||||
bygget og testet (15 moduler +
|
||||
topp-nivå wiring, showcase + korpus), inkl. OKF-adapter og aktivt-innhold-
|
||||
detektor (EchoLeak-klassen) i output-gaten. OKF-frontmatterens mapping-klasse
|
||||
har **én** uttrykkbar form (G3, 21.08): en flow-mapping (`generated: { by: x, at: y }`) — som verdi eller som blokkliste-
|
||||
element — der HVER nøkkel står på en ni-navns allowlist og hvert blad er en ren
|
||||
skalar. Formen er trygg fordi allowlisten inspiserer hver nøkkel; det blanke
|
||||
avslaget var håndhevelsen, ikke poenget. `resource` er bevisst UTE av
|
||||
allowlisten (peker, ikke etikett — den ene nøkkelen T3 finnes for). Blokk-,
|
||||
dotted- og inline-kolon-rutene raiser fortsatt, og en avvist mapping raiser —
|
||||
den degraderer aldri til en streng (1.1.0-defekten). Mode-b `import_bundle` skanner
|
||||
har **fire** uttrykkbare bærere (G3 21.08, G30 02.09): flow-mapping som verdi
|
||||
og som blokkliste-element, flow-sekvens av flow-mappinger, og blokk-sekvens av
|
||||
blokk-mappinger (SPEC §5.1s egen form). HVER nøkkel i alle fire står på
|
||||
allowlisten og hvert blad er en ren skalar. Formen er trygg fordi allowlisten
|
||||
inspiserer hver nøkkel; det blanke avslaget var håndhevelsen, ikke poenget.
|
||||
**`resource` er allowlistet KUN inne i en `sources`-oppføring** — foreldre-
|
||||
nøkkelen avgjør, så `executor`/`attester` sin `resource` (§10, dør C) avvises
|
||||
gjennom hver eneste bærer. 1.2.0s begrunnelse for å utelate den (parseren
|
||||
manglet foreldre-kontekst) var målt feil: konteksten var der, den var bare
|
||||
aldri tredd gjennom. Topp-nivå blokk-mapping, dotted- og inline-kolon-rutene
|
||||
raiser fortsatt, flow-sekvens av rene skalarer (`tags: [a, b]`) raiser, en
|
||||
blokkliste kan ikke blande skalarer og mappinger, og en avvist mapping raiser —
|
||||
den degraderer aldri til en streng (1.1.0-defekten). `sources[].resource`
|
||||
URL-valideres ALDRI (T3 ser kun topp-nivå `resource`) — §5.1 tillater
|
||||
bundle-relative stier og scope-beskrivelser, så en https-gate ville over-blokkert
|
||||
konforme bundles; konsumenten må selv kalle `validate_resource_url`. Mode-b `import_bundle` skanner
|
||||
reserverte strukturfiler (`index.md`/`log.md`) i mottatte bundles i stedet for å
|
||||
path-avvise dem; upload-front-end beholder shadow-reject (`allow_reserved=False`).
|
||||
Output-gatens decode-and-rescan mater dekodet base64-klartekst gjennom BÅDE lexicon
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
Write-time defensive layer for Python pipelines that persist LLM output: sanitize, fence, tool-less quarantined transform, capability isolation, scan before persist, fail-secure.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
@ -33,7 +33,7 @@ at write time, never assumed from the format. Any pipeline ingesting external da
|
|||
into an agent-read store has this shape; an OKF wiki is its canonical form — which
|
||||
is why the guard ships a first-class OKF adapter (below).
|
||||
|
||||
**Status:** `v1.2.0`. The stdlib-only core — its detector, contract, and
|
||||
**Status:** `v1.3.0`. The stdlib-only core — its detector, contract, and
|
||||
OKF-adapter modules plus the top-level wiring — is built and tested, exercised by
|
||||
an end-to-end showcase and adversarial + false-positive corpora. The exported
|
||||
Python surface is now frozen under semver: nothing exported is removed, renamed or
|
||||
|
|
@ -58,7 +58,7 @@ are real limitations, stated plainly below; read them.
|
|||
Not on PyPI. The guard is distributed from its Forgejo origin — pin a release tag:
|
||||
|
||||
```bash
|
||||
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v1.2.0"
|
||||
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v1.3.0"
|
||||
```
|
||||
|
||||
The `open/` mirror is anonymously readable, so CI needs no deploy key, token, or
|
||||
|
|
@ -269,7 +269,7 @@ a green scan means safe content. The highest-impact items:
|
|||
two of the three corpora are living, so the cells are not rewritten in place.
|
||||
Method and before/after: [`docs/rawhtml-census.py`](docs/rawhtml-census.py).
|
||||
|
||||
**Full list — 43 items, each with the mechanism, plus the out-of-scope boundary:**
|
||||
**Full list — 45 items, each with the mechanism, plus the out-of-scope boundary:**
|
||||
[`docs/LIMITATIONS.md`](docs/LIMITATIONS.md). Several carry field measurements from
|
||||
consumer corpora, including the false positives the URL-shape rule actually produces.
|
||||
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@
|
|||
especially one converging on Google's Open Knowledge Format (OKF v0.1) — and needs
|
||||
to decide **when** and **where** to add a write-time ingestion guard.
|
||||
|
||||
**Status of the guard:** `v1.2.0`. Stdlib-only core, framework-agnostic. The
|
||||
**Status of the guard:** `v1.3.0`. Stdlib-only core, framework-agnostic. The
|
||||
exported Python surface is frozen under semver — nothing exported is removed,
|
||||
renamed or given a different meaning without a `2.0.0`. Detection behaviour is
|
||||
*not* frozen: severities, thresholds and lexicon entries are calibration and move
|
||||
|
|
@ -142,9 +142,9 @@ live payload:
|
|||
python -m llm_ingestion_guard.coverage # exit 0 = all as documented
|
||||
```
|
||||
|
||||
As of `v1.2.0`: **130 / 130 defended classes demonstrated (recall 100%)** and **6 /
|
||||
As of `v1.3.0`: **130 / 130 defended classes demonstrated (recall 100%)** and **6 /
|
||||
6 documented gaps still hold** (a *closed* gap fails the test, forcing a doc
|
||||
update). The matrix is the single source of truth for the test suite (**834
|
||||
update). The matrix is the single source of truth for the test suite (**868
|
||||
passing**), which also asserts total recall, that every lexicon pattern has a
|
||||
case (so the matrix cannot fall behind the lexicon), the full LLM02 secret-egress
|
||||
set, and the container-layer front-end (CSV formula-injection, zip-slip/bomb,
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
**A reusable, minimal, dependency-light defensive layer for LLM *ingestion*
|
||||
pipelines — the write-time siblings of query-time chatbot guardrails.**
|
||||
|
||||
Status: implemented — v1.2.0, exported surface frozen under semver. This document defines what the repo contains
|
||||
Status: implemented — v1.3.0, exported surface frozen under semver. This document defines what the repo contains
|
||||
and why; the stdlib-only core is built and tested (see `README.md` for usage and
|
||||
`docs/PLAN.md` for the build order).
|
||||
|
||||
|
|
|
|||
|
|
@ -40,27 +40,31 @@ items; this is the full list, each with the mechanism.
|
|||
(an injection in a directory listing is caught) rather than path-rejecting the
|
||||
conformant bundle. A front-end materialising individual uploads keeps the opposite
|
||||
rule (`allow_reserved=False`): a reserved basename is a listing-shadow and refused.
|
||||
- **OKF frontmatter is a restricted grammar: the mapping class has exactly one
|
||||
expressible form.** Gate T2 accepts a line-oriented subset deliberately — full YAML
|
||||
is a larger parse-attack surface than a write-time gate needs. Flow sequences
|
||||
(`[a, b]`) and nested mappings are *rejected outright*, which fails secure.
|
||||
**Three of the four routes to a mapping fail, each on a different rule** — block
|
||||
(`k:\n sub: v`) on the nested-mapping check, dotted keys (`k.sub: v`) on the key
|
||||
pattern, and the inline second colon (`k: sub: v`) on the mapping-construct check.
|
||||
**The fourth, the flow form, is admitted only when every key is on an allowlist**
|
||||
- **OKF frontmatter is a restricted grammar: a mapping is expressible through
|
||||
four carriers, all of them key-allowlisted.** Gate T2 accepts a line-oriented
|
||||
subset deliberately — full YAML is a larger parse-attack surface than a
|
||||
write-time gate needs. The admitted carriers are the flow mapping as a value
|
||||
(`generated: { by: x, at: y }`), the flow mapping as a block-list item, the
|
||||
flow *sequence* of flow mappings (`sources: [{ id: a, resource: x }]`) and the
|
||||
block sequence of block mappings (SPEC.md §5.1's own form). **The routes that
|
||||
still fail, each on a different rule:** a top-level block *mapping*
|
||||
(`k:\n sub: v`) on the nested-mapping check, dotted keys (`k.sub: v`) on the
|
||||
key pattern, the inline second colon (`k: sub: v`) on the mapping-construct
|
||||
check, and a flow sequence of plain *scalars* (`tags: [a, b]`) on the `[`
|
||||
indicator — the sequence carrier is opened for the mapping element and nothing
|
||||
else. **Every carrier is admitted only when every key is on an allowlist**
|
||||
(`by`, `at`, `from`, `to`, `id`, `title`, `author`, `usage_count`,
|
||||
`last_modified` — the keys SPEC.md @ `62432a09` §5.1/§5.2 names inside a mapping)
|
||||
`last_modified` — the keys SPEC.md @ `62432a09` §5.1/§5.2 names inside a mapping,
|
||||
plus `resource` and `usage_window` *under `sources` only*, see below)
|
||||
and every leaf is a plain scalar, itself run through the same value predicates as a
|
||||
top-level scalar. Nested collections, quoted leaves, duplicate keys, an empty or
|
||||
unclosed mapping, and `{a:b}` (which PyYAML 6.0.3 reads as the *key* `a:b`, not as
|
||||
a scalar) all raise. The form is expressible, never trusted: the allowlist
|
||||
inspects every key, which is the property that carried the security when the
|
||||
blanket refusal was doing the enforcing. **`resource` is deliberately off the
|
||||
allowlist** although §5.1 names it inside a `sources` entry — it is a pointer
|
||||
rather than a label and the only key T3 exists for, so admitting it would let
|
||||
`executor: { resource: skills/run.md }` carry an executable-code pointer through a
|
||||
key the https allowlist never inspects. What else survives is scalars and flat
|
||||
lists of strings. **Two routes used to
|
||||
blanket refusal was doing the enforcing. **A block list may not mix scalar items
|
||||
and mappings** — YAML permits it, but a consumer iterating `sources` and reading
|
||||
`entry.get("id")` gets an `AttributeError` off the first `str`. What else
|
||||
survives is scalars and flat lists of strings. **Two routes used to
|
||||
degrade into a string instead of failing, and that defect is closed in `1.1.0`**:
|
||||
a block-sequence item carrying exactly one key (`sources:\n - uri: https://e.com/a`
|
||||
yielded the *string* `'uri: https://e.com/a'`) and the inline second colon
|
||||
|
|
@ -85,19 +89,58 @@ items; this is the full list, each with the mechanism.
|
|||
door A/B persist path, so frontmatter that fails secure on import passes
|
||||
`screen_output` unremarked. The grammar therefore bounds what a consumer can *receive*,
|
||||
never what a producer can *emit*. Verified identical on 0.2.0 and 0.3.1.
|
||||
- **An OKF v0.2 concept traverses the external-import path only if its `sources` are
|
||||
flat.** The wall used to be total: both of v0.2's backward-breaking migration targets
|
||||
are mappings — `timestamp` → `generated.at`, and body `# Citations` → a `sources`
|
||||
block list of mappings — and a consumer measured **0 of 53** upstream concepts
|
||||
through the gate. The trust and provenance layer now passes in its spec form
|
||||
(`generated`, `verified` bare or listed, `usage_window`), so `generated.at` is no
|
||||
longer a wall. **`sources` still is**: SPEC.md writes each entry as a block mapping
|
||||
under a block sequence (`- id: …\n resource: …`), and that carrier stays refused —
|
||||
it is the shape whose one-key degradation smuggled a pointer before `1.1.0`, and
|
||||
reopening it is a separate parse-safety decision, not a corollary of the flow form.
|
||||
A concept whose `sources` are flat strings, or absent, imports. The
|
||||
dangling-or-substituted `executor`/`attester` pointer question stays out of reach
|
||||
for the same reason: both are mappings whose payload key is `resource`.
|
||||
- **`sources` passes in both of its spec carriers; the per-entry `usage_window`
|
||||
does not.** The wall used to be total: both of v0.2's backward-breaking migration
|
||||
targets are mappings — `timestamp` → `generated.at`, and body `# Citations` → a
|
||||
`sources` block list of mappings — and a consumer measured **0 of 53** upstream
|
||||
concepts through the gate. `generated.at` stopped being a wall in `1.2.0`;
|
||||
`sources` stopped being one in `1.3.0`, which admits both the block sequence of
|
||||
block mappings (SPEC.md §5.1's own example) and the flow sequence of flow
|
||||
mappings (the form the OKF producers emit, measured 2026-09-02 by two consumers
|
||||
independently). **`resource` is allowlisted inside a `sources` entry and nowhere
|
||||
else.** `1.2.0` left it off on the argument that the parser could not tell
|
||||
`sources[].resource` (§5.1, a citation) from `executor.resource` / `attester.resource`
|
||||
(§10, a pointer to code to be run — door C). That premise was measured false: the
|
||||
owning key is in scope at every call site and was simply never threaded through.
|
||||
It is threaded now, so `executor: [{ resource: skills/run.md }]` and
|
||||
`attester:\n - resource: …` are refused on the allowlist through *every* carrier,
|
||||
including the two this opened. **What stays refused: a `usage_window` inside a
|
||||
`sources` entry.** SPEC §5.1 permits it per entry ("A single entry MAY carry its
|
||||
own `usage_window`"), and it is a mapping inside a mapping — depth 2, which this
|
||||
parser admits at no key. A bundle using the per-entry override is refused; the
|
||||
shared sibling `usage_window` (the §5.1 example's own form) passes. This is a
|
||||
registered conformance gap, not an oversight: no-nesting-past-depth-1 is a
|
||||
security property, and spending it was not what the fix was for.
|
||||
- **`sources[].resource` is scanned as text but never validated as a URL.** T3's
|
||||
https allowlist inspects the *top-level* `resource` and nothing else. It cannot
|
||||
be extended to `sources` entries without over-blocking conformant bundles: SPEC
|
||||
§5.1 explicitly permits a bundle-relative path, a path into `references/`, or a
|
||||
scope descriptor a consumer cannot follow at all (the OKF producers' own golden
|
||||
bundle emits `resource: fixture`). So a `sources` entry may carry
|
||||
`file://`, `javascript:` or any other string; it goes through T1's scan like any
|
||||
other frontmatter value, and nothing else. **A consumer that dereferences
|
||||
`sources[].resource` must validate it itself** — `okf.validate_resource_url` is
|
||||
exported for exactly that. The dangling-or-substituted `executor`/`attester`
|
||||
pointer question stays out of reach separately: both are top-level block
|
||||
mappings, a carrier that is still refused.
|
||||
- **`tags` and `description` block the OKF import corpus universally, before the
|
||||
trust layer is even reached.** The line-flat frontmatter parser has no
|
||||
sequence-value type at all: `tags` is present in 53/53 upstream concept
|
||||
documents — 9/53 as a flow sequence (`[a, b, c]`, rejected on the `[`
|
||||
indicator) and 44/53 as a block sequence (`- a` / `- b`, rejected as
|
||||
`"malformed frontmatter line"`) — 100% rejection regardless of form.
|
||||
`description` is present in 53/53; 29/53 is a folded plain scalar continuing
|
||||
on an indented second line, which the parser has no continuation-line model
|
||||
for and misreads as `"nested mappings are not supported"` (the remaining
|
||||
24/53 are single-line and parse fine). Measured directly on the upstream
|
||||
reference bundles (`_okf-upstream/okf` @ `3fcbb9f`): removing `tags` alone
|
||||
lets 4/53 documents pass; removing both `tags` and `description` together
|
||||
(trust layer untouched) lets the same 4/53 pass, and all four then parse
|
||||
`generated` correctly as a mapping. **Independent of the mapping-form work
|
||||
above:** neither `1.2.0`'s flow mapping nor `1.3.0`'s `sources` carriers move
|
||||
anything on this corpus, because `tags`/`description` reject before `sources`
|
||||
is ever read. No sequence-value type or continuation-line model exists in the
|
||||
stdlib-only parser to close this with.
|
||||
- **A persist gate cannot cover execution risk.** OKF v0.2 introduces concepts whose
|
||||
purpose is to *name code to be run* (`runtime`, `executor.resource`,
|
||||
`attester.resource`). This library answers "is this safe to **store**"; executable
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||
|
||||
[project]
|
||||
name = "llm-ingestion-guard"
|
||||
version = "1.2.0"
|
||||
version = "1.3.0"
|
||||
description = "Write-time defensive layer for Python pipelines that persist LLM output: sanitize, fence, tool-less quarantined transform, capability isolation, scan before persist, fail-secure."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
|
|
|
|||
|
|
@ -63,7 +63,7 @@ from .grounding import (
|
|||
)
|
||||
from . import okf
|
||||
|
||||
__version__ = "1.2.0"
|
||||
__version__ = "1.3.0"
|
||||
|
||||
|
||||
# --- §6 bookends: the two library-side halves around the transform ---------
|
||||
|
|
|
|||
|
|
@ -65,32 +65,49 @@ _KEY_RE = re.compile(r"^[A-Za-z0-9_][A-Za-z0-9_-]*$")
|
|||
# A plain OKF scalar cannot *begin* with a YAML structural indicator. Any value
|
||||
# starting with one signals an anchor (&), alias (*), explicit tag (!), block
|
||||
# scalar (|, >), flow collection ([ ] { }), directive (%) or reserved char
|
||||
# (@ `) — all outside the supported subset and all rejected. `{` is tried as the
|
||||
# allowlisted mapping form FIRST (G3); it reaches this predicate only as a leaf
|
||||
# inside one, where a nested collection is refused before it can be read.
|
||||
# (@ `) — all outside the supported subset and all rejected. `{` and `[` are
|
||||
# tried as the allowlisted mapping form (G3) and the flow sequence of them (G30)
|
||||
# FIRST; they reach this predicate only as a leaf inside one, where a nested
|
||||
# collection is refused before it can be read.
|
||||
_DANGEROUS_VALUE_STARTS = frozenset("&*!|>[]{}%@`")
|
||||
# A quoted scalar is a scalar in YAML however many colons it carries, so the
|
||||
# mapping check steps aside for one. The quotes are retained rather than
|
||||
# stripped — a pre-existing divergence, pinned in tests/test_okf.py.
|
||||
_QUOTE_STARTS = frozenset("\"'")
|
||||
|
||||
# G3 — the one mapping form T2 can express (operator decision, 2026-08-21).
|
||||
# G3 - the one mapping form T2 can express (operator decision, 2026-08-21).
|
||||
# Every key inside a mapping must be on this allowlist: the form is safe because
|
||||
# the allowlist inspects each key, not because mappings became trusted. The keys
|
||||
# are the ones OKF v0.2 names inside a mapping - `by`/`at` (SPEC.md @ 62432a09
|
||||
# §5.2 `generated`/`verified`), `from`/`to` (§5.1 `usage_window`) and the
|
||||
# `sources`-entry fields (§5.1). `resource` is the one §5.1 key deliberately
|
||||
# LEFT OFF: it is a pointer rather than a label, it is the only key T3 exists
|
||||
# for, and admitting it inside a mapping would re-open the door-C route closed
|
||||
# in 1.1.0 (`executor: {resource: skills/run.md}` puts an executable-code
|
||||
# pointer in a key the https allowlist never inspects). It costs nothing today,
|
||||
# because the conformant carrier for `sources[].resource` is the block-sequence
|
||||
# of block-mappings, which this form does not admit either way.
|
||||
# §5.2 `generated`/`verified`) and `from`/`to` (§5.1 `usage_window`), plus the
|
||||
# §5.1 `sources`-entry labels.
|
||||
_MAPPING_KEY_ALLOWLIST = frozenset({
|
||||
"by", "at", "from", "to", "id", "title", "author", "usage_count",
|
||||
"last_modified",
|
||||
})
|
||||
|
||||
# G30 - the two §5.1 keys admitted inside a `sources` entry and NOWHERE else
|
||||
# (operator decision, 2026-09-02). `resource` is REQUIRED within a `sources`
|
||||
# entry, so leaving it off left the whole provenance family unwritable; but the
|
||||
# same field name in §10 (`executor.resource`, `attester.resource`) names run
|
||||
# instructions and code - the door-C route closed in 1.1.0. 1.2.0 argued the
|
||||
# parser could not tell the two apart without parent-key context it did not
|
||||
# have. That premise was false: the owning key is in scope at every call site
|
||||
# below, it was simply never threaded through. It is threaded now, so the
|
||||
# discrimination is structural rather than a judgement about the value.
|
||||
# `usage_window` is allowlisted here for accuracy of refusal - §5.1 permits it
|
||||
# per entry, and it is then refused on the depth rule (a mapping inside a
|
||||
# mapping, which this parser admits at no key) rather than refused as if the
|
||||
# key were unknown.
|
||||
_SOURCES_ENTRY_KEYS = frozenset({"resource", "usage_window"})
|
||||
|
||||
|
||||
def _allowed_mapping_keys(parent_key):
|
||||
"""The mapping-key allowlist for a mapping owned by ``parent_key``."""
|
||||
if parent_key == "sources":
|
||||
return _MAPPING_KEY_ALLOWLIST | _SOURCES_ENTRY_KEYS
|
||||
return _MAPPING_KEY_ALLOWLIST
|
||||
|
||||
|
||||
class OKFError(Exception):
|
||||
"""Base class for OKF adapter rejections."""
|
||||
|
|
@ -184,8 +201,10 @@ def _value_regions(value):
|
|||
A mapping value (G3) is a new *shape* on this surface, not a new exemption:
|
||||
its leaves are scanned exactly like a scalar or a list item, so an injection
|
||||
parked in ``generated: { by: ... }`` reaches ``scan_output`` like any other
|
||||
frontmatter text. Mapping *keys* are not scanned because they cannot carry
|
||||
attacker text - the allowlist admits nine fixed names and nothing else.
|
||||
frontmatter text. The same holds for a *list* of mappings (G30, ``sources``),
|
||||
which this function already flattens through its list branch. Mapping *keys*
|
||||
are not scanned because they cannot carry attacker text - the allowlist
|
||||
admits a fixed, per-parent name set and nothing else.
|
||||
"""
|
||||
if isinstance(value, dict):
|
||||
return [leaf for leaf in value.values() if leaf]
|
||||
|
|
@ -616,16 +635,22 @@ def _parse_flat(fm_lines):
|
|||
raise OKFFrontmatterError("invalid frontmatter key: %r" % key)
|
||||
|
||||
if value == "":
|
||||
items, i = _consume_block_list(fm_lines, i + 1)
|
||||
items, i = _consume_block_list(fm_lines, i + 1, key)
|
||||
result[key] = items if items is not None else ""
|
||||
continue
|
||||
|
||||
mapping = _parse_flow_mapping(value)
|
||||
mapping = _parse_flow_mapping(value, key)
|
||||
if mapping is not None:
|
||||
result[key] = mapping
|
||||
i += 1
|
||||
continue
|
||||
|
||||
sequence = _parse_flow_sequence(value, key)
|
||||
if sequence is not None:
|
||||
result[key] = sequence
|
||||
i += 1
|
||||
continue
|
||||
|
||||
_reject_dangerous_value(value)
|
||||
_reject_mapping_construct(value)
|
||||
result[key] = value
|
||||
|
|
@ -634,14 +659,26 @@ def _parse_flat(fm_lines):
|
|||
return result
|
||||
|
||||
|
||||
def _consume_block_list(fm_lines, start):
|
||||
def _consume_block_list(fm_lines, start, parent_key=None):
|
||||
"""Consume `` - item`` lines following a bare ``key:``.
|
||||
|
||||
Returns ``(items, next_index)`` — ``items`` is ``None`` (and ``next_index``
|
||||
Returns ``(items, next_index)`` - ``items`` is ``None`` (and ``next_index``
|
||||
unchanged) when no list item follows, so the caller can treat the key as an
|
||||
empty scalar and let the next line trip the nested-structure guard.
|
||||
|
||||
An item is one of three shapes, decided by the item text alone: a flow
|
||||
mapping (G3), a block mapping (G30 - an unquoted ``key: value`` opening a
|
||||
run of more-indented sibling entries, which is SPEC.md §5.1's own carrier
|
||||
for ``sources``), or a plain scalar. ``parent_key`` is the key that owns the
|
||||
list; it decides the mapping-key allowlist, which is how ``sources[].resource``
|
||||
is admitted while ``executor``/``attester`` ``resource`` stays refused.
|
||||
|
||||
A list may not mix scalars and mappings. YAML permits it, but a consumer
|
||||
iterating ``sources`` and reading ``entry.get("id")`` gets an
|
||||
``AttributeError`` off the first ``str`` - refusing is the cheaper failure.
|
||||
"""
|
||||
items = []
|
||||
kinds = set()
|
||||
i = start
|
||||
n = len(fm_lines)
|
||||
while i < n:
|
||||
|
|
@ -650,24 +687,113 @@ def _consume_block_list(fm_lines, start):
|
|||
if stripped == "" or stripped.startswith("#"):
|
||||
i += 1
|
||||
continue
|
||||
if raw[:1] in (" ", "\t") and stripped.startswith("- "):
|
||||
item = stripped[2:].strip()
|
||||
mapping = _parse_flow_mapping(item)
|
||||
if mapping is not None:
|
||||
items.append(mapping)
|
||||
i += 1
|
||||
continue
|
||||
_reject_dangerous_value(item)
|
||||
_reject_mapping_construct(item)
|
||||
items.append(item)
|
||||
if not (raw[:1] in (" ", "\t") and stripped.startswith("- ")):
|
||||
break
|
||||
|
||||
item = stripped[2:].strip()
|
||||
|
||||
mapping = _parse_flow_mapping(item, parent_key)
|
||||
if mapping is not None:
|
||||
items.append(mapping)
|
||||
kinds.add("mapping")
|
||||
i += 1
|
||||
continue
|
||||
break
|
||||
|
||||
entry = _block_mapping_entry(item)
|
||||
if entry is not None:
|
||||
mapping, i = _consume_block_mapping(fm_lines, i + 1, entry, parent_key)
|
||||
items.append(mapping)
|
||||
kinds.add("mapping")
|
||||
continue
|
||||
|
||||
_reject_dangerous_value(item)
|
||||
_reject_mapping_construct(item)
|
||||
items.append(item)
|
||||
kinds.add("scalar")
|
||||
i += 1
|
||||
|
||||
if len(kinds) > 1:
|
||||
raise OKFFrontmatterError(
|
||||
"a block list may not mix scalar items and mappings: %r" % (parent_key,)
|
||||
)
|
||||
if not items:
|
||||
return None, start
|
||||
return items, i
|
||||
|
||||
|
||||
def _block_mapping_entry(text):
|
||||
"""Read ``text`` as one ``key: value`` block-mapping entry, or return ``None``.
|
||||
|
||||
The trigger is deliberately the same shape ``_reject_mapping_construct``
|
||||
uses to *refuse* a scalar: an unquoted ``": "``. What changes in 1.3.0 is
|
||||
only what happens next - the entry is admitted key-by-key against the
|
||||
allowlist instead of refused wholesale. Every shape that is a scalar to
|
||||
PyYAML stays one here: a quoted item, a colon with no space
|
||||
(``domain:security``, ``https://e.com:8443/a``) and a trailing colon all
|
||||
return ``None`` and fall through to the unchanged scalar rules.
|
||||
"""
|
||||
if not text or text[0] in _QUOTE_STARTS:
|
||||
return None
|
||||
key, sep, leaf = text.partition(": ")
|
||||
if not sep:
|
||||
return None
|
||||
key = key.strip()
|
||||
if not _KEY_RE.match(key):
|
||||
return None
|
||||
return key, leaf.strip()
|
||||
|
||||
|
||||
def _consume_block_mapping(fm_lines, start, first_entry, parent_key):
|
||||
"""Consume the sibling entries of a block mapping opened by a ``- `` item.
|
||||
|
||||
Returns ``(mapping, next_index)``. A sibling is an indented line that does
|
||||
not open a new list item; the run ends at a blank line, a comment, a new
|
||||
``- `` item, or a line at column zero. Depth is capped at one by giving the
|
||||
leaves the *unchanged* scalar predicates: a nested collection opens with
|
||||
``{`` or ``[`` and is refused by ``_reject_dangerous_value``, and a further
|
||||
block level is refused by ``_reject_mapping_construct``.
|
||||
"""
|
||||
allowed = _allowed_mapping_keys(parent_key)
|
||||
mapping = {}
|
||||
_admit_mapping_entry(mapping, first_entry[0], first_entry[1], allowed, parent_key)
|
||||
|
||||
i = start
|
||||
n = len(fm_lines)
|
||||
while i < n:
|
||||
raw = fm_lines[i]
|
||||
stripped = raw.strip()
|
||||
if stripped == "" or stripped.startswith("#"):
|
||||
break
|
||||
if raw[:1] not in (" ", "\t") or stripped.startswith("- "):
|
||||
break
|
||||
entry = _block_mapping_entry(stripped)
|
||||
if entry is None:
|
||||
_reject_dangerous_value(stripped)
|
||||
_reject_mapping_construct(stripped)
|
||||
raise OKFFrontmatterError(
|
||||
"a block-mapping entry must be 'key: value': %r" % (raw,)
|
||||
)
|
||||
_admit_mapping_entry(mapping, entry[0], entry[1], allowed, parent_key)
|
||||
i += 1
|
||||
return mapping, i
|
||||
|
||||
|
||||
def _admit_mapping_entry(mapping, key, leaf, allowed, parent_key):
|
||||
"""Admit one mapping entry, or raise. The single gate both carriers pass."""
|
||||
if not _KEY_RE.match(key):
|
||||
raise OKFFrontmatterError("invalid mapping key: %r" % (key,))
|
||||
if key not in allowed:
|
||||
raise OKFFrontmatterError(
|
||||
"mapping key %r is not on the OKF mapping allowlist under %r"
|
||||
% (key, parent_key)
|
||||
)
|
||||
if key in mapping:
|
||||
raise OKFFrontmatterError("duplicate mapping key %r" % (key,))
|
||||
_reject_dangerous_value(leaf)
|
||||
_reject_mapping_construct(leaf)
|
||||
mapping[key] = leaf
|
||||
|
||||
|
||||
def _reject_dangerous_value(value):
|
||||
if value and value[0] in _DANGEROUS_VALUE_STARTS:
|
||||
raise OKFFrontmatterError(
|
||||
|
|
@ -704,7 +830,7 @@ def _reject_mapping_construct(value):
|
|||
)
|
||||
|
||||
|
||||
def _parse_flow_mapping(value):
|
||||
def _parse_flow_mapping(value, parent_key=None):
|
||||
"""Parse ``{ key: value, ... }`` into a typed dict, or refuse it (G3).
|
||||
|
||||
Returns ``None`` when ``value`` does not open a flow mapping, so the caller
|
||||
|
|
@ -773,6 +899,7 @@ def _parse_flow_mapping(value):
|
|||
% (value,)
|
||||
)
|
||||
|
||||
allowed = _allowed_mapping_keys(parent_key)
|
||||
mapping = {}
|
||||
for entry in inner.split(","):
|
||||
entry = entry.strip()
|
||||
|
|
@ -781,20 +908,68 @@ def _parse_flow_mapping(value):
|
|||
raise OKFFrontmatterError(
|
||||
"a flow-mapping entry must be 'key: value': %r" % (entry,)
|
||||
)
|
||||
key = key.strip()
|
||||
leaf = leaf.strip()
|
||||
if not _KEY_RE.match(key):
|
||||
raise OKFFrontmatterError("invalid flow-mapping key: %r" % (key,))
|
||||
if key not in _MAPPING_KEY_ALLOWLIST:
|
||||
raise OKFFrontmatterError(
|
||||
"flow-mapping key %r is not on the OKF mapping allowlist: %r"
|
||||
% (key, value)
|
||||
)
|
||||
if key in mapping:
|
||||
raise OKFFrontmatterError(
|
||||
"duplicate flow-mapping key %r: %r" % (key, value)
|
||||
)
|
||||
_reject_dangerous_value(leaf)
|
||||
_reject_mapping_construct(leaf)
|
||||
mapping[key] = leaf
|
||||
_admit_mapping_entry(mapping, key.strip(), leaf.strip(), allowed, parent_key)
|
||||
return mapping
|
||||
|
||||
|
||||
def _parse_flow_sequence(value, parent_key=None):
|
||||
"""Parse ``[{ ... }, { ... }]`` into a list of typed dicts, or refuse it (G30).
|
||||
|
||||
Returns ``None`` when ``value`` does not open a flow sequence, so the caller
|
||||
falls through to the unchanged scalar rules - where ``[`` is still a
|
||||
disallowed indicator. This carrier is opened for the flow-mapping element
|
||||
and nothing else: it is the form the OKF producers emit for ``sources``
|
||||
(measured 02.09 against llm-ingestion-okf's golden bundle, where a
|
||||
one-element sequence raised on the ``[`` just as a two-element one did).
|
||||
|
||||
A flow sequence of plain *scalars* (``tags: [a, b, c]``) stays refused. It
|
||||
is a different shape with its own quoting and comma-splitting problem, whose
|
||||
failure mode would be accepting something YAML reads differently - and the
|
||||
block-sequence carrier already covers it for every consumer measured so far.
|
||||
|
||||
Elements are split on ``}`` rather than on commas, which is sound precisely
|
||||
because ``_parse_flow_mapping`` admits no nested collection: a ``}`` inside
|
||||
an element cannot occur, so the first ``}`` after ``{`` always closes it.
|
||||
Anything between elements that is not a separating comma is refused, which
|
||||
is what makes trailing junk and a mixed sequence fail rather than parse.
|
||||
"""
|
||||
if not value or value[0] != "[":
|
||||
return None
|
||||
if not value.endswith("]"):
|
||||
raise OKFFrontmatterError(
|
||||
"a flow sequence must be closed by ']' on the same line: %r" % (value,)
|
||||
)
|
||||
|
||||
inner = value[1:-1].strip()
|
||||
if not inner:
|
||||
raise OKFFrontmatterError("an empty flow sequence carries nothing: %r" % (value,))
|
||||
|
||||
items = []
|
||||
i = 0
|
||||
n = len(inner)
|
||||
while True:
|
||||
while i < n and inner[i] in " \t":
|
||||
i += 1
|
||||
if i >= n:
|
||||
break
|
||||
if inner[i] != "{":
|
||||
raise OKFFrontmatterError(
|
||||
"a flow sequence admits flow mappings only: %r" % (value,)
|
||||
)
|
||||
close = inner.find("}", i)
|
||||
if close == -1:
|
||||
raise OKFFrontmatterError(
|
||||
"an unclosed flow mapping inside a flow sequence: %r" % (value,)
|
||||
)
|
||||
items.append(_parse_flow_mapping(inner[i:close + 1], parent_key))
|
||||
i = close + 1
|
||||
while i < n and inner[i] in " \t":
|
||||
i += 1
|
||||
if i >= n:
|
||||
break
|
||||
if inner[i] != ",":
|
||||
raise OKFFrontmatterError(
|
||||
"trailing junk after a flow-sequence element: %r" % (value,)
|
||||
)
|
||||
i += 1
|
||||
return items
|
||||
|
|
|
|||
|
|
@ -548,7 +548,10 @@ _V02_REJECTED = [
|
|||
("generated (nested)", "generated:\n at: 2026-07-26T10:00:00Z\n"),
|
||||
("executor (nested)", "executor:\n resource: skills/run-on-bq.md\n"),
|
||||
("attester (nested)", "attester:\n resource: attesters/sql_equality.py\n"),
|
||||
("sources (block list of mappings)",
|
||||
# The carrier is admitted as of 1.3.0 (G30); this row now measures the KEY
|
||||
# SET - `uri`/`kind` are producer-invented, not SPEC §5.1 - and stays red for
|
||||
# that reason. SPEC's own §5.1 keys parse; see the G30 block at the foot.
|
||||
("sources (block list, off-allowlist keys)",
|
||||
"sources:\n - uri: https://e.com/a\n kind: doc\n"),
|
||||
("flow sequence", "tags: [a, b, c]\n"),
|
||||
("flow mapping", "executor: {resource: skills/run.md}\n"),
|
||||
|
|
@ -586,6 +589,8 @@ def test_v02_flat_frontmatter_still_parses(cid, fm):
|
|||
|
||||
_DEGRADED_TO_STRING = [
|
||||
# (id, frontmatter, what PyYAML 6.0.3 makes of it)
|
||||
# Still red after G30 opened the block-mapping carrier: `uri` is not a §5.1
|
||||
# key, so the item is refused by the allowlist instead of by the grammar.
|
||||
("one key per item", "sources:\n - uri: https://e.com/a\n", "[{'uri': ...}]"),
|
||||
("item, trailing colon", "sources:\n - uri:\n", "[{'uri': None}]"),
|
||||
("inline double colon", "attester: resource: attesters/sql_equality.py\n", "parse error"),
|
||||
|
|
@ -653,13 +658,16 @@ def test_pointer_in_a_degraded_mapping_no_longer_reaches_the_consumer_tree(cid,
|
|||
assert result.disposition is Disposition.FAIL_SECURE, "hole reopened — see LIMITATIONS.md"
|
||||
|
||||
|
||||
def test_exactly_one_route_to_a_mapping_is_expressible():
|
||||
# Was: ALL FOUR routes failed, each on its own rule, so the mapping *class* had
|
||||
# no expressible form (and v0.2's `generated` could not be written at all). G3
|
||||
# opens exactly ONE of them - the allowlisted flow form - and the other three
|
||||
# still fail, each on its own rule. That the openable route is the one whose
|
||||
# every key the allowlist inspects is the whole design: block, dotted and inline
|
||||
# give the allowlist nothing to inspect, so they stay shut.
|
||||
def test_the_expressible_mapping_routes_are_the_ones_the_allowlist_inspects():
|
||||
# Was `test_exactly_one_route_to_a_mapping_is_expressible` (1.2.0), and before
|
||||
# that ALL FOUR routes failed so the mapping *class* had no expressible form.
|
||||
# There are four expressible carriers as of 1.3.0 - flow mapping as a value,
|
||||
# flow mapping as a list item, flow sequence of flow mappings, block sequence
|
||||
# of block mappings - and the criterion that admits them is unchanged: each
|
||||
# hands the allowlist every key. The routes below stay shut for the same
|
||||
# reason, each on its own rule: a top-level block MAPPING (not a sequence),
|
||||
# a dotted key, and an inline second colon give the allowlist nothing to
|
||||
# inspect.
|
||||
assert parse_frontmatter("---\nid: x\ngenerated: { by: x, at: y }\n---\n\nbody\n")[0][
|
||||
"generated"] == {"by": "x", "at": "y"}
|
||||
|
||||
|
|
@ -695,19 +703,33 @@ def test_block_lists_admitted_by_item_shape(cid, fm, expected):
|
|||
assert parse_frontmatter(f"---\nid: x\n{fm}---\n\nbody\n")[0][key] == expected
|
||||
|
||||
|
||||
def test_two_keys_per_item_is_where_the_block_list_hard_rejects():
|
||||
def test_the_block_list_rejects_on_the_key_set_not_on_arity():
|
||||
# Was `test_two_keys_per_item_is_where_the_block_list_hard_rejects`: a
|
||||
# two-key item was refused because the block mapping had no expressible form
|
||||
# at all. G30 gives it one (SPEC.md §5.1's own carrier), so arity is no
|
||||
# longer the boundary - the key set is. The same two-key item parses when its
|
||||
# keys are §5.1's, and still raises when one of them is not.
|
||||
parsed = parse_frontmatter(
|
||||
"---\nid: x\nsources:\n - id: a\n resource: file://x\n---\n\nbody\n"
|
||||
)[0]
|
||||
assert parsed["sources"] == [{"id": "a", "resource": "file://x"}]
|
||||
with pytest.raises(OKFFrontmatterError):
|
||||
parse_frontmatter(
|
||||
"---\nid: x\nsources:\n - id: a\n resource: file://x\n---\n\nbody\n"
|
||||
"---\nid: x\nsources:\n - id: a\n uri: file://x\n---\n\nbody\n"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("fm", [
|
||||
# The flow row carries a key OFF the G3 allowlist: the shape is admitted, the
|
||||
# key is not, so this stays a T2 rejection and the door A/B half still holds.
|
||||
"generated: { by: x, tool: y }\n", "sources: [{ id: a }]\n", "tags: [a, b]\n",
|
||||
# The two `sources` rows that lived here until 1.3.0 now PARSE - that is the
|
||||
# G30 fix, not a weakening of this property. Their replacements are the same
|
||||
# carriers with a key off the allowlist, so the row still measures what it
|
||||
# says: the shape is admitted, the key set is not.
|
||||
"generated: { by: x, tool: y }\n", "sources: [{ id: a, uri: u }]\n", "tags: [a, b]\n",
|
||||
"generated:\n by: x\n", "generated.by: x\n",
|
||||
"sources:\n - id: a\n resource: file://x\n",
|
||||
"sources:\n - id: a\n uri: file://x\n",
|
||||
"executor: [{ resource: skills/run.md }]\n",
|
||||
])
|
||||
def test_t2_constrains_import_not_emission(fm):
|
||||
# T2 runs on door C only. The same frontmatter that FAIL_SECUREs through
|
||||
|
|
@ -903,3 +925,210 @@ def test_a_conformant_v02_trust_layer_now_reaches_the_gate():
|
|||
result = import_bundle({"tables/users.md": doc})
|
||||
assert result.disposition is Disposition.WARN
|
||||
assert result.concepts[0].error is None
|
||||
|
||||
|
||||
# --- G30: the `sources` provenance layer becomes reachable (2026-09-02) ------
|
||||
# Door 2. G3 gave the mapping *class* one expressible form but left `sources`
|
||||
# unreachable: SPEC.md §5.1 writes an entry as a MAPPING carrying a REQUIRED
|
||||
# `resource`, so neither of the two carriers the spec and the producers actually
|
||||
# use could parse. Measured 02.09 by two consumers independently -- a flow
|
||||
# sequence of flow mappings raised on the `[` indicator, a block sequence of
|
||||
# block mappings raised "nested mappings are not supported".
|
||||
#
|
||||
# Why `resource` is admissible now when 1.2.0 argued it was not: the old
|
||||
# argument was that the parser could not tell `sources[].resource` (§5.1, a
|
||||
# citation) from `executor.resource` (§10, a code pointer). That premise was
|
||||
# false -- the owning key is in scope at every call site, it was simply never
|
||||
# threaded through. `resource` is allowlisted for `sources` entries ONLY, so
|
||||
# the door-C routes above stay shut on the same input.
|
||||
|
||||
_SPEC_51_BLOCK = (
|
||||
"sources:\n"
|
||||
" - id: ga4-schema\n"
|
||||
" resource: https://developers.google.com/analytics/bigquery/export-schema\n"
|
||||
" title: GA4 BigQuery Export schema\n"
|
||||
" author: team:ga4-docs\n"
|
||||
" usage_count: 5000\n"
|
||||
" last_modified: 2026-05-30T00:00:00Z\n"
|
||||
)
|
||||
|
||||
|
||||
def test_spec_sources_block_sequence_of_block_mappings_parses():
|
||||
# SPEC.md §5.1's own example block, verbatim. It is the spec's canonical
|
||||
# carrier for a REQUIRED field, so §11.1 ("parseable YAML frontmatter") makes
|
||||
# a bundle written this way conformant -- refusing it refuses a conformant
|
||||
# bundle, which is the failure mode G3 was opened to end.
|
||||
fm, _ = parse_frontmatter(f"---\ntype: table\n{_SPEC_51_BLOCK}---\n\nbody\n")
|
||||
assert fm["sources"] == [{
|
||||
"id": "ga4-schema",
|
||||
"resource": "https://developers.google.com/analytics/bigquery/export-schema",
|
||||
"title": "GA4 BigQuery Export schema",
|
||||
"author": "team:ga4-docs",
|
||||
"usage_count": "5000",
|
||||
"last_modified": "2026-05-30T00:00:00Z",
|
||||
}]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cid,fm,expected", [
|
||||
("one entry",
|
||||
"sources: [{ id: golden-v0-2-sales, resource: fixture }]\n",
|
||||
[{"id": "golden-v0-2-sales", "resource": "fixture"}]),
|
||||
("two entries",
|
||||
"sources: [{ id: a, resource: https://e.com/a }, { id: b, resource: https://e.com/b }]\n",
|
||||
[{"id": "a", "resource": "https://e.com/a"},
|
||||
{"id": "b", "resource": "https://e.com/b"}]),
|
||||
])
|
||||
def test_sources_flow_sequence_of_flow_mappings_parses(cid, fm, expected):
|
||||
# The form the producer emits today (llm-ingestion-okf's golden
|
||||
# expected-bundle/ingest-sales.md, measured 02.09). One entry raised too, so
|
||||
# this is not an arity bug: the `[` indicator refused the carrier outright.
|
||||
assert parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")[0]["sources"] == expected
|
||||
|
||||
|
||||
def test_the_two_sources_carriers_parse_to_the_same_value():
|
||||
block = parse_frontmatter(
|
||||
"---\ntype: t\nsources:\n - id: a\n resource: https://e.com/a\n---\n\nb\n")[0]
|
||||
flow = parse_frontmatter(
|
||||
"---\ntype: t\nsources: [{ id: a, resource: https://e.com/a }]\n---\n\nb\n")[0]
|
||||
assert block["sources"] == flow["sources"] == [{"id": "a", "resource": "https://e.com/a"}]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cid,fm", [
|
||||
("unknown key, flow", "sources: [{ id: a, uri: https://e.com/a }]\n"),
|
||||
("unknown key, block", "sources:\n - id: a\n uri: https://e.com/a\n"),
|
||||
("unknown key, flow value", "sources: { id: a, kind: doc }\n"),
|
||||
])
|
||||
def test_an_unknown_key_in_a_sources_entry_is_still_rejected(cid, fm):
|
||||
# The negative control. The carrier is admitted; the key set is not. A
|
||||
# producer-invented key gets no free ride on the new shape.
|
||||
with pytest.raises(OKFFrontmatterError) as exc:
|
||||
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
|
||||
assert "allowlist" in str(exc.value)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cid,fm", [
|
||||
("executor, block sequence", "executor:\n - resource: skills/run-on-bq.md\n"),
|
||||
("executor, flow sequence", "executor: [{ resource: skills/run-on-bq.md }]\n"),
|
||||
("attester, flow sequence", "attester: [{ resource: attesters/sql_equality.py }]\n"),
|
||||
("attester, block sequence", "attester:\n - resource: attesters/sql_equality.py\n"),
|
||||
])
|
||||
def test_resource_is_allowlisted_for_sources_entries_only(cid, fm):
|
||||
# The whole reason `resource` can be admitted at all: the owning key decides.
|
||||
# §10's `executor.resource` / `attester.resource` name run instructions and
|
||||
# code -- door C -- and stay refused through EVERY carrier, including the two
|
||||
# this change opens for `sources`.
|
||||
doc = f"---\nid: x\ntype: Attested Computation\n{fm}---\n\nbody\n"
|
||||
with pytest.raises(OKFFrontmatterError) as exc:
|
||||
parse_frontmatter(doc)
|
||||
assert "allowlist" in str(exc.value)
|
||||
assert import_bundle({"computations/x.md": doc}).disposition is Disposition.FAIL_SECURE
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cid,fm", [
|
||||
# SPEC §5.1: "A single entry MAY carry its own `usage_window`". That is a
|
||||
# mapping inside a mapping -- depth 2 -- and stays refused: no nesting deeper
|
||||
# than one is a security property this change does not spend. Registered as a
|
||||
# conformance gap in docs/LIMITATIONS.md, not as an oversight.
|
||||
("per-entry usage_window, flow",
|
||||
"sources: [{ id: a, usage_window: { from: x, to: y } }]\n"),
|
||||
("per-entry usage_window, block",
|
||||
"sources:\n - id: a\n usage_window: { from: x, to: y }\n"),
|
||||
# A sequence inside an entry is the same depth violation.
|
||||
("nested sequence", "sources: [{ id: a, title: [x, y] }]\n"),
|
||||
])
|
||||
def test_a_sources_entry_admits_scalar_leaves_only(cid, fm):
|
||||
with pytest.raises(OKFFrontmatterError):
|
||||
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cid,fm", [
|
||||
("scalar then mapping", "sources:\n - https://e.com/a\n - id: b\n title: B\n"),
|
||||
("mapping then scalar", "sources:\n - id: a\n title: A\n - https://e.com/b\n"),
|
||||
])
|
||||
def test_a_block_list_may_not_mix_scalars_and_mappings(cid, fm):
|
||||
# A consumer that reads `entry.get("id")` over the list crashes on the str.
|
||||
# One list, one item type -- refuse rather than hand back a mixed tree.
|
||||
with pytest.raises(OKFFrontmatterError) as exc:
|
||||
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
|
||||
assert "mix" in str(exc.value)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cid,fm", [
|
||||
("flow sequence of scalars", "tags: [a, b, c]\n"),
|
||||
("empty flow sequence", "sources: []\n"),
|
||||
("flow sequence, unclosed", "sources: [{ id: a }\n"),
|
||||
("flow sequence, trailing junk", "sources: [{ id: a }] x\n"),
|
||||
("flow sequence, mixed", "sources: [{ id: a }, plain]\n"),
|
||||
("flow sequence, nested sequence", "sources: [[ id ]]\n"),
|
||||
])
|
||||
def test_the_flow_sequence_admits_flow_mappings_only(cid, fm):
|
||||
# The sequence carrier is opened for the mapping form and nothing else. A
|
||||
# flow sequence of plain scalars stays refused -- it is a separate shape with
|
||||
# its own quoting and splitting problems, and no measured consumer needs it.
|
||||
with pytest.raises(OKFFrontmatterError):
|
||||
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
|
||||
|
||||
|
||||
def test_injection_in_a_sources_entry_leaf_is_caught_by_the_scan():
|
||||
# T1 over the new shape: every leaf of every entry reaches scan_output.
|
||||
doc = f"---\ntype: table\nsources: [{{ id: a, title: {_INJECTION} }}]\n---\nclean\n"
|
||||
assert scan_concept(doc).found is True
|
||||
doc_block = f"---\ntype: table\nsources:\n - id: a\n title: {_INJECTION}\n---\nclean\n"
|
||||
assert scan_concept(doc_block).found is True
|
||||
|
||||
|
||||
def test_the_producer_golden_now_passes_the_gate():
|
||||
# llm-ingestion-okf's expected-bundle/ingest-sales.md, the K5 blocker.
|
||||
doc = (
|
||||
"---\n"
|
||||
"type: dataset\n"
|
||||
"title: Regional Sales\n"
|
||||
"source_system: golden-v0-2-sales\n"
|
||||
"ingested_at: 2026-07-16T12:00:00Z\n"
|
||||
"generated: { by: process:okf-ingest, at: 2026-07-16T12:00:00Z }\n"
|
||||
"sources: [{ id: golden-v0-2-sales, resource: fixture }]\n"
|
||||
"---\n\n| region | units |\n| --- | --- |\n| nord | 412 |\n"
|
||||
)
|
||||
result = import_bundle({"datasets/sales.md": doc})
|
||||
assert result.disposition is Disposition.WARN
|
||||
assert result.concepts[0].error is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cid,fm", [
|
||||
# A `- ` item whose text begins with a YAML indicator AND carries a `": "`
|
||||
# must fall through to the scalar predicates, not into the block-mapping
|
||||
# route with a half-validated key. Each of these fails `_KEY_RE` on the key
|
||||
# side, so `_reject_dangerous_value` gets the item intact.
|
||||
("anchor", "sources:\n - &anchor id: a\n"),
|
||||
("alias", "sources:\n - *anchor id: a\n"),
|
||||
("tag", "sources:\n - !!str id: a\n"),
|
||||
("directive", "sources:\n - %YAML id: a\n"),
|
||||
("reserved", "sources:\n - `x id: a\n"),
|
||||
("merge key", "sources:\n - id: a\n <<: *base\n"),
|
||||
])
|
||||
def test_an_indicator_in_a_block_item_is_refused_before_the_mapping_route(cid, fm):
|
||||
with pytest.raises(OKFFrontmatterError):
|
||||
parse_frontmatter(f"---\ntype: t\n{fm}---\n\nbody\n")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cid,fm", [
|
||||
# The block mapping ends at a blank line and at a line in column zero. Both
|
||||
# hand control back with an index that must not skip or re-read a line: a
|
||||
# dangling `resource:` line left over from a mapping that closed early must
|
||||
# RAISE, never be silently dropped -- a pointer that vanishes rather than
|
||||
# failing is exactly this repo's failure class.
|
||||
("blank line inside the mapping", "sources:\n - id: a\n\n resource: b\n"),
|
||||
("top-level key interleaved", "sources:\n - id: a\ntags: x\n resource: b\n"),
|
||||
])
|
||||
def test_a_line_orphaned_by_the_mapping_boundary_raises_rather_than_vanishing(cid, fm):
|
||||
with pytest.raises(OKFFrontmatterError):
|
||||
parse_frontmatter(f"---\ntype: t\n{fm}---\n\nbody\n")
|
||||
|
||||
|
||||
def test_the_block_list_hands_back_an_index_that_resumes_at_the_next_key():
|
||||
# The return-index contract: a top-level key following a multi-entry block
|
||||
# list is neither swallowed by the list nor re-read as a list item.
|
||||
fm, _ = parse_frontmatter(
|
||||
"---\ntype: t\nsources:\n - id: a\n resource: b\n - id: c\ntitle: T\n---\n\nbody\n")
|
||||
assert fm == {"type": "t", "sources": [{"id": "a", "resource": "b"}, {"id": "c"}],
|
||||
"title": "T"}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue