feat(okf): the mapping class gets one expressible form, typed and allowlisted
OKF v0.2 writes its whole trust and provenance layer as mappings, and T2 gave
the mapping class no expressible form. A consumer measured 0 of 53 upstream
concepts through the gate on 0.3.4, 1.0.0 and 1.1.0. That was a contract
collision, not a calibration setting: SPEC.md @ 62432a09 uses flow mappings in
its own 5.1/5.2 examples, and 11 carries a hard MUST for consumers ("MUST treat
a bare `verified` mapping as a one-element list") that presupposes they parse.
Admitted: a flow mapping, as a value or as a block-list item, whose every key is
on a nine-name allowlist and whose every leaf is a plain scalar run through the
unchanged dangerous-value and mapping-construct predicates. The form is safe
because the allowlist inspects every key -- the blanket refusal was the
enforcement, not the point.
Refused, each on its own rule and ground-truthed against PyYAML 6.0.3: a key off
the allowlist, a nested collection, a quoted leaf, a duplicate key, an empty or
unclosed mapping, trailing junk, and {a:b} (which PyYAML reads as the KEY a:b).
A refused mapping still raises rather than degrading into a string, so the 1.1.0
type-confusion defect is not reopened, and the block, dotted and inline-colon
routes still raise.
`resource` is deliberately off the allowlist though SPEC.md 5.1 names it inside
a `sources` entry: it is a pointer rather than a label and the only key T3
exists for, so admitting it would let `executor: { resource: skills/run.md }`
carry an executable-code pointer through in typed clothes -- the door-C route
closed in 1.1.0. It costs nothing today, because the conformant carrier for
sources[].resource is a block sequence of block mappings, which stays refused.
Mapping leaves are scanned like every other frontmatter value (T1). Coverage
matrix 130/130 (new row: the off-allowlist key). No exported surface, detector
behaviour or calibration changed.
This commit is contained in:
parent
7cb4553301
commit
58704834b6
7 changed files with 432 additions and 43 deletions
39
CHANGELOG.md
39
CHANGELOG.md
|
|
@ -7,6 +7,45 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
|
||||
## [Unreleased]
|
||||
|
||||
### Added — OKF frontmatter can express one mapping form: typed and allowlisted
|
||||
|
||||
`okf.parse_frontmatter` gave the mapping *class* no expressible form at all. OKF
|
||||
v0.2 writes its whole trust and provenance layer as mappings — SPEC.md @
|
||||
`62432a09` uses flow form in its own §5.1/§5.2 examples, and §11 carries a hard
|
||||
MUST for consumers ("MUST treat a bare `verified` mapping as a one-element
|
||||
list") that presupposes they parse. A consumer measured **0 of 53** upstream
|
||||
concepts through the gate on 0.3.4, 1.0.0 and 1.1.0. That was a contract
|
||||
collision, not a calibration setting: no threshold would have moved it.
|
||||
|
||||
Admitted now: a flow mapping (`generated: { by: x, at: y }`), as a value or as a
|
||||
block-list item, whose every key is on a nine-name allowlist (`by`, `at`, `from`,
|
||||
`to`, `id`, `title`, `author`, `usage_count`, `last_modified`) and whose every
|
||||
leaf is a plain scalar run through the *unchanged* dangerous-value and
|
||||
mapping-construct predicates.
|
||||
|
||||
The form is additive and refusal stays the default. A key off the allowlist, a
|
||||
nested collection, a quoted leaf, a duplicate key, an empty or unclosed mapping,
|
||||
and `{a:b}` (which PyYAML 6.0.3 reads as the *key* `a:b`) all raise, and a
|
||||
refused mapping still raises rather than degrading into a string — the 1.1.0
|
||||
defect is not reopened. Nested-block (`k:\n sub: v`), dotted (`k.sub: v`) and
|
||||
inline-second-colon (`k: sub: v`) routes to a mapping still raise, each on its
|
||||
own rule.
|
||||
|
||||
**`resource` is deliberately off the allowlist**, though SPEC.md §5.1 names it
|
||||
inside a `sources` entry. It is a pointer rather than a label and the only key
|
||||
T3 exists for: admitting it would let `executor: { resource: skills/run.md }`
|
||||
carry an executable-code pointer through in typed clothes, which is the door-C
|
||||
route closed in 1.1.0. It costs nothing today — the conformant carrier for
|
||||
`sources[].resource` is a block sequence of block mappings, which this form does
|
||||
not admit either way.
|
||||
|
||||
Mapping leaves are scanned like every other frontmatter value (T1), so an
|
||||
injection parked in `generated: { by: ... }` reaches `scan_output`. Coverage
|
||||
matrix: 130 classes, up from 129 (the new row is the off-allowlist key).
|
||||
|
||||
No exported surface changed; no detector behaviour and no calibration changed.
|
||||
|
||||
|
||||
### Changed — the ReDoS sweep now measures on the same clock as the bounds it justifies
|
||||
|
||||
`docs/redos-sweep.py` timed on `time.monotonic()` while every ReDoS bound in the
|
||||
|
|
|
|||
|
|
@ -15,7 +15,14 @@ Repoet er på **v1.1.0** — den eksporterte Python-surfacen er frosset under se
|
|||
(deteksjonsatferd er det IKKE; kalibrering flytter seg i 1.x). Stdlib-kjernen er
|
||||
bygget og testet (15 moduler +
|
||||
topp-nivå wiring, showcase + korpus), inkl. OKF-adapter og aktivt-innhold-
|
||||
detektor (EchoLeak-klassen) i output-gaten. Mode-b `import_bundle` skanner
|
||||
detektor (EchoLeak-klassen) i output-gaten. OKF-frontmatterens mapping-klasse
|
||||
har **én** uttrykkbar form (G3, 21.08): en flow-mapping (`generated: { by: x, at: y }`) — som verdi eller som blokkliste-
|
||||
element — der HVER nøkkel står på en ni-navns allowlist og hvert blad er en ren
|
||||
skalar. Formen er trygg fordi allowlisten inspiserer hver nøkkel; det blanke
|
||||
avslaget var håndhevelsen, ikke poenget. `resource` er bevisst UTE av
|
||||
allowlisten (peker, ikke etikett — den ene nøkkelen T3 finnes for). Blokk-,
|
||||
dotted- og inline-kolon-rutene raiser fortsatt, og en avvist mapping raiser —
|
||||
den degraderer aldri til en streng (1.1.0-defekten). Mode-b `import_bundle` skanner
|
||||
reserverte strukturfiler (`index.md`/`log.md`) i mottatte bundles i stedet for å
|
||||
path-avvise dem; upload-front-end beholder shadow-reject (`allow_reserved=False`).
|
||||
Output-gatens decode-and-rescan mater dekodet base64-klartekst gjennom BÅDE lexicon
|
||||
|
|
|
|||
12
README.md
12
README.md
|
|
@ -165,7 +165,15 @@ Per-concept gates: **path / reserved-name** (rejects `..` traversal and reserved
|
|||
reject-by-default loader that refuses anchors, aliases, and explicit tags *by
|
||||
construction*, so a billion-laughs alias expansion or a `!!python/object` coercion
|
||||
cannot occur (it is deliberately **not** a general YAML engine, whose own features
|
||||
are the attack surface); **`resource` https-allowlist** (hard-rejects
|
||||
are the attack surface). The one mapping form it accepts is OKF v0.2's flow
|
||||
mapping — `generated: { by: x, at: y }`, `verified: { … }` bare or listed,
|
||||
`usage_window: { from: …, to: … }` — admitted key-by-key against a nine-name
|
||||
allowlist (`by`, `at`, `from`, `to`, `id`, `title`, `author`, `usage_count`,
|
||||
`last_modified`) with plain-scalar leaves only. A key off that list, a nested
|
||||
collection or a duplicate key is refused, and `resource` is deliberately not on
|
||||
it; the block, dotted and inline-colon routes to a mapping still raise. See
|
||||
[LIMITATIONS](docs/LIMITATIONS.md) for what that admits and what it still walls
|
||||
off (a `sources` block list of mappings is still refused); **`resource` https-allowlist** (hard-rejects
|
||||
`data:`/`javascript:`/`file:` before commit — a reject-gate, not defang);
|
||||
**whole-concept scan** (frontmatter *values* + body through `scan_output`);
|
||||
**cross-link graph** (surfaces dangling targets, the dormant-injection signal, and
|
||||
|
|
@ -179,7 +187,7 @@ driven by a **live payload** in the coverage matrix — run it to watch all 134
|
|||
in your own environment:
|
||||
|
||||
```bash
|
||||
python -m llm_ingestion_guard.coverage # 129/129 classes; exit 0 = all as documented
|
||||
python -m llm_ingestion_guard.coverage # 130/130 classes; exit 0 = all as documented
|
||||
```
|
||||
|
||||
| Anchor | Attack classes it stops (representative) |
|
||||
|
|
|
|||
|
|
@ -40,16 +40,27 @@ items; this is the full list, each with the mechanism.
|
|||
(an injection in a directory listing is caught) rather than path-rejecting the
|
||||
conformant bundle. A front-end materialising individual uploads keeps the opposite
|
||||
rule (`allow_reserved=False`): a reserved basename is a listing-shadow and refused.
|
||||
- **OKF frontmatter is a restricted grammar: the mapping class has no expressible
|
||||
form.** Gate T2 accepts a line-oriented subset deliberately — full YAML is a larger
|
||||
parse-attack surface than a write-time gate needs. Nested mappings and flow
|
||||
collections (`[a, b]`, `{k: v}`) are *rejected outright*, which fails secure.
|
||||
**All four routes to a mapping fail, each on a different rule** — flow (`{k: v}`)
|
||||
on the disallowed value-start indicator, block (`k:\n sub: v`) on the
|
||||
nested-mapping check, dotted keys (`k.sub: v`) on the key pattern, and the inline
|
||||
second colon (`k: sub: v`) on the mapping-construct check — so the mapping *class*
|
||||
has no expressible form, rather than one form being preferable to another. What
|
||||
survives is scalars and flat lists of strings. **Two of those routes used to
|
||||
- **OKF frontmatter is a restricted grammar: the mapping class has exactly one
|
||||
expressible form.** Gate T2 accepts a line-oriented subset deliberately — full YAML
|
||||
is a larger parse-attack surface than a write-time gate needs. Flow sequences
|
||||
(`[a, b]`) and nested mappings are *rejected outright*, which fails secure.
|
||||
**Three of the four routes to a mapping fail, each on a different rule** — block
|
||||
(`k:\n sub: v`) on the nested-mapping check, dotted keys (`k.sub: v`) on the key
|
||||
pattern, and the inline second colon (`k: sub: v`) on the mapping-construct check.
|
||||
**The fourth, the flow form, is admitted only when every key is on an allowlist**
|
||||
(`by`, `at`, `from`, `to`, `id`, `title`, `author`, `usage_count`,
|
||||
`last_modified` — the keys SPEC.md @ `62432a09` §5.1/§5.2 names inside a mapping)
|
||||
and every leaf is a plain scalar, itself run through the same value predicates as a
|
||||
top-level scalar. Nested collections, quoted leaves, duplicate keys, an empty or
|
||||
unclosed mapping, and `{a:b}` (which PyYAML 6.0.3 reads as the *key* `a:b`, not as
|
||||
a scalar) all raise. The form is expressible, never trusted: the allowlist
|
||||
inspects every key, which is the property that carried the security when the
|
||||
blanket refusal was doing the enforcing. **`resource` is deliberately off the
|
||||
allowlist** although §5.1 names it inside a `sources` entry — it is a pointer
|
||||
rather than a label and the only key T3 exists for, so admitting it would let
|
||||
`executor: { resource: skills/run.md }` carry an executable-code pointer through a
|
||||
key the https allowlist never inspects. What else survives is scalars and flat
|
||||
lists of strings. **Two routes used to
|
||||
degrade into a string instead of failing, and that defect is closed in `1.1.0`**:
|
||||
a block-sequence item carrying exactly one key (`sources:\n - uri: https://e.com/a`
|
||||
yielded the *string* `'uri: https://e.com/a'`) and the inline second colon
|
||||
|
|
@ -74,13 +85,19 @@ items; this is the full list, each with the mechanism.
|
|||
door A/B persist path, so frontmatter that fails secure on import passes
|
||||
`screen_output` unremarked. The grammar therefore bounds what a consumer can *receive*,
|
||||
never what a producer can *emit*. Verified identical on 0.2.0 and 0.3.1.
|
||||
- **Consequence: an OKF v0.2 concept cannot traverse the external-import path.** Both
|
||||
of v0.2's backward-breaking migration targets are nested — `timestamp` → `generated.at`,
|
||||
and body `# Citations` → a `sources` block list of mappings — so a conformant v0.2
|
||||
concept fails secure at the frontmatter gate. This is the correct direction but it is
|
||||
a compatibility wall, not a policy: v0.2 support requires a deliberate parse-safety
|
||||
decision about widening the grammar, and the dangling-or-substituted `executor`/
|
||||
`attester` pointer question only becomes live once that decision is made.
|
||||
- **An OKF v0.2 concept traverses the external-import path only if its `sources` are
|
||||
flat.** The wall used to be total: both of v0.2's backward-breaking migration targets
|
||||
are mappings — `timestamp` → `generated.at`, and body `# Citations` → a `sources`
|
||||
block list of mappings — and a consumer measured **0 of 53** upstream concepts
|
||||
through the gate. The trust and provenance layer now passes in its spec form
|
||||
(`generated`, `verified` bare or listed, `usage_window`), so `generated.at` is no
|
||||
longer a wall. **`sources` still is**: SPEC.md writes each entry as a block mapping
|
||||
under a block sequence (`- id: …\n resource: …`), and that carrier stays refused —
|
||||
it is the shape whose one-key degradation smuggled a pointer before `1.1.0`, and
|
||||
reopening it is a separate parse-safety decision, not a corollary of the flow form.
|
||||
A concept whose `sources` are flat strings, or absent, imports. The
|
||||
dangling-or-substituted `executor`/`attester` pointer question stays out of reach
|
||||
for the same reason: both are mappings whose payload key is `resource`.
|
||||
- **A persist gate cannot cover execution risk.** OKF v0.2 introduces concepts whose
|
||||
purpose is to *name code to be run* (`runtime`, `executor.resource`,
|
||||
`attester.resource`). This library answers "is this safe to **store**"; executable
|
||||
|
|
|
|||
|
|
@ -542,8 +542,11 @@ def _build_cases() -> list[Case]:
|
|||
lambda: okf.parse_frontmatter("---\nkey:\n nested: x\n---\nbody\n"), owasp="LLM10"),
|
||||
_raise_case("okf", "T2 frontmatter block scalar", "OKFFrontmatterError",
|
||||
lambda: okf.parse_frontmatter("---\ndesc: |\n block\n---\nbody\n"), owasp="LLM10"),
|
||||
_raise_case("okf", "T2 frontmatter flow collection", "OKFFrontmatterError",
|
||||
_raise_case("okf", "T2 frontmatter flow sequence", "OKFFrontmatterError",
|
||||
lambda: okf.parse_frontmatter("---\ntags: [a, b]\n---\nbody\n"), owasp="LLM10"),
|
||||
_raise_case("okf", "T2 mapping key off the allowlist", "OKFFrontmatterError",
|
||||
lambda: okf.parse_frontmatter(
|
||||
"---\ngenerated: { by: a, tool: shell }\n---\nbody\n"), owasp="LLM10"),
|
||||
_raise_case("okf", "T3 resource non-https (http)", "OKFResourceError",
|
||||
lambda: okf.validate_resource_url("http://insecure.test/x"), owasp="LLM05"),
|
||||
_raise_case("okf", "T3 resource data: scheme", "OKFResourceError",
|
||||
|
|
|
|||
|
|
@ -7,7 +7,8 @@ and feeds scannable text regions into the existing ``sanitize`` / ``scan_output`
|
|||
|
||||
T2 — frontmatter parse-safety gate. ``parse_frontmatter`` is a *strict,
|
||||
reject-by-default* loader for the minimal OKF frontmatter subset: flat
|
||||
``key: value`` scalars plus block ``- item`` lists. Every construct the
|
||||
``key: value`` scalars, block ``- item`` lists, and one typed, allowlisted
|
||||
mapping form (``{ by: x, at: y }`` — see :func:`_parse_flow_mapping`). Every construct the
|
||||
"block anchor/alias DoS + dangerous type coercion" requirement names is refused
|
||||
*by construction* — you cannot suffer a billion-laughs alias expansion or a
|
||||
``!!python/object`` coercion if anchors, aliases and explicit tags are rejected
|
||||
|
|
@ -16,7 +17,9 @@ philosophy, the frontmatter analogue of the ``resource`` reject-gate (T3).
|
|||
|
||||
Deliberately NOT a general YAML parser. A security tool whose thesis is
|
||||
minimal-dependency should not pull in a full YAML engine whose own features
|
||||
(anchors, tags, merges) are the attack surface being defended against. Quoted
|
||||
(anchors, tags, merges) are the attack surface being defended against. The one
|
||||
mapping form it does admit is admitted key-by-key against an allowlist, not
|
||||
parsed generally: the mapping class is expressible, never trusted. Quoted
|
||||
scalars are kept verbatim (quotes included) rather than unquoted — the value is
|
||||
still scanned as text downstream, so an injection inside a quoted value is not
|
||||
lost; richer scalar forms are a future refinement, not a silent parse.
|
||||
|
|
@ -62,13 +65,32 @@ _KEY_RE = re.compile(r"^[A-Za-z0-9_][A-Za-z0-9_-]*$")
|
|||
# A plain OKF scalar cannot *begin* with a YAML structural indicator. Any value
|
||||
# starting with one signals an anchor (&), alias (*), explicit tag (!), block
|
||||
# scalar (|, >), flow collection ([ ] { }), directive (%) or reserved char
|
||||
# (@ `) — all outside the supported subset and all rejected.
|
||||
# (@ `) — all outside the supported subset and all rejected. `{` is tried as the
|
||||
# allowlisted mapping form FIRST (G3); it reaches this predicate only as a leaf
|
||||
# inside one, where a nested collection is refused before it can be read.
|
||||
_DANGEROUS_VALUE_STARTS = frozenset("&*!|>[]{}%@`")
|
||||
# A quoted scalar is a scalar in YAML however many colons it carries, so the
|
||||
# mapping check steps aside for one. The quotes are retained rather than
|
||||
# stripped — a pre-existing divergence, pinned in tests/test_okf.py.
|
||||
_QUOTE_STARTS = frozenset("\"'")
|
||||
|
||||
# G3 — the one mapping form T2 can express (operator decision, 2026-08-21).
|
||||
# Every key inside a mapping must be on this allowlist: the form is safe because
|
||||
# the allowlist inspects each key, not because mappings became trusted. The keys
|
||||
# are the ones OKF v0.2 names inside a mapping - `by`/`at` (SPEC.md @ 62432a09
|
||||
# §5.2 `generated`/`verified`), `from`/`to` (§5.1 `usage_window`) and the
|
||||
# `sources`-entry fields (§5.1). `resource` is the one §5.1 key deliberately
|
||||
# LEFT OFF: it is a pointer rather than a label, it is the only key T3 exists
|
||||
# for, and admitting it inside a mapping would re-open the door-C route closed
|
||||
# in 1.1.0 (`executor: {resource: skills/run.md}` puts an executable-code
|
||||
# pointer in a key the https allowlist never inspects). It costs nothing today,
|
||||
# because the conformant carrier for `sources[].resource` is the block-sequence
|
||||
# of block-mappings, which this form does not admit either way.
|
||||
_MAPPING_KEY_ALLOWLIST = frozenset({
|
||||
"by", "at", "from", "to", "id", "title", "author", "usage_count",
|
||||
"last_modified",
|
||||
})
|
||||
|
||||
|
||||
class OKFError(Exception):
|
||||
"""Base class for OKF adapter rejections."""
|
||||
|
|
@ -106,7 +128,10 @@ def parse_frontmatter(document):
|
|||
|
||||
Raises ``OKFFrontmatterError`` on an unterminated fence or any construct
|
||||
outside the minimal flat subset (anchors, aliases, explicit tags, merge
|
||||
keys, block scalars, flow collections, nested mappings).
|
||||
keys, block scalars, flow sequences, nested mappings). The single exception
|
||||
is the typed, allowlisted flow mapping (:func:`_parse_flow_mapping`), which
|
||||
parses into a ``dict`` of allowlisted keys with plain-scalar leaves — every
|
||||
other route to a mapping still raises.
|
||||
"""
|
||||
lines = document.split("\n")
|
||||
if not lines or lines[0].strip() != _FENCE:
|
||||
|
|
@ -149,13 +174,29 @@ def _scannable_regions(frontmatter, body):
|
|||
"""The text regions of a concept that carry attacker-controlled content."""
|
||||
regions = [body]
|
||||
for value in frontmatter.values():
|
||||
if isinstance(value, list):
|
||||
regions.extend(value)
|
||||
elif value:
|
||||
regions.append(value)
|
||||
regions.extend(_value_regions(value))
|
||||
return regions
|
||||
|
||||
|
||||
def _value_regions(value):
|
||||
"""Every scannable leaf of one frontmatter value.
|
||||
|
||||
A mapping value (G3) is a new *shape* on this surface, not a new exemption:
|
||||
its leaves are scanned exactly like a scalar or a list item, so an injection
|
||||
parked in ``generated: { by: ... }`` reaches ``scan_output`` like any other
|
||||
frontmatter text. Mapping *keys* are not scanned because they cannot carry
|
||||
attacker text - the allowlist admits nine fixed names and nothing else.
|
||||
"""
|
||||
if isinstance(value, dict):
|
||||
return [leaf for leaf in value.values() if leaf]
|
||||
if isinstance(value, list):
|
||||
regions = []
|
||||
for item in value:
|
||||
regions.extend(_value_regions(item))
|
||||
return regions
|
||||
return [value] if value else []
|
||||
|
||||
|
||||
def validate_concept_path(path, *, allow_reserved=False):
|
||||
"""Validate a bundle-relative concept path and return its concept-ID.
|
||||
|
||||
|
|
@ -579,6 +620,12 @@ def _parse_flat(fm_lines):
|
|||
result[key] = items if items is not None else ""
|
||||
continue
|
||||
|
||||
mapping = _parse_flow_mapping(value)
|
||||
if mapping is not None:
|
||||
result[key] = mapping
|
||||
i += 1
|
||||
continue
|
||||
|
||||
_reject_dangerous_value(value)
|
||||
_reject_mapping_construct(value)
|
||||
result[key] = value
|
||||
|
|
@ -605,6 +652,11 @@ def _consume_block_list(fm_lines, start):
|
|||
continue
|
||||
if raw[:1] in (" ", "\t") and stripped.startswith("- "):
|
||||
item = stripped[2:].strip()
|
||||
mapping = _parse_flow_mapping(item)
|
||||
if mapping is not None:
|
||||
items.append(mapping)
|
||||
i += 1
|
||||
continue
|
||||
_reject_dangerous_value(item)
|
||||
_reject_mapping_construct(item)
|
||||
items.append(item)
|
||||
|
|
@ -627,8 +679,10 @@ def _reject_dangerous_value(value):
|
|||
def _reject_mapping_construct(value):
|
||||
"""Reject a scalar that YAML reads as a mapping rather than as a string.
|
||||
|
||||
T2 gives the mapping *class* no expressible form — flow, nested-block and
|
||||
dotted-key routes all raise. Two routes used to escape that by degrading
|
||||
T2 gives the mapping *class* exactly one expressible form, the typed
|
||||
allowlisted flow mapping (G3); the nested-block and dotted-key routes still
|
||||
raise, and this predicate is what keeps them raising — both at the top level
|
||||
and on a leaf *inside* an admitted mapping. Two routes used to escape by degrading
|
||||
into a string instead: a block-sequence item carrying exactly one key
|
||||
(``- uri: x``), and an inline second colon (``attester: resource: x``).
|
||||
Both parsed "successfully" into the wrong *type*, and a pointer parked in
|
||||
|
|
@ -648,3 +702,99 @@ def _reject_mapping_construct(value):
|
|||
raise OKFFrontmatterError(
|
||||
"a mapping is not expressible in OKF frontmatter: %r" % (value,)
|
||||
)
|
||||
|
||||
|
||||
def _parse_flow_mapping(value):
|
||||
"""Parse ``{ key: value, ... }`` into a typed dict, or refuse it (G3).
|
||||
|
||||
Returns ``None`` when ``value`` does not open a flow mapping, so the caller
|
||||
falls through to the unchanged scalar rules. Otherwise the value either
|
||||
parses into a ``dict`` of allowlisted keys with plain-scalar leaves, or
|
||||
raises - it never degrades into a string, which is the defect closed in
|
||||
1.1.0 and not reopened here.
|
||||
|
||||
Why the mapping class needed *a* form at all: OKF v0.2 writes its whole
|
||||
trust and provenance layer as mappings, and SPEC.md @ ``62432a09`` uses flow
|
||||
form in its own examples (§5.1 ``usage_window``, §5.2 ``generated`` /
|
||||
``verified``). §11 goes further than "should": a consumer *MUST* treat a
|
||||
bare ``verified`` mapping as a one-element list - a rule that presupposes
|
||||
the mapping parses. With no form, 0 of 53 upstream concepts reached the
|
||||
gate, and no threshold would have changed that.
|
||||
|
||||
Why this form is safe: the allowlist inspects **every key**, which is the
|
||||
property that actually carried the security in T2 - the blanket refusal was
|
||||
the enforcement, not the point. Admitted, ground-truthed against PyYAML
|
||||
6.0.3:
|
||||
|
||||
- one flow mapping per value, closed on the same line (``{ a: b }``);
|
||||
- keys on :data:`_MAPPING_KEY_ALLOWLIST` and matching ``_KEY_RE``, no
|
||||
duplicates - PyYAML resolves a duplicate last-wins, which is a way to
|
||||
show one claim and mean another;
|
||||
- plain-scalar leaves only, each run through the *unchanged*
|
||||
``_reject_dangerous_value`` / ``_reject_mapping_construct`` predicates, so
|
||||
a leaf can no more open an anchor, a tag or a nested mapping than a
|
||||
top-level scalar can.
|
||||
|
||||
Refused, each on its own rule: nested collections (``{ a: { b: c } }``,
|
||||
``{ a: [1] }``), quoted leaves, an empty mapping, an unclosed or
|
||||
trailing-junk value (``{ a: b } x``, which PyYAML also refuses), a key
|
||||
outside the allowlist, and ``{a:b}`` - which PyYAML reads as the *key*
|
||||
``a:b``, not as a scalar, and which the required ``": "`` separator catches.
|
||||
|
||||
Two deliberate divergences from PyYAML, both toward refusal: a quoted leaf
|
||||
(``{ title: 'a, b' }``) and a trailing comment (``{ a: b } # note``) are
|
||||
conformant YAML that this rejects. Splitting quoted commas correctly needs a
|
||||
quote state machine whose failure mode is *accepting* something YAML would
|
||||
refuse; refusing is the cheaper side to be wrong on, and the keys that
|
||||
plausibly need a comma (``title``, ``author``) only occur inside ``sources``
|
||||
entries, whose block-sequence carrier is refused anyway.
|
||||
"""
|
||||
if not value or value[0] != "{":
|
||||
return None
|
||||
if not value.endswith("}"):
|
||||
raise OKFFrontmatterError(
|
||||
"a flow mapping must be closed by '}' on the same line: %r" % (value,)
|
||||
)
|
||||
|
||||
inner = value[1:-1].strip()
|
||||
if inner.endswith(","): # a trailing comma is legal YAML; one, and only one
|
||||
inner = inner[:-1].strip()
|
||||
if not inner:
|
||||
raise OKFFrontmatterError("an empty flow mapping carries nothing: %r" % (value,))
|
||||
for char in "{}[]":
|
||||
if char in inner:
|
||||
raise OKFFrontmatterError(
|
||||
"a flow mapping admits scalar leaves only, not %r: %r" % (char, value)
|
||||
)
|
||||
for quote in _QUOTE_STARTS:
|
||||
if quote in inner:
|
||||
raise OKFFrontmatterError(
|
||||
"a quoted scalar inside a flow mapping is not a supported form: %r"
|
||||
% (value,)
|
||||
)
|
||||
|
||||
mapping = {}
|
||||
for entry in inner.split(","):
|
||||
entry = entry.strip()
|
||||
key, sep, leaf = entry.partition(": ")
|
||||
if not sep:
|
||||
raise OKFFrontmatterError(
|
||||
"a flow-mapping entry must be 'key: value': %r" % (entry,)
|
||||
)
|
||||
key = key.strip()
|
||||
leaf = leaf.strip()
|
||||
if not _KEY_RE.match(key):
|
||||
raise OKFFrontmatterError("invalid flow-mapping key: %r" % (key,))
|
||||
if key not in _MAPPING_KEY_ALLOWLIST:
|
||||
raise OKFFrontmatterError(
|
||||
"flow-mapping key %r is not on the OKF mapping allowlist: %r"
|
||||
% (key, value)
|
||||
)
|
||||
if key in mapping:
|
||||
raise OKFFrontmatterError(
|
||||
"duplicate flow-mapping key %r: %r" % (key, value)
|
||||
)
|
||||
_reject_dangerous_value(leaf)
|
||||
_reject_mapping_construct(leaf)
|
||||
mapping[key] = leaf
|
||||
return mapping
|
||||
|
|
|
|||
|
|
@ -596,9 +596,11 @@ _DEGRADED_TO_STRING = [
|
|||
@pytest.mark.parametrize("cid,fm,yaml_reads_as", _DEGRADED_TO_STRING,
|
||||
ids=[c[0] for c in _DEGRADED_TO_STRING])
|
||||
def test_a_mapping_construct_never_degrades_into_a_string(cid, fm, yaml_reads_as):
|
||||
# The mapping *class* has no expressible form through T2 — so a mapping
|
||||
# construct must RAISE, never parse "successfully" into the wrong type. A
|
||||
# consumer reading frontmatter["sources"][0].get("uri") must not be handed a str.
|
||||
# None of these shapes is the one form T2 admits (G3, the allowlisted flow
|
||||
# mapping) — so each must RAISE, never parse "successfully" into the wrong
|
||||
# type. A consumer reading frontmatter["sources"][0].get("uri") must not be
|
||||
# handed a str, and that holds whether the mapping class has no expressible
|
||||
# form or one.
|
||||
with pytest.raises(OKFFrontmatterError):
|
||||
parse_frontmatter(f"---\nid: x\n{fm}---\n\nbody\n")
|
||||
|
||||
|
|
@ -651,13 +653,17 @@ def test_pointer_in_a_degraded_mapping_no_longer_reaches_the_consumer_tree(cid,
|
|||
assert result.disposition is Disposition.FAIL_SECURE, "hole reopened — see LIMITATIONS.md"
|
||||
|
||||
|
||||
def test_every_route_to_a_mapping_fails_on_a_different_rule():
|
||||
# The v0.2 wall is not a choice between two forms where one is better: ALL three
|
||||
# ways to express a mapping fail, each on its own rule, so the mapping *class* has
|
||||
# no expressible form through T2. v0.2's `generated` IS a mapping (`by` required
|
||||
# when present), so it cannot be expressed at all.
|
||||
def test_exactly_one_route_to_a_mapping_is_expressible():
|
||||
# Was: ALL FOUR routes failed, each on its own rule, so the mapping *class* had
|
||||
# no expressible form (and v0.2's `generated` could not be written at all). G3
|
||||
# opens exactly ONE of them - the allowlisted flow form - and the other three
|
||||
# still fail, each on its own rule. That the openable route is the one whose
|
||||
# every key the allowlist inspects is the whole design: block, dotted and inline
|
||||
# give the allowlist nothing to inspect, so they stay shut.
|
||||
assert parse_frontmatter("---\nid: x\ngenerated: { by: x, at: y }\n---\n\nbody\n")[0][
|
||||
"generated"] == {"by": "x", "at": "y"}
|
||||
|
||||
routes = {
|
||||
"flow": "generated: { by: x, at: y }\n",
|
||||
"block": "generated:\n by: x\n",
|
||||
"dotted": "generated.by: x\n",
|
||||
"inline": "generated: by: x\n",
|
||||
|
|
@ -667,11 +673,10 @@ def test_every_route_to_a_mapping_fails_on_a_different_rule():
|
|||
with pytest.raises(OKFFrontmatterError) as exc:
|
||||
parse_frontmatter(f"---\nid: x\n{fm}---\n\nbody\n")
|
||||
errors[name] = str(exc.value)
|
||||
assert "indicator" in errors["flow"]
|
||||
assert "nested mappings" in errors["block"]
|
||||
assert "key" in errors["dotted"]
|
||||
assert "mapping" in errors["inline"]
|
||||
assert len(set(errors.values())) == 4, "routes must fail distinctly, not collapse"
|
||||
assert len(set(errors.values())) == 3, "routes must fail distinctly, not collapse"
|
||||
|
||||
|
||||
_BLOCK_LIST_ITEM_SHAPES = [
|
||||
|
|
@ -698,7 +703,9 @@ def test_two_keys_per_item_is_where_the_block_list_hard_rejects():
|
|||
|
||||
|
||||
@pytest.mark.parametrize("fm", [
|
||||
"generated: { by: x, at: y }\n", "sources: [{ id: a }]\n", "tags: [a, b]\n",
|
||||
# The flow row carries a key OFF the G3 allowlist: the shape is admitted, the
|
||||
# key is not, so this stays a T2 rejection and the door A/B half still holds.
|
||||
"generated: { by: x, tool: y }\n", "sources: [{ id: a }]\n", "tags: [a, b]\n",
|
||||
"generated:\n by: x\n", "generated.by: x\n",
|
||||
"sources:\n - id: a\n resource: file://x\n",
|
||||
])
|
||||
|
|
@ -738,3 +745,161 @@ def test_link_extraction_survives_the_redos_fix():
|
|||
assert extract_link_targets("see [x](./a.md) and [y](/b.md)") == ["./a.md", "/b.md"]
|
||||
assert extract_link_targets("[a b](./c.md)") == ["./c.md"]
|
||||
assert extract_link_targets("text [](./t.md)") == ["./i.png"]
|
||||
|
||||
|
||||
# --- G3: the typed, allowlisted mapping form (2026-08-21) --------------------
|
||||
# Door 1 of three (operator decision, 2026-08-21). The mapping *class* had no
|
||||
# expressible form, and OKF v0.2 writes its whole trust and provenance layer as
|
||||
# mappings — SPEC.md @ 62432a09 §5.2 uses flow form in its own examples, and §11
|
||||
# carries a hard MUST that presupposes they parse ("consumers MUST treat a bare
|
||||
# `verified` mapping as a one-element list"). A consumer measured 0 of 53
|
||||
# upstream concepts through the gate. This admits ONE shape: a flow mapping whose
|
||||
# every key is on the allowlist and whose every leaf is a plain scalar.
|
||||
|
||||
def test_spec_flow_mapping_parses_into_a_typed_mapping():
|
||||
# SPEC.md §5.2, verbatim. This is the red test: it must fail before the form
|
||||
# exists and pass after, with a real dict — never a degraded string.
|
||||
doc = (
|
||||
"---\ntype: table\n"
|
||||
"generated: { by: reference_agent/gemini-2.5-pro, at: 2026-06-20T22:53:05Z }\n"
|
||||
"---\nbody\n"
|
||||
)
|
||||
assert parse_frontmatter(doc)[0]["generated"] == {
|
||||
"by": "reference_agent/gemini-2.5-pro",
|
||||
"at": "2026-06-20T22:53:05Z",
|
||||
}
|
||||
|
||||
|
||||
def test_spec_bare_verified_mapping_parses():
|
||||
# SPEC.md §5.2's bare form, which §11 turns into a hard MUST for consumers
|
||||
# ("MUST treat a bare `verified` mapping as a one-element list") - a rule that
|
||||
# cannot be obeyed by a consumer that cannot parse the mapping.
|
||||
doc = "---\ntype: table\nverified: { by: human:ahormati, at: 2026-06-25T09:00:00Z }\n---\nb\n"
|
||||
assert parse_frontmatter(doc)[0]["verified"] == {
|
||||
"by": "human:ahormati", "at": "2026-06-25T09:00:00Z"}
|
||||
|
||||
|
||||
def test_spec_verified_list_of_flow_mappings_parses():
|
||||
# §5.2's list form. This is the SAME typed form in list position, not the
|
||||
# block-sequence-with-one-key route (`- uri: x`), which stays shut below.
|
||||
doc = (
|
||||
"---\ntype: table\nverified:\n"
|
||||
" - { by: human:ahormati, at: 2026-06-25T09:00:00Z }\n"
|
||||
" - { by: process:finance-nightly, at: 2026-06-26T02:00:00Z }\n"
|
||||
"---\nbody\n"
|
||||
)
|
||||
assert parse_frontmatter(doc)[0]["verified"] == [
|
||||
{"by": "human:ahormati", "at": "2026-06-25T09:00:00Z"},
|
||||
{"by": "process:finance-nightly", "at": "2026-06-26T02:00:00Z"},
|
||||
]
|
||||
|
||||
|
||||
def test_spec_usage_window_parses():
|
||||
doc = "---\ntype: table\nusage_window: { from: 2026-06-01T00:00:00Z, to: 2026-06-30T00:00:00Z }\n---\nb\n"
|
||||
assert parse_frontmatter(doc)[0]["usage_window"] == {
|
||||
"from": "2026-06-01T00:00:00Z", "to": "2026-06-30T00:00:00Z"}
|
||||
|
||||
|
||||
def test_an_unknown_key_inside_a_mapping_is_still_rejected():
|
||||
# The rejection side of the allowlist. Without this test the allowlist could
|
||||
# silently grow to "anything" - or be emptied - and nothing would fail.
|
||||
with pytest.raises(OKFFrontmatterError) as exc:
|
||||
parse_frontmatter("---\nid: x\ngenerated: { by: a, tool: shell }\n---\n\nbody\n")
|
||||
assert "allowlist" in str(exc.value)
|
||||
|
||||
|
||||
def test_the_allowlist_is_not_empty_and_admits_only_the_spec_keys():
|
||||
# Both directions of the same guard: a shrunk allowlist breaks the first
|
||||
# assertion, a widened one the second.
|
||||
for key in ("by", "at", "from", "to", "id", "title", "author", "usage_count",
|
||||
"last_modified"):
|
||||
assert parse_frontmatter(f"---\nid: x\nk: {{ {key}: v }}\n---\n\nb\n")[0]["k"] == {key: "v"}
|
||||
for key in ("resource", "executor", "attester", "runtime", "command", "uri"):
|
||||
with pytest.raises(OKFFrontmatterError):
|
||||
parse_frontmatter(f"---\nid: x\nk: {{ {key}: v }}\n---\n\nb\n")
|
||||
|
||||
|
||||
_FLOW_REJECTED = [
|
||||
# (id, value, what PyYAML 6.0.3 makes of it)
|
||||
("nested mapping", "{ by: { at: x } }", "a nested mapping"),
|
||||
("nested sequence", "{ by: [a, b] }", "a sequence leaf"),
|
||||
("anchor leaf", "{ by: &a x }", "an anchor definition, silently"),
|
||||
("tag leaf", "{ by: !!python/object:os.system x }", "refused outright"),
|
||||
("block scalar leaf", "{ by: | }", "a scanner error"),
|
||||
("nested colon leaf", "{ by: sub: v }", "refused outright"),
|
||||
("no space after colon", "{by:x}", "the KEY 'by:x', not a scalar"),
|
||||
("quoted leaf", "{ title: 'a, b' }", "a scalar - we refuse, deliberately"),
|
||||
("empty mapping", "{}", "an empty mapping"),
|
||||
("empty leaf", "{ by: }", "None"),
|
||||
("unclosed", "{ by: x", "a parse error"),
|
||||
("trailing junk", "{ by: x } more", "a parse error"),
|
||||
("duplicate key", "{ by: a, by: b }", "last-wins, silently"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cid,value,yaml_reads_as", _FLOW_REJECTED,
|
||||
ids=[c[0] for c in _FLOW_REJECTED])
|
||||
def test_the_mapping_form_admits_scalar_leaves_on_allowlisted_keys_only(cid, value, yaml_reads_as):
|
||||
with pytest.raises(OKFFrontmatterError):
|
||||
parse_frontmatter(f"---\nid: x\ngenerated: {value}\n---\n\nbody\n")
|
||||
|
||||
|
||||
def test_a_rejected_mapping_never_degrades_into_a_string():
|
||||
# The 1.1.0 defect, re-asserted against the NEW form: a refused mapping must
|
||||
# raise, not arrive as a str a consumer will .get() a key out of.
|
||||
for value in ("{ by: { at: x } }", "{ tool: shell }", "{ by: x"):
|
||||
with pytest.raises(OKFFrontmatterError):
|
||||
parse_frontmatter(f"---\nid: x\ngenerated: {value}\n---\n\nbody\n")
|
||||
|
||||
|
||||
def test_the_admitted_mapping_is_a_dict_not_a_string():
|
||||
value = parse_frontmatter("---\nid: x\ngenerated: { by: a, at: b }\n---\n\nb\n")[0]["generated"]
|
||||
assert isinstance(value, dict), "a typed form that arrives as a str is the 1.1.0 defect"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cid,fm", [
|
||||
("block sequence, one key", "attester:\n - resource: attesters/sql_equality.py\n"),
|
||||
("inline second colon", "attester: resource: attesters/sql_equality.py\n"),
|
||||
("block mapping", "attester:\n resource: attesters/sql_equality.py\n"),
|
||||
("flow mapping, pointer key", "attester: { resource: attesters/sql_equality.py }\n"),
|
||||
])
|
||||
def test_the_pointer_routes_stay_shut(cid, fm):
|
||||
# G3 is additive: none of the routes that put an executable-code pointer in a
|
||||
# key the https allowlist never inspects is reopened. The fourth row is why
|
||||
# `resource` is off the allowlist - the form would otherwise have carried the
|
||||
# door-C pointer through in typed clothes instead of degraded ones.
|
||||
doc = f"---\nid: x\ntype: Attested Computation\n{fm}---\n\nbody\n"
|
||||
with pytest.raises(OKFFrontmatterError):
|
||||
parse_frontmatter(doc)
|
||||
assert import_bundle({"computations/x.md": doc}).disposition is Disposition.FAIL_SECURE
|
||||
|
||||
|
||||
def test_injection_in_a_mapping_leaf_is_caught_by_the_scan():
|
||||
# T1 is not weakened by the new shape: a mapping leaf is scanned exactly like a
|
||||
# scalar value or a list item. A typed form that parses but is not scanned would
|
||||
# be a hole, not a fix.
|
||||
doc = f"---\ntype: table\ngenerated: {{ by: {_INJECTION} }}\n---\nclean body\n"
|
||||
assert scan_concept(doc).found is True
|
||||
|
||||
|
||||
def test_injection_in_a_listed_mapping_leaf_is_caught_by_the_scan():
|
||||
doc = f"---\ntype: table\nverified:\n - {{ by: {_INJECTION} }}\n---\nclean body\n"
|
||||
assert scan_concept(doc).found is True
|
||||
|
||||
|
||||
def test_a_conformant_v02_trust_layer_now_reaches_the_gate():
|
||||
# The measured consequence: a consumer reported 0 of 53 upstream concepts through
|
||||
# the gate, because every one of them carries §5.2 trust frontmatter.
|
||||
doc = (
|
||||
"---\n"
|
||||
"type: table\n"
|
||||
"title: Users\n"
|
||||
"resource: https://example.com/users\n"
|
||||
"generated: { by: reference_agent/gemini-2.5-pro, at: 2026-06-20T22:53:05Z }\n"
|
||||
"verified: { by: human:ahormati, at: 2026-06-25T09:00:00Z }\n"
|
||||
"usage_window: { from: 2026-06-01T00:00:00Z, to: 2026-06-30T00:00:00Z }\n"
|
||||
"---\nThe users table.\n"
|
||||
)
|
||||
result = import_bundle({"tables/users.md": doc})
|
||||
assert result.disposition is Disposition.WARN
|
||||
assert result.concepts[0].error is None
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue