open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Releases 4 Activity

feat(okf): sources becomes expressible, and the parent key is what admits resource

Browse source 
Order 20260902T150716Z from .claude -- a K5 blocker in the OKF programme.
`parse_frontmatter` rejected `sources` in every form the spec and its
producers actually use. Measured 02.09 by two consumers independently:
`sources: [{ id: a, resource: x }]` raised on the `[` indicator (one entry
as well as two), and the block sequence of block mappings -- SPEC.md 5.1's
OWN example -- raised "nested mappings are not supported". `resource` is
REQUIRED within a `sources` entry, so the whole provenance family was
unwritable and a bundle written the way the spec documents it was refused.

Measured against the spec before coding, not reasoned: 5.1's example block
is the canonical carrier for a REQUIRED field and 11.1 defines conformance
as parseable frontmatter, so refusing it refuses a conformant bundle. Both
carriers now parse to the same list of dicts.

The load-bearing change is not the carrier, it is WHO admits `resource`.
1.2.0 left it off the allowlist arguing the parser could not tell
`sources[].resource` (5.1, a citation) from `executor.resource` /
`attester.resource` (10, a pointer to code to be run -- the door-C route
closed in 1.1.0). That premise was false: the owning key is in scope at
every call site and was simply never threaded through. It is threaded now,
so the discrimination is structural, and door C stays shut through EVERY
carrier including the two this adds -- pinned by a new test that drives
`executor`/`attester` through all four.

Refusal stays the default elsewhere. A flow sequence of plain scalars
(`tags: [a, b]`) still raises: the sequence carrier is opened for the flow
mapping element and nothing else. A `sources` entry admits scalar leaves
only, so 5.1's optional PER-ENTRY `usage_window` is refused -- no nesting
past depth 1 is a security property and it was not spent here; registered
as a conformance gap rather than left as an oversight. A block list may not
mix scalars and mappings, because a consumer reading `entry.get("id")` over
one gets an AttributeError off the first str.

New residual registered: `sources[].resource` is scanned as text (T1) but
never URL-validated. T3's https allowlist cannot reach it without
over-blocking conformant bundles -- 5.1 permits bundle-relative paths and
scope descriptors, and the producers' own golden emits `resource: fixture`.
A consumer that dereferences it must call `validate_resource_url` itself.

Suite 834 -> 859 green. 25 new rows; four pre-existing rows changed because
this release changed the behaviour they pinned, two of them renamed since
their names asserted the old invariant (`exactly_one_route_to_a_mapping`,
`two_keys_per_item_is_where_the_block_list_hard_rejects`). Not "unchanged".
130/130 classes, 6/6 gaps hold, 44 -> 45 limitations, ReDoS 0/152 (the
sweep adds no evidence here -- this change adds no regex and the splitting
is linear). Six version surfaces bumped by hand, no sed. Re-measured alone
after the bump.

No exported surface changed; no detector behaviour and no calibration
changed.
This commit is contained in:
Kjell Tore Guttormsen 2026-09-02 17:21:26 +02:00
commit a965e8ac5b
10 changed files with 555 additions and 105 deletions
Download patch file Download diff file Expand all files Collapse all files

2
src/llm_ingestion_guard/__init__.py
View file

@ -63,7 +63,7 @@ from .grounding import (
)
from . import okf
__version__ = "1.2.0"
__version__ = "1.3.0"
# --- §6 bookends: the two library-side halves around the transform ---------

267
src/llm_ingestion_guard/okf.py
View file

@ -65,32 +65,49 @@ _KEY_RE = re.compile(r"^[A-Za-z0-9_][A-Za-z0-9_-]*$")
# A plain OKF scalar cannot *begin* with a YAML structural indicator. Any value
# starting with one signals an anchor (&), alias (*), explicit tag (!), block
# scalar (|, >), flow collection ([ ] { }), directive (%) or reserved char
# (@ `) — all outside the supported subset and all rejected. `{` is tried as the
# allowlisted mapping form FIRST (G3); it reaches this predicate only as a leaf
# inside one, where a nested collection is refused before it can be read.
# (@ `) — all outside the supported subset and all rejected. `{` and `[` are
# tried as the allowlisted mapping form (G3) and the flow sequence of them (G30)
# FIRST; they reach this predicate only as a leaf inside one, where a nested
# collection is refused before it can be read.
_DANGEROUS_VALUE_STARTS = frozenset("&*!|>[]{}%@`")
# A quoted scalar is a scalar in YAML however many colons it carries, so the
# mapping check steps aside for one. The quotes are retained rather than
# stripped — a pre-existing divergence, pinned in tests/test_okf.py.
_QUOTE_STARTS = frozenset("\"'")
# G3 — the one mapping form T2 can express (operator decision, 2026-08-21).
# G3 - the one mapping form T2 can express (operator decision, 2026-08-21).
# Every key inside a mapping must be on this allowlist: the form is safe because
# the allowlist inspects each key, not because mappings became trusted. The keys
# are the ones OKF v0.2 names inside a mapping - `by`/`at` (SPEC.md @ 62432a09
# §5.2 `generated`/`verified`), `from`/`to` (§5.1 `usage_window`) and the
# `sources`-entry fields (§5.1). `resource` is the one §5.1 key deliberately
# LEFT OFF: it is a pointer rather than a label, it is the only key T3 exists
# for, and admitting it inside a mapping would re-open the door-C route closed
# in 1.1.0 (`executor: {resource: skills/run.md}` puts an executable-code
# pointer in a key the https allowlist never inspects). It costs nothing today,
# because the conformant carrier for `sources[].resource` is the block-sequence
# of block-mappings, which this form does not admit either way.
# §5.2 `generated`/`verified`) and `from`/`to` (§5.1 `usage_window`), plus the
# §5.1 `sources`-entry labels.
_MAPPING_KEY_ALLOWLIST = frozenset({
"by", "at", "from", "to", "id", "title", "author", "usage_count",
"last_modified",
})
# G30 - the two §5.1 keys admitted inside a `sources` entry and NOWHERE else
# (operator decision, 2026-09-02). `resource` is REQUIRED within a `sources`
# entry, so leaving it off left the whole provenance family unwritable; but the
# same field name in §10 (`executor.resource`, `attester.resource`) names run
# instructions and code - the door-C route closed in 1.1.0. 1.2.0 argued the
# parser could not tell the two apart without parent-key context it did not
# have. That premise was false: the owning key is in scope at every call site
# below, it was simply never threaded through. It is threaded now, so the
# discrimination is structural rather than a judgement about the value.
# `usage_window` is allowlisted here for accuracy of refusal - §5.1 permits it
# per entry, and it is then refused on the depth rule (a mapping inside a
# mapping, which this parser admits at no key) rather than refused as if the
# key were unknown.
_SOURCES_ENTRY_KEYS = frozenset({"resource", "usage_window"})
def _allowed_mapping_keys(parent_key):
"""The mapping-key allowlist for a mapping owned by ``parent_key``."""
if parent_key == "sources":
return _MAPPING_KEY_ALLOWLIST | _SOURCES_ENTRY_KEYS
return _MAPPING_KEY_ALLOWLIST
class OKFError(Exception):
"""Base class for OKF adapter rejections."""
@ -184,8 +201,10 @@ def _value_regions(value):
A mapping value (G3) is a new *shape* on this surface, not a new exemption:
its leaves are scanned exactly like a scalar or a list item, so an injection
parked in ``generated: { by: ... }`` reaches ``scan_output`` like any other
frontmatter text. Mapping *keys* are not scanned because they cannot carry
attacker text - the allowlist admits nine fixed names and nothing else.
frontmatter text. The same holds for a *list* of mappings (G30, ``sources``),
which this function already flattens through its list branch. Mapping *keys*
are not scanned because they cannot carry attacker text - the allowlist
admits a fixed, per-parent name set and nothing else.
"""
if isinstance(value, dict):
return [leaf for leaf in value.values() if leaf]
@ -616,16 +635,22 @@ def _parse_flat(fm_lines):
raise OKFFrontmatterError("invalid frontmatter key: %r" % key)
if value == "":
items, i = _consume_block_list(fm_lines, i + 1)
items, i = _consume_block_list(fm_lines, i + 1, key)
result[key] = items if items is not None else ""
continue
mapping = _parse_flow_mapping(value)
mapping = _parse_flow_mapping(value, key)
if mapping is not None:
result[key] = mapping
i += 1
continue
sequence = _parse_flow_sequence(value, key)
if sequence is not None:
result[key] = sequence
i += 1
continue
_reject_dangerous_value(value)
_reject_mapping_construct(value)
result[key] = value
@ -634,14 +659,26 @@ def _parse_flat(fm_lines):
return result
def _consume_block_list(fm_lines, start):
def _consume_block_list(fm_lines, start, parent_key=None):
"""Consume `` - item`` lines following a bare ``key:``.
Returns ``(items, next_index)`` — ``items`` is ``None`` (and ``next_index``
Returns ``(items, next_index)`` - ``items`` is ``None`` (and ``next_index``
unchanged) when no list item follows, so the caller can treat the key as an
empty scalar and let the next line trip the nested-structure guard.
An item is one of three shapes, decided by the item text alone: a flow
mapping (G3), a block mapping (G30 - an unquoted ``key: value`` opening a
run of more-indented sibling entries, which is SPEC.md §5.1's own carrier
for ``sources``), or a plain scalar. ``parent_key`` is the key that owns the
list; it decides the mapping-key allowlist, which is how ``sources[].resource``
is admitted while ``executor``/``attester`` ``resource`` stays refused.
A list may not mix scalars and mappings. YAML permits it, but a consumer
iterating ``sources`` and reading ``entry.get("id")`` gets an
``AttributeError`` off the first ``str`` - refusing is the cheaper failure.
"""
items = []
kinds = set()
i = start
n = len(fm_lines)
while i < n:
@ -650,24 +687,113 @@ def _consume_block_list(fm_lines, start):
if stripped == "" or stripped.startswith("#"):
i += 1
continue
if raw[:1] in (" ", "\t") and stripped.startswith("- "):
item = stripped[2:].strip()
mapping = _parse_flow_mapping(item)
if mapping is not None:
items.append(mapping)
i += 1
continue
_reject_dangerous_value(item)
_reject_mapping_construct(item)
items.append(item)
if not (raw[:1] in (" ", "\t") and stripped.startswith("- ")):
break
item = stripped[2:].strip()
mapping = _parse_flow_mapping(item, parent_key)
if mapping is not None:
items.append(mapping)
kinds.add("mapping")
i += 1
continue
break
entry = _block_mapping_entry(item)
if entry is not None:
mapping, i = _consume_block_mapping(fm_lines, i + 1, entry, parent_key)
items.append(mapping)
kinds.add("mapping")
continue
_reject_dangerous_value(item)
_reject_mapping_construct(item)
items.append(item)
kinds.add("scalar")
i += 1
if len(kinds) > 1:
raise OKFFrontmatterError(
"a block list may not mix scalar items and mappings: %r" % (parent_key,)
)
if not items:
return None, start
return items, i
def _block_mapping_entry(text):
"""Read ``text`` as one ``key: value`` block-mapping entry, or return ``None``.
The trigger is deliberately the same shape ``_reject_mapping_construct``
uses to *refuse* a scalar: an unquoted ``": "``. What changes in 1.3.0 is
only what happens next - the entry is admitted key-by-key against the
allowlist instead of refused wholesale. Every shape that is a scalar to
PyYAML stays one here: a quoted item, a colon with no space
(``domain:security``, ``https://e.com:8443/a``) and a trailing colon all
return ``None`` and fall through to the unchanged scalar rules.
"""
if not text or text[0] in _QUOTE_STARTS:
return None
key, sep, leaf = text.partition(": ")
if not sep:
return None
key = key.strip()
if not _KEY_RE.match(key):
return None
return key, leaf.strip()
def _consume_block_mapping(fm_lines, start, first_entry, parent_key):
"""Consume the sibling entries of a block mapping opened by a ``- `` item.
Returns ``(mapping, next_index)``. A sibling is an indented line that does
not open a new list item; the run ends at a blank line, a comment, a new
``- `` item, or a line at column zero. Depth is capped at one by giving the
leaves the *unchanged* scalar predicates: a nested collection opens with
``{`` or ``[`` and is refused by ``_reject_dangerous_value``, and a further
block level is refused by ``_reject_mapping_construct``.
"""
allowed = _allowed_mapping_keys(parent_key)
mapping = {}
_admit_mapping_entry(mapping, first_entry[0], first_entry[1], allowed, parent_key)
i = start
n = len(fm_lines)
while i < n:
raw = fm_lines[i]
stripped = raw.strip()
if stripped == "" or stripped.startswith("#"):
break
if raw[:1] not in (" ", "\t") or stripped.startswith("- "):
break
entry = _block_mapping_entry(stripped)
if entry is None:
_reject_dangerous_value(stripped)
_reject_mapping_construct(stripped)
raise OKFFrontmatterError(
"a block-mapping entry must be 'key: value': %r" % (raw,)
)
_admit_mapping_entry(mapping, entry[0], entry[1], allowed, parent_key)
i += 1
return mapping, i
def _admit_mapping_entry(mapping, key, leaf, allowed, parent_key):
"""Admit one mapping entry, or raise. The single gate both carriers pass."""
if not _KEY_RE.match(key):
raise OKFFrontmatterError("invalid mapping key: %r" % (key,))
if key not in allowed:
raise OKFFrontmatterError(
"mapping key %r is not on the OKF mapping allowlist under %r"
% (key, parent_key)
)
if key in mapping:
raise OKFFrontmatterError("duplicate mapping key %r" % (key,))
_reject_dangerous_value(leaf)
_reject_mapping_construct(leaf)
mapping[key] = leaf
def _reject_dangerous_value(value):
if value and value[0] in _DANGEROUS_VALUE_STARTS:
raise OKFFrontmatterError(
@ -704,7 +830,7 @@ def _reject_mapping_construct(value):
)
def _parse_flow_mapping(value):
def _parse_flow_mapping(value, parent_key=None):
"""Parse ``{ key: value, ... }`` into a typed dict, or refuse it (G3).
Returns ``None`` when ``value`` does not open a flow mapping, so the caller
@ -773,6 +899,7 @@ def _parse_flow_mapping(value):
% (value,)
)
allowed = _allowed_mapping_keys(parent_key)
mapping = {}
for entry in inner.split(","):
entry = entry.strip()
@ -781,20 +908,68 @@ def _parse_flow_mapping(value):
raise OKFFrontmatterError(
"a flow-mapping entry must be 'key: value': %r" % (entry,)
)
key = key.strip()
leaf = leaf.strip()
if not _KEY_RE.match(key):
raise OKFFrontmatterError("invalid flow-mapping key: %r" % (key,))
if key not in _MAPPING_KEY_ALLOWLIST:
raise OKFFrontmatterError(
"flow-mapping key %r is not on the OKF mapping allowlist: %r"
% (key, value)
)
if key in mapping:
raise OKFFrontmatterError(
"duplicate flow-mapping key %r: %r" % (key, value)
)
_reject_dangerous_value(leaf)
_reject_mapping_construct(leaf)
mapping[key] = leaf
_admit_mapping_entry(mapping, key.strip(), leaf.strip(), allowed, parent_key)
return mapping
def _parse_flow_sequence(value, parent_key=None):
"""Parse ``[{ ... }, { ... }]`` into a list of typed dicts, or refuse it (G30).
Returns ``None`` when ``value`` does not open a flow sequence, so the caller
falls through to the unchanged scalar rules - where ``[`` is still a
disallowed indicator. This carrier is opened for the flow-mapping element
and nothing else: it is the form the OKF producers emit for ``sources``
(measured 02.09 against llm-ingestion-okf's golden bundle, where a
one-element sequence raised on the ``[`` just as a two-element one did).
A flow sequence of plain *scalars* (``tags: [a, b, c]``) stays refused. It
is a different shape with its own quoting and comma-splitting problem, whose
failure mode would be accepting something YAML reads differently - and the
block-sequence carrier already covers it for every consumer measured so far.
Elements are split on ``}`` rather than on commas, which is sound precisely
because ``_parse_flow_mapping`` admits no nested collection: a ``}`` inside
an element cannot occur, so the first ``}`` after ``{`` always closes it.
Anything between elements that is not a separating comma is refused, which
is what makes trailing junk and a mixed sequence fail rather than parse.
"""
if not value or value[0] != "[":
return None
if not value.endswith("]"):
raise OKFFrontmatterError(
"a flow sequence must be closed by ']' on the same line: %r" % (value,)
)
inner = value[1:-1].strip()
if not inner:
raise OKFFrontmatterError("an empty flow sequence carries nothing: %r" % (value,))
items = []
i = 0
n = len(inner)
while True:
while i < n and inner[i] in " \t":
i += 1
if i >= n:
break
if inner[i] != "{":
raise OKFFrontmatterError(
"a flow sequence admits flow mappings only: %r" % (value,)
)
close = inner.find("}", i)
if close == -1:
raise OKFFrontmatterError(
"an unclosed flow mapping inside a flow sequence: %r" % (value,)
)
items.append(_parse_flow_mapping(inner[i:close + 1], parent_key))
i = close + 1
while i < n and inner[i] in " \t":
i += 1
if i >= n:
break
if inner[i] != ",":
raise OKFFrontmatterError(
"trailing junk after a flow-sequence element: %r" % (value,)
)
i += 1
return items