open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Releases 4 Activity

feat(okf): a flow sequence of plain scalars parses, and the corpus number is 6/53

Browse source 
`tags: [a, b, c]` is the form SPEC.md 4.1's own frontmatter skeleton writes out,
and 9/53 upstream reference concepts use it. It raised on the `[` indicator.
It parses now, to the same list its block-sequence sibling already produced.

The predicate is character-level, inside `_parse_flow_sequence`: an element is a
plain scalar only if it is non-empty and carries none of `{ } [ ] : , " ' #`,
and it then passes the unchanged scalar-indicator rule. Everything that would
need YAML semantics to split or unquote still raises - a quoted element (quotes
are retained here, never stripped), a colon or comma inside an element, a
sequence inside a sequence, an empty element, an anchor, an alias. A sequence
may not mix scalars and mappings, the rule the block list already carries, and
the mixing verdict is reached before the element is parsed so the caller is told
about the mix rather than about a key the allowlist would have named instead.
The 1.3.0 `sources` flow-mapping carrier is unchanged and pinned against
regression. Depth 1 is not spent: the elements are leaves.

Measured with the denominator, against the pinned corpus (`_okf-upstream` @
3fcbb9f, 53 non-reserved documents) and the pinned SPEC (`_okf-canonical` @
ad30107). Baseline reproduced first, with a known-positive control, at 0/53.
After: 6/53, all six in acme_retail. It does not close the corpus - 44/53 still
stop on `generated` written as a top-level block mapping, which spends the
no-nesting-past-depth-1 rule and is a security decision, out of scope here.

P1 alone, per the operator decision of 08.09. The two neighbouring predicates
were measured and deliberately not built: a flush-left block sequence and a
folded plain scalar release 0/53 each on their own, and stacked on this one they
still measure 6/53. `_consume_block_list`, the `description` continuation and
the allowlist are untouched.

docs/LIMITATIONS.md's tags/description entry is rewritten against the
measurement: three of its claims were wrong. The parser does have a
sequence-value type (since 1.3.0 - what it lacks is the indentation the corpus
omits); the figure is 6/53, not the 1.2.0-era 4/53; and tags/description are
not the residual that blocks the corpus. README gains the sequence carrier in
the paragraph that already describes the mapping one.

Self-safety: the predicate compiles no regex, so docs/redos-sweep.py cannot see
it. Measured instead on the CPU clock - linear in element length (exponent
0.86-0.99) and in element count (0.97-1.05) over four doublings to 800_000 -
and pinned by two bounds in tests/test_okf.py.

Six rows that pinned the old refusal are re-aimed at the class that still
holds - the quoted element - the way the 1.3.0 rows were when the carrier
opened. One of them lives in src/llm_ingestion_guard/coverage.py, which is why
the src diff is three files rather than one.

Version 1.4.0 in the code only. The CHANGELOG entry stays under Unreleased and
no tag is cut: README's badge and install pin must keep naming a tag that
exists.

Gates after `git add`: 893 passed (was 868), coverage 130/130 + 6/6 gaps,
redos-sweep exit 0, LIMITATIONS still 45 entries.
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 05:34:55 +02:00
commit 3e324a1f86
9 changed files with 341 additions and 59 deletions
Download patch file Download diff file Expand all files Collapse all files

47
CHANGELOG.md
View file

@ -5,6 +5,53 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Added — `tags: [a, b, c]`, the one flow-sequence form SPEC §4.1 writes out
`okf.parse_frontmatter` refused a flow sequence of plain scalars. That is the
form SPEC.md §4.1's own frontmatter skeleton uses for `tags`, and `tags` is
present in 53/53 upstream reference concepts, 9/53 of them in exactly this form.
Admitted now, parsing to the same value its block-sequence sibling already did:
```yaml
tags: [finance, revenue, headline-metric]
```
An element is a plain scalar only if it is non-empty and carries none of
`{ } [ ] : , " ' #`, and it then passes the unchanged scalar-indicator rule.
The refusals that make that a rule rather than a preference: a quoted element
(this parser retains quotes rather than stripping them, so reading one would
hand back a value YAML does not), an element carrying a colon or a comma, a
sequence inside a sequence, an empty element, an anchor or an alias. A sequence
may not **mix** scalars and mappings — the same rule the block list already
carries, for the same reason: a consumer iterating the value and reading
`entry.get("id")` crashes on the first `str`. The `sources` flow-mapping
carrier added in 1.3.0 is unchanged and pinned against regression.
**Measured, with the denominator: this takes the pinned OKF corpus from 0/53 to
6/53** (`_okf-upstream/okf` @ `3fcbb9f`, 53 non-reserved documents; the six are
all in `acme_retail`). It does not close the corpus. **44/53 still stop on
`generated` written as a top-level block mapping** — a form refused on the
no-nesting-past-depth-1 rule, which is a security decision and out of scope
here. Two neighbouring predicates were measured and deliberately not built: a
flush-left block sequence and a folded plain scalar release **0/53** each on
their own and leave the corpus at 6/53 when stacked on this one. Detection
behaviour is not frozen under semver; the exported surface is unchanged.
### Changed
- `docs/LIMITATIONS.md`'s `tags`/`description` entry is rewritten against the
measurement. Three of its claims were wrong: the parser *does* have a
sequence-value type (since 1.3.0 — what it lacks is the indentation the
corpus omits), the figure is 6/53 and not the 1.2.0-era 4/53, and
`tags`/`description` are *not* the residual that blocks the corpus.
- Self-safety: the new predicate is character-level with no regex, so
`docs/redos-sweep.py` cannot see it (it collects compiled patterns). It is
measured instead — linear in both element length (exponent 0.86-0.99) and
element count (0.97-1.05) over four doublings to 800_000, and pinned by two
CPU-clock bounds in `tests/test_okf.py`.
## [1.3.0] — 2026-09-02
### Added — the `sources` provenance family becomes expressible in both spec carriers

21
CLAUDE.md
View file

@ -11,7 +11,11 @@ framework-agnostisk kode.
Referanse-implementasjon: `claude-code-llm-wiki` Stage B (`tools/wiki_ingest/`).
Lexikon-seed: `injection-patterns.mjs` fra `llm-security`-pluginen.
Repoet er på **v1.3.0** — den eksporterte Python-surfacen er frosset under semver
Repoet er på **v1.4.0 i koden, UUTGITT** (`pyproject.toml` + `__init__.py` er
bumpet; README-badge, install-pinnen, ADOPTION-BRIEF og BRIEF står med vilje
igjen på `1.3.0`, som er den siste taggen som FINNES — en install-pin må peke på
en ekte tag). Release-commiten (CHANGELOG-overskrift datert, de fire
dokumentflatene bumpet, tag) er ikke tatt. Den eksporterte Python-surfacen er frosset under semver
(deteksjonsatferd er det IKKE; kalibrering flytter seg i 1.x). Stdlib-kjernen er
bygget og testet (15 moduler +
topp-nivå wiring, showcase + korpus), inkl. OKF-adapter og aktivt-innhold-
@ -26,9 +30,18 @@ nøkkelen avgjør, så `executor`/`attester` sin `resource` (§10, dør C) avvis
gjennom hver eneste bærer. 1.2.0s begrunnelse for å utelate den (parseren
manglet foreldre-kontekst) var målt feil: konteksten var der, den var bare
aldri tredd gjennom. Topp-nivå blokk-mapping, dotted- og inline-kolon-rutene
raiser fortsatt, flow-sekvens av rene skalarer (`tags: [a, b]`) raiser, en
blokkliste kan ikke blande skalarer og mappinger, og en avvist mapping raiser —
den degraderer aldri til en streng (1.1.0-defekten). `sources[].resource`
raiser fortsatt, en blokkliste kan ikke blande skalarer og mappinger, og en
avvist mapping raiser — den degraderer aldri til en streng (1.1.0-defekten).
**Flow-sekvens av rene skalarer (`tags: [a, b]`) PARSER fra 1.4.0** (P1,
operatørbeslutning 08.09) — SPEC §4.1s eget skjelett. Et element er en ren
skalar kun hvis det er ikke-tomt og uten `{ } [ ] : , " ' #`, og så gjelder den
uendrete indikator-regelen; sitert element, kolon/komma i elementet, sekvens i
sekvens, tomt element, anker og alias raiser fortsatt, og en flow-sekvens kan
ikke blande skalarer og mappinger. **Målt med nevner: 0/53 → 6/53** på pinnet
OKF-korpus (`3fcbb9f`). Den bindende skranken er IKKE tags/description, men
`generated` som topp-nivå blokk-mapping (44/53) — den bruker opp dybde-1 og er
en sikkerhetsbeslutning. P2 (blokksekvens uten innrykk) og P3 (foldet plain
scalar) er MÅLT til 0/53 hver og bevisst IKKE bygget. `sources[].resource`
URL-valideres ALDRI (T3 ser kun topp-nivå `resource`) — §5.1 tillater
bundle-relative stier og scope-beskrivelser, så en https-gate ville over-blokkert
konforme bundles; konsumenten må selv kalle `validate_resource_url`. Mode-b `import_bundle` skanner

6
README.md
View file

@ -171,7 +171,11 @@ mapping — `generated: { by: x, at: y }`, `verified: { … }` bare or listed,
allowlist (`by`, `at`, `from`, `to`, `id`, `title`, `author`, `usage_count`,
`last_modified`) with plain-scalar leaves only. A key off that list, a nested
collection or a duplicate key is refused, and `resource` is deliberately not on
it; the block, dotted and inline-colon routes to a mapping still raise. See
it; the block, dotted and inline-colon routes to a mapping still raise. A
*sequence* value has two carriers — the block list, and (as of `1.4.0`) the flow
sequence `tags: [a, b, c]`, which is SPEC §4.1's own skeleton — whose elements
are either all plain scalars or all flow mappings, never a mix. A scalar element
carrying any of `{ } [ ] : , " ' #` is refused rather than guessed at. See
[LIMITATIONS](docs/LIMITATIONS.md) for what that admits and what it still walls
off (a `sources` block list of mappings is still refused); **`resource` https-allowlist** (hard-rejects
`data:`/`javascript:`/`file:` before commit — a reject-gate, not defang);

54
docs/LIMITATIONS.md
View file

@ -123,24 +123,42 @@ items; this is the full list, each with the mechanism.
exported for exactly that. The dangling-or-substituted `executor`/`attester`
pointer question stays out of reach separately: both are top-level block
mappings, a carrier that is still refused.
- **`tags` and `description` block the OKF import corpus universally, before the
trust layer is even reached.** The line-flat frontmatter parser has no
sequence-value type at all: `tags` is present in 53/53 upstream concept
documents — 9/53 as a flow sequence (`[a, b, c]`, rejected on the `[`
indicator) and 44/53 as a block sequence (`- a` / `- b`, rejected as
`"malformed frontmatter line"`) — 100% rejection regardless of form.
`description` is present in 53/53; 29/53 is a folded plain scalar continuing
on an indented second line, which the parser has no continuation-line model
for and misreads as `"nested mappings are not supported"` (the remaining
24/53 are single-line and parse fine). Measured directly on the upstream
reference bundles (`_okf-upstream/okf` @ `3fcbb9f`): removing `tags` alone
lets 4/53 documents pass; removing both `tags` and `description` together
(trust layer untouched) lets the same 4/53 pass, and all four then parse
`generated` correctly as a mapping. **Independent of the mapping-form work
above:** neither `1.2.0`'s flow mapping nor `1.3.0`'s `sources` carriers move
anything on this corpus, because `tags`/`description` reject before `sources`
is ever read. No sequence-value type or continuation-line model exists in the
stdlib-only parser to close this with.
- **`tags` parses in two of the three forms the OKF corpus writes it in;
`description`'s folded form still does not — and neither is what caps the
corpus.** Re-measured 2026-09-07/08 against the pinned upstream reference
bundles (`_okf-upstream/okf` @ `3fcbb9f`, denominator **53** — every
non-reserved `.md`, which is SPEC §11.1's own unit) and the pinned SPEC
(`_okf-canonical` @ `ad30107`). This entry carried three claims that the
measurement showed to be wrong, all three inherited from `1.2.0` and left
standing for twelve days after `1.3.0` moved them; they are corrected here.
- **A sequence-value type exists** — `_consume_block_list` has parsed block
lists of plain scalars since `1.3.0`, and `1.4.0` adds the flow sequence of
plain scalars (`tags: [a, b, c]`, SPEC §4.1's own skeleton). What is missing
is not the type but the **indentation requirement**: `_consume_block_list`
demands a leading space, and the corpus writes `- a` flush at column 0
(36/53 of `tags`, 44/53 of `sources`). Shape census of `tags` over the 53:
36/53 flush block sequence, 9/53 flow sequence, 8/53 single-line scalar.
- **The number is 6/53, not 4/53.** `4` was the `1.2.0` parser's figure.
Measured on `1.4.0`, the corpus goes from **0/53 to 6/53** — all six in
`acme_retail`, being 6 of the 9 documents whose `tags` is a flow sequence;
the other three stop on the mapping-key allowlist (`parameters.name` ×2,
`not.term` ×1), not on `tags`.
- **`tags`/`description` are not the binding constraint.** `description`
alone releases **0/53** — 29/53 write it as a folded plain scalar over an
indented second line, which the parser has no continuation-line model for,
but removing that obstacle by itself lets nothing through. With all three
surface forms closed the corpus still measures **6/53**, because **44/53
stop on `generated` written as a top-level block mapping**, and behind that
wall sit 44/53 single-quoted scalars inside the mapping. That constraint is
refused deliberately: it spends the no-nesting-past-depth-1 rule, which is
a security decision and not a parser detail.
**Said plainly: `1.4.0` takes this corpus to 6/53, not to 53/53.** The two
neighbouring predicates were measured and deliberately **not** built — a
flush-left block sequence and a folded plain scalar each release **0/53** on
their own, and stacked on top of `1.4.0` they still measure 6/53. They would
cost parser surface and buy no measured conformance. The full measurement,
including the SPEC quotations and the per-candidate table, is
`docs/2026-09-07-limitations-44-maaling.md`.
- **A persist gate cannot cover execution risk.** OKF v0.2 introduces concepts whose
purpose is to *name code to be run* (`runtime`, `executor.resource`,
`attester.resource`). This library answers "is this safe to **store**"; executable

2
pyproject.toml
View file

@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "llm-ingestion-guard"
version = "1.3.0"
version = "1.4.0"
description = "Write-time defensive layer for Python pipelines that persist LLM output: sanitize, fence, tool-less quarantined transform, capability isolation, scan before persist, fail-secure."
readme = "README.md"
requires-python = ">=3.10"

2
src/llm_ingestion_guard/__init__.py
View file

@ -63,7 +63,7 @@ from .grounding import (
)
from . import okf
__version__ = "1.3.0"
__version__ = "1.4.0"
# --- §6 bookends: the two library-side halves around the transform ---------

10
src/llm_ingestion_guard/coverage.py
View file

@ -542,8 +542,14 @@ def _build_cases() -> list[Case]:
lambda: okf.parse_frontmatter("---\nkey:\n nested: x\n---\nbody\n"), owasp="LLM10"),
_raise_case("okf", "T2 frontmatter block scalar", "OKFFrontmatterError",
lambda: okf.parse_frontmatter("---\ndesc: |\n block\n---\nbody\n"), owasp="LLM10"),
_raise_case("okf", "T2 frontmatter flow sequence", "OKFFrontmatterError",
lambda: okf.parse_frontmatter("---\ntags: [a, b]\n---\nbody\n"), owasp="LLM10"),
# The plain-scalar flow sequence is ADMITTED as of 1.4.0 (P1, SPEC §4.1's
# own skeleton for `tags`); the class this row measures is the element
# shape the parser would have to interpret rather than read -- here a
# quoted one, which it would have to strip a quote from to return.
_raise_case("okf", "T2 frontmatter flow sequence, quoted element",
"OKFFrontmatterError",
lambda: okf.parse_frontmatter("---\ntags: ['a', 'b']\n---\nbody\n"),
owasp="LLM10"),
_raise_case("okf", "T2 mapping key off the allowlist", "OKFFrontmatterError",
lambda: okf.parse_frontmatter(
"---\ngenerated: { by: a, tool: shell }\n---\nbody\n"), owasp="LLM10"),

102
src/llm_ingestion_guard/okf.py
View file

@ -75,6 +75,14 @@ _DANGEROUS_VALUE_STARTS = frozenset("&*!|>[]{}%@`")
# stripped — a pre-existing divergence, pinned in tests/test_okf.py.
_QUOTE_STARTS = frozenset("\"'")
# P1 - what disqualifies a flow-sequence element from being a plain scalar
# (operator decision, 2026-09-08). Each character is one this parser would have
# to interpret rather than read: the two quotes (retained, never stripped), the
# two splitters, the two collection openers and their closers, and the comment
# indicator. Refusing them is what lets the element be split on commas at the
# character level without a YAML quote state machine.
_FLOW_SCALAR_REFUSED = "{}[]:,\"'#"
# G3 - the one mapping form T2 can express (operator decision, 2026-08-21).
# Every key inside a mapping must be on this allowlist: the form is safe because
# the allowlist inspects each key, not because mappings became trusted. The keys
@ -922,16 +930,29 @@ def _parse_flow_sequence(value, parent_key=None):
(measured 02.09 against llm-ingestion-okf's golden bundle, where a
one-element sequence raised on the ``[`` just as a two-element one did).
A flow sequence of plain *scalars* (``tags: [a, b, c]``) stays refused. It
is a different shape with its own quoting and comma-splitting problem, whose
failure mode would be accepting something YAML reads differently - and the
block-sequence carrier already covers it for every consumer measured so far.
As of 1.4.0 it also carries a sequence of plain *scalars*
(``tags: [a, b, c]``) - SPEC.md §4.1's own skeleton for ``tags``, and the
one candidate form measured to move the upstream corpus at all (0/53 ->
6/53 against ``_okf-upstream`` @ 3fcbb9f, denominator 53; see
docs/2026-09-07-limitations-44-maaling.md). The quoting and comma-splitting
problem that kept it refused is answered by refusing the characters that
create it rather than by parsing them: an element is a plain scalar only if
it is non-empty and carries none of ``{ } [ ] : , " ' #``, and it then
passes the unchanged scalar indicator rule. Everything needing YAML
semantics to split or unquote correctly still raises.
Elements are split on ``}`` rather than on commas, which is sound precisely
because ``_parse_flow_mapping`` admits no nested collection: a ``}`` inside
an element cannot occur, so the first ``}`` after ``{`` always closes it.
Anything between elements that is not a separating comma is refused, which
is what makes trailing junk and a mixed sequence fail rather than parse.
A mapping element is split on ``}`` rather than on commas, which is sound
precisely because ``_parse_flow_mapping`` admits no nested collection: a
``}`` inside an element cannot occur, so the first ``}`` after ``{`` always
closes it. A scalar element runs to the next comma, which cannot occur
inside one. Anything between elements that is not a separating comma is
refused, which is what makes trailing junk fail rather than parse.
A sequence may not mix the two, for the reason the block list may not: a
consumer iterating the value and reading ``entry.get("id")`` gets an
``AttributeError`` off the first ``str``. The mixing verdict is reached
before the element is parsed, so the caller is told about the mix rather
than about a key the allowlist would have complained of instead.
"""
if not value or value[0] != "[":
return None
@ -945,6 +966,7 @@ def _parse_flow_sequence(value, parent_key=None):
raise OKFFrontmatterError("an empty flow sequence carries nothing: %r" % (value,))
items = []
kinds = set()
i = 0
n = len(inner)
while True:
@ -952,17 +974,22 @@ def _parse_flow_sequence(value, parent_key=None):
i += 1
if i >= n:
break
if inner[i] != "{":
raise OKFFrontmatterError(
"a flow sequence admits flow mappings only: %r" % (value,)
)
close = inner.find("}", i)
if close == -1:
raise OKFFrontmatterError(
"an unclosed flow mapping inside a flow sequence: %r" % (value,)
)
items.append(_parse_flow_mapping(inner[i:close + 1], parent_key))
i = close + 1
if inner[i] == "{":
_refuse_mixed_flow_sequence(kinds, "mapping", value)
close = inner.find("}", i)
if close == -1:
raise OKFFrontmatterError(
"an unclosed flow mapping inside a flow sequence: %r" % (value,)
)
items.append(_parse_flow_mapping(inner[i:close + 1], parent_key))
i = close + 1
else:
_refuse_mixed_flow_sequence(kinds, "scalar", value)
end = i
while end < n and inner[end] != ",":
end += 1
items.append(_flow_sequence_scalar(inner[i:end].strip(), value))
i = end
while i < n and inner[i] in " \t":
i += 1
if i >= n:
@ -973,3 +1000,38 @@ def _parse_flow_sequence(value, parent_key=None):
)
i += 1
return items
def _refuse_mixed_flow_sequence(kinds, kind, value):
kinds.add(kind)
if len(kinds) > 1:
raise OKFFrontmatterError(
"a flow sequence may not mix scalar items and mappings: %r" % (value,)
)
def _flow_sequence_scalar(element, value):
"""Read one flow-sequence element as a plain scalar, or refuse it (P1).
Character-level, with no YAML semantics: the element must be non-empty and
carry none of :data:`_FLOW_SCALAR_REFUSED`. That set is not a style rule -
each member is a character whose meaning this parser would have to guess at.
A quote would have to be stripped (this parser retains quotes, so it would
hand back a different value than YAML reads); a comma or a colon would have
to be split on; a bracket or a brace would open a second collection level,
which no carrier here admits; a ``#`` opens a comment. The unchanged
indicator rule then applies to what is left, exactly as it does to a
block-list item, so an anchor or an alias is no more a scalar here.
"""
if not element:
raise OKFFrontmatterError(
"an empty element in a flow sequence carries nothing: %r" % (value,)
)
for char in _FLOW_SCALAR_REFUSED:
if char in element:
raise OKFFrontmatterError(
"a flow-sequence scalar admits plain scalars only, not %r: %r"
% (char, value)
)
_reject_dangerous_value(element)
return element

154
tests/test_okf.py
View file

@ -116,9 +116,13 @@ def test_rejects_unterminated_frontmatter():
def test_rejects_flow_collection():
# inline flow collections are outside the supported subset -> reject, don't
# silently mis-parse the bracket string as a scalar.
doc = "---\ntype: table\ntags: [pii, customers]\n---\nbody\n"
# An inline flow collection this parser cannot read without guessing ->
# reject, don't silently mis-parse the bracket string as a scalar. The
# QUOTED elements are what makes this row stay red: the plain-scalar form
# (`tags: [pii, customers]`) is admitted as of 1.4.0 -- see the P1 block at
# the foot of this file -- while a quoted element would have to be stripped
# to be read, and this parser retains quotes.
doc = "---\ntype: table\ntags: ['pii', 'customers']\n---\nbody\n"
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(doc)
@ -553,7 +557,10 @@ _V02_REJECTED = [
# that reason. SPEC's own §5.1 keys parse; see the G30 block at the foot.
("sources (block list, off-allowlist keys)",
"sources:\n - uri: https://e.com/a\n kind: doc\n"),
("flow sequence", "tags: [a, b, c]\n"),
# The plain-scalar flow sequence is admitted as of 1.4.0 (P1); this row now
# measures the QUOTED element, which stays refused for the same reason it
# always did -- reading it would mean stripping a quote this parser retains.
("flow sequence, quoted elements", "tags: ['a', 'b']\n"),
("flow mapping", "executor: {resource: skills/run.md}\n"),
]
@ -572,6 +579,8 @@ _V02_ADMITTED = [
("status/stale_after", "status: active\nstale_after: 2026-12-01\n"),
("verified bool", "verified: true\n"),
("block sequence of scalars", "tags:\n - alpha\n - beta\n"),
# SPEC §4.1's own skeleton for `tags`, admitted as of 1.4.0 (P1).
("flow sequence of scalars", "tags: [alpha, beta]\n"),
]
@ -726,7 +735,9 @@ def test_the_block_list_rejects_on_the_key_set_not_on_arity():
# G30 fix, not a weakening of this property. Their replacements are the same
# carriers with a key off the allowlist, so the row still measures what it
# says: the shape is admitted, the key set is not.
"generated: { by: x, tool: y }\n", "sources: [{ id: a, uri: u }]\n", "tags: [a, b]\n",
# The `tags` row moved the same way in 1.4.0: `[a, b]` now PARSES (P1), so
# the row carries the quoted form, which is still a T2 rejection.
"generated: { by: x, tool: y }\n", "sources: [{ id: a, uri: u }]\n", "tags: ['a', 'b']\n",
"generated:\n by: x\n", "generated.by: x\n",
"sources:\n - id: a\n uri: file://x\n",
"executor: [{ resource: skills/run.md }]\n",
@ -1054,21 +1065,142 @@ def test_a_block_list_may_not_mix_scalars_and_mappings(cid, fm):
@pytest.mark.parametrize("cid,fm", [
("flow sequence of scalars", "tags: [a, b, c]\n"),
("empty flow sequence", "sources: []\n"),
("flow sequence, unclosed", "sources: [{ id: a }\n"),
("flow sequence, trailing junk", "sources: [{ id: a }] x\n"),
("flow sequence, mixed", "sources: [{ id: a }, plain]\n"),
("flow sequence, nested sequence", "sources: [[ id ]]\n"),
])
def test_the_flow_sequence_admits_flow_mappings_only(cid, fm):
# The sequence carrier is opened for the mapping form and nothing else. A
# flow sequence of plain scalars stays refused -- it is a separate shape with
# its own quoting and splitting problems, and no measured consumer needs it.
def test_the_flow_sequence_refuses_a_carrier_it_cannot_read(cid, fm):
# What no element shape rescues: nothing at all, an unclosed mapping, junk
# between elements, and a sequence inside a sequence. The plain-scalar
# element admitted in 1.4.0 (P1) is a THIRD shape, not a loosening of these
# -- each row here still raises through it.
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
# --- P1: the flow sequence of plain scalars (operator decision, 2026-09-08) --
# `tags: [a, b, c]` is the ONE form SPEC.md §4.1's own skeleton writes out for
# `tags`, and the only candidate measured to move the upstream corpus at all:
# 0/53 -> 6/53 against the pinned reference bundles (`_okf-upstream` @ 3fcbb9f,
# denominator 53). The two neighbours measured 0/53 EACH and are deliberately
# NOT built -- P2 (a flush-left block sequence) and P3 (a folded plain scalar)
# would cost parser surface and buy no measured conformance on this corpus;
# see docs/2026-09-07-limitations-44-maaling.md §§7-8.
#
# It does not spend the depth-1 rule: the elements are leaves, the same depth
# the block list of scalars already carries. The binding constraint on the
# corpus is elsewhere and stays refused -- 44/53 stop on `generated` written as
# a top-level block mapping, which is a security decision, not this one.
@pytest.mark.parametrize("cid,fm,expected", [
("SPEC §4.1 skeleton", "tags: [finance, revenue, headline-metric]\n",
["finance", "revenue", "headline-metric"]),
("two elements", "tags: [a, b]\n", ["a", "b"]),
("one element", "tags: [solo]\n", ["solo"]),
("uneven spacing", "tags: [ a ,b ]\n", ["a", "b"]),
("one trailing comma, as the flow mapping already allows", "tags: [a, b,]\n", ["a", "b"]),
])
def test_a_flow_sequence_of_plain_scalars_parses(cid, fm, expected):
# The red test for P1: `[a, b]` must come back as a real list of strings,
# never a degraded string (the 1.1.0 defect) and never a refusal.
assert parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")[0]["tags"] == expected
def test_the_two_scalar_sequence_carriers_parse_to_the_same_value():
flow = parse_frontmatter("---\ntype: t\ntags: [a, b]\n---\n\nb\n")[0]
block = parse_frontmatter("---\ntype: t\ntags:\n - a\n - b\n---\n\nb\n")[0]
assert flow["tags"] == block["tags"] == ["a", "b"]
@pytest.mark.parametrize("cid,fm", [
# Quoting is the failure mode this shape is refused for elsewhere: the
# parser retains quotes rather than stripping them, so admitting a quoted
# element would hand back a value YAML reads differently.
("single-quoted element", "tags: ['a']\n"),
("double-quoted element", 'tags: ["a", "b"]\n'),
# Comma splitting is character-level, so anything that would need YAML
# semantics to split correctly is refused rather than guessed at.
("colon inside an element", "tags: [a: b]\n"),
("comment indicator", "tags: [a #b]\n"),
("brace inside an element", "tags: [a{b}]\n"),
("bracket inside an element", "tags: [a[b]]\n"),
# Depth: a sequence inside a sequence opens a second level and is refused
# on the same character rule, with no YAML semantics involved.
("nested flow sequence", "tags: [[a]]\n"),
("empty element", "tags: [a, , b]\n"),
("leading empty element", "tags: [, a]\n"),
("empty sequence", "tags: []\n"),
("unterminated sequence", "tags: [a, b\n"),
# The unchanged scalar indicators still apply to an element, exactly as they
# do to a block-list item: an anchor or an alias is not a plain scalar.
("anchor element", "tags: [&anchor]\n"),
("alias element", "tags: [*alias]\n"),
("explicit tag element", "tags: [!!python/object]\n"),
])
def test_a_flow_sequence_scalar_element_must_be_a_plain_scalar(cid, fm):
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
@pytest.mark.parametrize("cid,fm", [
("scalar then mapping", "tags: [a, {b: c}]\n"),
("mapping then scalar", "sources: [{ id: a }, plain]\n"),
])
def test_a_flow_sequence_may_not_mix_scalars_and_mappings(cid, fm):
# Same rule, same reason as the block list: a consumer iterating the value
# and reading `entry.get("id")` crashes on the first str. The mixing verdict
# is reached BEFORE the element is parsed, so it is what the caller sees --
# not an allowlist complaint about a key that was never the problem.
with pytest.raises(OKFFrontmatterError) as exc:
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
assert "mix" in str(exc.value)
@pytest.mark.parametrize("cid,fm,expected", [
("one entry", "sources: [{ id: x }]\n", [{"id": "x"}]),
("two entries", "sources: [{ id: a, title: A }, { id: b, title: B }]\n",
[{"id": "a", "title": "A"}, {"id": "b", "title": "B"}]),
])
def test_the_g30_flow_mapping_carrier_is_unchanged_by_the_scalar_element(cid, fm, expected):
# The no-regression pin. P1 adds a branch to the same function that carries
# `sources`; the mapping element must parse exactly as it did in 1.3.0.
assert parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")[0]["sources"] == expected
# The self-safety row for P1 (OWASP LLM10). The predicate is character-level
# with no regex, so there is no backtracking engine to blow up -- but "no regex"
# is an argument, and the sweep in docs/redos-sweep.py cannot check it, because
# it collects compiled patterns and this predicate compiles none. So it is
# measured instead, on the same CPU clock every other bound here uses. Over four
# doublings (50k -> 800k, 2026-09-08) the exponent is 0.86-0.99 in the element
# LENGTH and 0.97-1.05 in the element COUNT: linear in both, 0.66s at 800_000
# elements and 0.03s at an 800_000-character element. The refusal path is the
# same shape -- a forbidden character at the very end of an 800_000-character
# element is found in 0.04s -- because the element is scanned once per rule, not
# rescanned per start position.
_FLOW_SCALAR_PERF_N = 400_000
def test_a_long_flow_sequence_of_scalars_stays_bounded():
many = "---\ntype: t\ntags: [" + ", ".join(["ab"] * _FLOW_SCALAR_PERF_N) + "]\n---\nbody\n"
assert scan_seconds(parse_frontmatter, many) < 2.0
one_long = "---\ntype: t\ntags: [" + "a" * _FLOW_SCALAR_PERF_N + "]\n---\nbody\n"
assert scan_seconds(parse_frontmatter, one_long) < 2.0
def test_a_long_flow_sequence_element_is_refused_without_a_rescan():
# The refusal is the half an attacker controls: a forbidden character parked
# at the END of a long element is the worst case for any per-start rescan.
doc = "---\ntype: t\ntags: [" + "a" * _FLOW_SCALAR_PERF_N + ":]\n---\nbody\n"
def refuse(payload):
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(payload)
assert scan_seconds(refuse, doc) < 2.0
def test_injection_in_a_sources_entry_leaf_is_caught_by_the_scan():
# T1 over the new shape: every leaf of every entry reaches scan_output.
doc = f"---\ntype: table\nsources: [{{ id: a, title: {_INJECTION} }}]\n---\nclean\n"