feat(okf): a flow sequence of plain scalars parses, and the corpus number is 6/53
`tags: [a, b, c]` is the form SPEC.md 4.1's own frontmatter skeleton writes out,
and 9/53 upstream reference concepts use it. It raised on the `[` indicator.
It parses now, to the same list its block-sequence sibling already produced.
The predicate is character-level, inside `_parse_flow_sequence`: an element is a
plain scalar only if it is non-empty and carries none of `{ } [ ] : , " ' #`,
and it then passes the unchanged scalar-indicator rule. Everything that would
need YAML semantics to split or unquote still raises - a quoted element (quotes
are retained here, never stripped), a colon or comma inside an element, a
sequence inside a sequence, an empty element, an anchor, an alias. A sequence
may not mix scalars and mappings, the rule the block list already carries, and
the mixing verdict is reached before the element is parsed so the caller is told
about the mix rather than about a key the allowlist would have named instead.
The 1.3.0 `sources` flow-mapping carrier is unchanged and pinned against
regression. Depth 1 is not spent: the elements are leaves.
Measured with the denominator, against the pinned corpus (`_okf-upstream` @
3fcbb9f, 53 non-reserved documents) and the pinned SPEC (`_okf-canonical` @
ad30107). Baseline reproduced first, with a known-positive control, at 0/53.
After: 6/53, all six in acme_retail. It does not close the corpus - 44/53 still
stop on `generated` written as a top-level block mapping, which spends the
no-nesting-past-depth-1 rule and is a security decision, out of scope here.
P1 alone, per the operator decision of 08.09. The two neighbouring predicates
were measured and deliberately not built: a flush-left block sequence and a
folded plain scalar release 0/53 each on their own, and stacked on this one they
still measure 6/53. `_consume_block_list`, the `description` continuation and
the allowlist are untouched.
docs/LIMITATIONS.md's tags/description entry is rewritten against the
measurement: three of its claims were wrong. The parser does have a
sequence-value type (since 1.3.0 - what it lacks is the indentation the corpus
omits); the figure is 6/53, not the 1.2.0-era 4/53; and tags/description are
not the residual that blocks the corpus. README gains the sequence carrier in
the paragraph that already describes the mapping one.
Self-safety: the predicate compiles no regex, so docs/redos-sweep.py cannot see
it. Measured instead on the CPU clock - linear in element length (exponent
0.86-0.99) and in element count (0.97-1.05) over four doublings to 800_000 -
and pinned by two bounds in tests/test_okf.py.
Six rows that pinned the old refusal are re-aimed at the class that still
holds - the quoted element - the way the 1.3.0 rows were when the carrier
opened. One of them lives in src/llm_ingestion_guard/coverage.py, which is why
the src diff is three files rather than one.
Version 1.4.0 in the code only. The CHANGELOG entry stays under Unreleased and
no tag is cut: README's badge and install pin must keep naming a tag that
exists.
Gates after `git add`: 893 passed (was 868), coverage 130/130 + 6/6 gaps,
redos-sweep exit 0, LIMITATIONS still 45 entries.
This commit is contained in:
parent
6e7c8d2b98
commit
3e324a1f86
9 changed files with 341 additions and 59 deletions
|
|
@ -63,7 +63,7 @@ from .grounding import (
|
|||
)
|
||||
from . import okf
|
||||
|
||||
__version__ = "1.3.0"
|
||||
__version__ = "1.4.0"
|
||||
|
||||
|
||||
# --- §6 bookends: the two library-side halves around the transform ---------
|
||||
|
|
|
|||
|
|
@ -542,8 +542,14 @@ def _build_cases() -> list[Case]:
|
|||
lambda: okf.parse_frontmatter("---\nkey:\n nested: x\n---\nbody\n"), owasp="LLM10"),
|
||||
_raise_case("okf", "T2 frontmatter block scalar", "OKFFrontmatterError",
|
||||
lambda: okf.parse_frontmatter("---\ndesc: |\n block\n---\nbody\n"), owasp="LLM10"),
|
||||
_raise_case("okf", "T2 frontmatter flow sequence", "OKFFrontmatterError",
|
||||
lambda: okf.parse_frontmatter("---\ntags: [a, b]\n---\nbody\n"), owasp="LLM10"),
|
||||
# The plain-scalar flow sequence is ADMITTED as of 1.4.0 (P1, SPEC §4.1's
|
||||
# own skeleton for `tags`); the class this row measures is the element
|
||||
# shape the parser would have to interpret rather than read -- here a
|
||||
# quoted one, which it would have to strip a quote from to return.
|
||||
_raise_case("okf", "T2 frontmatter flow sequence, quoted element",
|
||||
"OKFFrontmatterError",
|
||||
lambda: okf.parse_frontmatter("---\ntags: ['a', 'b']\n---\nbody\n"),
|
||||
owasp="LLM10"),
|
||||
_raise_case("okf", "T2 mapping key off the allowlist", "OKFFrontmatterError",
|
||||
lambda: okf.parse_frontmatter(
|
||||
"---\ngenerated: { by: a, tool: shell }\n---\nbody\n"), owasp="LLM10"),
|
||||
|
|
|
|||
|
|
@ -75,6 +75,14 @@ _DANGEROUS_VALUE_STARTS = frozenset("&*!|>[]{}%@`")
|
|||
# stripped — a pre-existing divergence, pinned in tests/test_okf.py.
|
||||
_QUOTE_STARTS = frozenset("\"'")
|
||||
|
||||
# P1 - what disqualifies a flow-sequence element from being a plain scalar
|
||||
# (operator decision, 2026-09-08). Each character is one this parser would have
|
||||
# to interpret rather than read: the two quotes (retained, never stripped), the
|
||||
# two splitters, the two collection openers and their closers, and the comment
|
||||
# indicator. Refusing them is what lets the element be split on commas at the
|
||||
# character level without a YAML quote state machine.
|
||||
_FLOW_SCALAR_REFUSED = "{}[]:,\"'#"
|
||||
|
||||
# G3 - the one mapping form T2 can express (operator decision, 2026-08-21).
|
||||
# Every key inside a mapping must be on this allowlist: the form is safe because
|
||||
# the allowlist inspects each key, not because mappings became trusted. The keys
|
||||
|
|
@ -922,16 +930,29 @@ def _parse_flow_sequence(value, parent_key=None):
|
|||
(measured 02.09 against llm-ingestion-okf's golden bundle, where a
|
||||
one-element sequence raised on the ``[`` just as a two-element one did).
|
||||
|
||||
A flow sequence of plain *scalars* (``tags: [a, b, c]``) stays refused. It
|
||||
is a different shape with its own quoting and comma-splitting problem, whose
|
||||
failure mode would be accepting something YAML reads differently - and the
|
||||
block-sequence carrier already covers it for every consumer measured so far.
|
||||
As of 1.4.0 it also carries a sequence of plain *scalars*
|
||||
(``tags: [a, b, c]``) - SPEC.md §4.1's own skeleton for ``tags``, and the
|
||||
one candidate form measured to move the upstream corpus at all (0/53 ->
|
||||
6/53 against ``_okf-upstream`` @ 3fcbb9f, denominator 53; see
|
||||
docs/2026-09-07-limitations-44-maaling.md). The quoting and comma-splitting
|
||||
problem that kept it refused is answered by refusing the characters that
|
||||
create it rather than by parsing them: an element is a plain scalar only if
|
||||
it is non-empty and carries none of ``{ } [ ] : , " ' #``, and it then
|
||||
passes the unchanged scalar indicator rule. Everything needing YAML
|
||||
semantics to split or unquote correctly still raises.
|
||||
|
||||
Elements are split on ``}`` rather than on commas, which is sound precisely
|
||||
because ``_parse_flow_mapping`` admits no nested collection: a ``}`` inside
|
||||
an element cannot occur, so the first ``}`` after ``{`` always closes it.
|
||||
Anything between elements that is not a separating comma is refused, which
|
||||
is what makes trailing junk and a mixed sequence fail rather than parse.
|
||||
A mapping element is split on ``}`` rather than on commas, which is sound
|
||||
precisely because ``_parse_flow_mapping`` admits no nested collection: a
|
||||
``}`` inside an element cannot occur, so the first ``}`` after ``{`` always
|
||||
closes it. A scalar element runs to the next comma, which cannot occur
|
||||
inside one. Anything between elements that is not a separating comma is
|
||||
refused, which is what makes trailing junk fail rather than parse.
|
||||
|
||||
A sequence may not mix the two, for the reason the block list may not: a
|
||||
consumer iterating the value and reading ``entry.get("id")`` gets an
|
||||
``AttributeError`` off the first ``str``. The mixing verdict is reached
|
||||
before the element is parsed, so the caller is told about the mix rather
|
||||
than about a key the allowlist would have complained of instead.
|
||||
"""
|
||||
if not value or value[0] != "[":
|
||||
return None
|
||||
|
|
@ -945,6 +966,7 @@ def _parse_flow_sequence(value, parent_key=None):
|
|||
raise OKFFrontmatterError("an empty flow sequence carries nothing: %r" % (value,))
|
||||
|
||||
items = []
|
||||
kinds = set()
|
||||
i = 0
|
||||
n = len(inner)
|
||||
while True:
|
||||
|
|
@ -952,17 +974,22 @@ def _parse_flow_sequence(value, parent_key=None):
|
|||
i += 1
|
||||
if i >= n:
|
||||
break
|
||||
if inner[i] != "{":
|
||||
raise OKFFrontmatterError(
|
||||
"a flow sequence admits flow mappings only: %r" % (value,)
|
||||
)
|
||||
close = inner.find("}", i)
|
||||
if close == -1:
|
||||
raise OKFFrontmatterError(
|
||||
"an unclosed flow mapping inside a flow sequence: %r" % (value,)
|
||||
)
|
||||
items.append(_parse_flow_mapping(inner[i:close + 1], parent_key))
|
||||
i = close + 1
|
||||
if inner[i] == "{":
|
||||
_refuse_mixed_flow_sequence(kinds, "mapping", value)
|
||||
close = inner.find("}", i)
|
||||
if close == -1:
|
||||
raise OKFFrontmatterError(
|
||||
"an unclosed flow mapping inside a flow sequence: %r" % (value,)
|
||||
)
|
||||
items.append(_parse_flow_mapping(inner[i:close + 1], parent_key))
|
||||
i = close + 1
|
||||
else:
|
||||
_refuse_mixed_flow_sequence(kinds, "scalar", value)
|
||||
end = i
|
||||
while end < n and inner[end] != ",":
|
||||
end += 1
|
||||
items.append(_flow_sequence_scalar(inner[i:end].strip(), value))
|
||||
i = end
|
||||
while i < n and inner[i] in " \t":
|
||||
i += 1
|
||||
if i >= n:
|
||||
|
|
@ -973,3 +1000,38 @@ def _parse_flow_sequence(value, parent_key=None):
|
|||
)
|
||||
i += 1
|
||||
return items
|
||||
|
||||
|
||||
def _refuse_mixed_flow_sequence(kinds, kind, value):
|
||||
kinds.add(kind)
|
||||
if len(kinds) > 1:
|
||||
raise OKFFrontmatterError(
|
||||
"a flow sequence may not mix scalar items and mappings: %r" % (value,)
|
||||
)
|
||||
|
||||
|
||||
def _flow_sequence_scalar(element, value):
|
||||
"""Read one flow-sequence element as a plain scalar, or refuse it (P1).
|
||||
|
||||
Character-level, with no YAML semantics: the element must be non-empty and
|
||||
carry none of :data:`_FLOW_SCALAR_REFUSED`. That set is not a style rule -
|
||||
each member is a character whose meaning this parser would have to guess at.
|
||||
A quote would have to be stripped (this parser retains quotes, so it would
|
||||
hand back a different value than YAML reads); a comma or a colon would have
|
||||
to be split on; a bracket or a brace would open a second collection level,
|
||||
which no carrier here admits; a ``#`` opens a comment. The unchanged
|
||||
indicator rule then applies to what is left, exactly as it does to a
|
||||
block-list item, so an anchor or an alias is no more a scalar here.
|
||||
"""
|
||||
if not element:
|
||||
raise OKFFrontmatterError(
|
||||
"an empty element in a flow sequence carries nothing: %r" % (value,)
|
||||
)
|
||||
for char in _FLOW_SCALAR_REFUSED:
|
||||
if char in element:
|
||||
raise OKFFrontmatterError(
|
||||
"a flow-sequence scalar admits plain scalars only, not %r: %r"
|
||||
% (char, value)
|
||||
)
|
||||
_reject_dangerous_value(element)
|
||||
return element
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue