1
0
Fork 0

fix(okf): a flow sequence admits '#' and ':' where YAML reads them as text

1.4.0 refused a flow-sequence scalar element carrying '#' or ':' anywhere.
okf writes `references: [...]` as a flow sequence of doc links, and an
entry may carry a #fragment or a scheme://. claude-code-llm-wiki measured
parse_frontmatter raising on 2 038 of 5 467 concepts of a bundle okf
produced (2026-09-17); reproduced here on the same bundle, 2 038 / 5 467
before and 0 / 5 467 after (0 / 4 466 and 0 / 5 530 on the two other
builds).

The two characters are now refused by position, where PyYAML 6.0.3 gives
them a meaning: '#' opening the element or following whitespace (a
comment), ':' opening or ending the element or preceding whitespace (a
mapping). Elsewhere they are scalar characters. The rest of the refused
set, the indicator rule and the no-mixing rule are unchanged.

Chosen over admitting the characters wholesale because `[a: b]` is a
mapping and `[a #b]` a comment to YAML; admitting those would hand the
consumer a value YAML does not read. The position checks are substring
tests, no regex: CPU time stays linear in element count and length
(100k -> 800k), about 10 % over 1.4.0, inside the existing 2 s bound.

Pinned upstream corpus unmoved at 6 / 53. 910 passed (+17), coverage
exit 0, redos-sweep exit 0, 45 LIMITATIONS entries.
This commit is contained in:
Kjell Tore Guttormsen 2026-09-23 12:44:09 +02:00
commit 639da03f8c
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
4 changed files with 119 additions and 6 deletions

View file

@ -81,7 +81,15 @@ _QUOTE_STARTS = frozenset("\"'")
# two splitters, the two collection openers and their closers, and the comment
# indicator. Refusing them is what lets the element be split on commas at the
# character level without a YAML quote state machine.
_FLOW_SCALAR_REFUSED = "{}[]:,\"'#"
#
# 1.4.1 - `:` and `#` are refused only where YAML gives them a meaning, not
# anywhere in the element: okf writes `references: [...]` with `#fragment`s and
# `scheme://` links, and refusing those raised on 2 038 of 5 467 concepts of a
# consumer's bundle (measured 2026-09-17, reproduced 2026-09-23). Ground truth,
# PyYAML 6.0.3: `#` opens a comment at the start of an element or after
# whitespace; `:` opens a mapping at the start of an element, at its end, or
# before whitespace. Elsewhere both are ordinary scalar characters.
_FLOW_SCALAR_REFUSED = "{}[],\"'"
# G3 - the one mapping form T2 can express (operator decision, 2026-08-21).
# Every key inside a mapping must be on this allowlist: the form is safe because
@ -937,7 +945,8 @@ def _parse_flow_sequence(value, parent_key=None):
docs/2026-09-07-limitations-44-maaling.md). The quoting and comma-splitting
problem that kept it refused is answered by refusing the characters that
create it rather than by parsing them: an element is a plain scalar only if
it is non-empty and carries none of ``{ } [ ] : , " ' #``, and it then
it is non-empty and carries none of ``{ } [ ] , " '``, carries ``#`` and
``:`` only where YAML reads them as scalar characters (1.4.1), and it then
passes the unchanged scalar indicator rule. Everything needing YAML
semantics to split or unquote correctly still raises.
@ -1019,9 +1028,17 @@ def _flow_sequence_scalar(element, value):
A quote would have to be stripped (this parser retains quotes, so it would
hand back a different value than YAML reads); a comma or a colon would have
to be split on; a bracket or a brace would open a second collection level,
which no carrier here admits; a ``#`` opens a comment. The unchanged
indicator rule then applies to what is left, exactly as it does to a
block-list item, so an anchor or an alias is no more a scalar here.
which no carrier here admits. The unchanged indicator rule then applies to
what is left, exactly as it does to a block-list item, so an anchor or an
alias is no more a scalar here.
``#`` and ``:`` are refused by position, where YAML reads them (1.4.1):
a ``#`` that opens the element or follows whitespace opens a comment, and a
``:`` that opens or ends the element, or precedes whitespace, opens a
mapping. ``a#b``, ``x://y`` and ``https://e.com:8443/a`` are plain scalars
to YAML and to this parser alike. The element has already been cut at the
next comma and stripped, so its end is the only place a ``:`` can meet the
separator.
"""
if not element:
raise OKFFrontmatterError(
@ -1033,5 +1050,15 @@ def _flow_sequence_scalar(element, value):
"a flow-sequence scalar admits plain scalars only, not %r: %r"
% (char, value)
)
if element[0] == "#" or " #" in element or "\t#" in element:
raise OKFFrontmatterError(
"a '#' at the start of a flow-sequence element or after whitespace "
"opens a comment: %r" % (value,)
)
if element[0] == ":" or element[-1] == ":" or ": " in element or ":\t" in element:
raise OKFFrontmatterError(
"a ':' at the start or end of a flow-sequence element, or before "
"whitespace, opens a mapping: %r" % (value,)
)
_reject_dangerous_value(element)
return element