open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Releases 5 Activity

fix(okf): a flow sequence admits '#' and ':' where YAML reads them as text

Browse source 
1.4.0 refused a flow-sequence scalar element carrying '#' or ':' anywhere.
okf writes `references: [...]` as a flow sequence of doc links, and an
entry may carry a #fragment or a scheme://. claude-code-llm-wiki measured
parse_frontmatter raising on 2 038 of 5 467 concepts of a bundle okf
produced (2026-09-17); reproduced here on the same bundle, 2 038 / 5 467
before and 0 / 5 467 after (0 / 4 466 and 0 / 5 530 on the two other
builds).

The two characters are now refused by position, where PyYAML 6.0.3 gives
them a meaning: '#' opening the element or following whitespace (a
comment), ':' opening or ending the element or preceding whitespace (a
mapping). Elsewhere they are scalar characters. The rest of the refused
set, the indicator rule and the no-mixing rule are unchanged.

Chosen over admitting the characters wholesale because `[a: b]` is a
mapping and `[a #b]` a comment to YAML; admitting those would hand the
consumer a value YAML does not read. The position checks are substring
tests, no regex: CPU time stays linear in element count and length
(100k -> 800k), about 10 % over 1.4.0, inside the existing 2 s bound.

Pinned upstream corpus unmoved at 6 / 53. 910 passed (+17), coverage
exit 0, redos-sweep exit 0, 45 LIMITATIONS entries.
This commit is contained in:
Kjell Tore Guttormsen 2026-09-23 12:44:09 +02:00
commit 639da03f8c
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
4 changed files with 119 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

46
tests/test_okf.py
View file

@ -1143,6 +1143,52 @@ def test_a_flow_sequence_scalar_element_must_be_a_plain_scalar(cid, fm):
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
# --- 1.4.1: `#` and `:` are refused where YAML gives them a meaning, not anywhere
# 1.4.0 refused both characters anywhere in an element. A consumer measured what
# that costs (claude-code-llm-wiki, 2026-09-17, reproduced here 2026-09-23 on
# the same bundle): okf writes `references: [...]` as a flow sequence of doc
# links, an entry may carry a `#fragment` or a `scheme://`, and 2 038 of 5 467
# concepts raised on it. YAML itself reads those as plain scalars. The boundary
# is where YAML puts it, ground-truthed against PyYAML 6.0.3 on each row below:
# `#` opens a comment only at the start of an element or after whitespace, and
# `:` opens a mapping only at the start of an element, at its end, or before
# whitespace. Everywhere else both are ordinary characters of the scalar.
@pytest.mark.parametrize("cid,fm,expected", [
("okf's own references line",
"references: [/docs/en/claude-platform-on-aws, /docs/en/amazon-bedrock#3-configure-claude-code]\n",
["/docs/en/claude-platform-on-aws", "/docs/en/amazon-bedrock#3-configure-claude-code"]),
("fragment mid-element", "tags: [a#b]\n", ["a#b"]),
("fragment at the end", "tags: [a#]\n", ["a#"]),
("custom URL scheme", "references: [claude-cli://open]\n", ["claude-cli://open"]),
("scheme then a second element", "references: [vscode://ext/x, y]\n", ["vscode://ext/x", "y"]),
("host and port", "references: [https://e.com:8443/a]\n", ["https://e.com:8443/a"]),
("colon between two characters", "tags: [a:b]\n", ["a:b"]),
("space before the colon, none after", "tags: [a :b]\n", ["a :b"]),
("scheme and fragment together", "references: [x://y#z]\n", ["x://y#z"]),
])
def test_a_flow_sequence_scalar_admits_hash_and_colon_where_yaml_does(cid, fm, expected):
key = fm.split(":", 1)[0]
assert parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")[0][key] == expected
@pytest.mark.parametrize("cid,fm", [
# Each of these is a ParserError or a mapping in PyYAML 6.0.3, so each stays
# refused: admitting it would hand back a value YAML does not read.
("hash opening an element", "tags: [#b]\n"),
("hash opening a second element", "tags: [a, #b]\n"),
("hash after a space", "tags: [a # b, c]\n"),
("hash after a tab", "tags: [a\t#b]\n"),
("colon ending an element", "tags: [a:]\n"),
("colon before the separator", "tags: [a:, b]\n"),
("colon opening an element", "tags: [:a]\n"),
("colon before a tab", "tags: [a:\tb]\n"),
])
def test_a_flow_sequence_scalar_still_refuses_hash_and_colon_where_yaml_reads_them(cid, fm):
with pytest.raises(OKFFrontmatterError):
parse_frontmatter(f"---\ntype: table\n{fm}---\n\nbody\n")
@pytest.mark.parametrize("cid,fm", [
("scalar then mapping", "tags: [a, {b: c}]\n"),
("mapping then scalar", "sources: [{ id: a }, plain]\n"),