Compare commits
10 commits
| Author | SHA1 | Date | |
|---|---|---|---|
| c362717818 | |||
| e56812eb39 | |||
| f0a511369d | |||
| c75c546614 | |||
| c23aea9062 | |||
| 757570dd49 | |||
| 4356caa689 | |||
| 4a6f6ffc16 | |||
| 2d9ee9c434 | |||
| 27b31701e0 |
8 changed files with 684 additions and 15 deletions
294
CHANGELOG.md
294
CHANGELOG.md
|
|
@ -9,6 +9,300 @@ Versioning note: the repository tag versions **the contract** (file set, key nam
|
|||
case ids, disposition semantics). Each JSON file additionally carries its own
|
||||
`"version"` field, bumped when that file changes.
|
||||
|
||||
## [0.9.0] — 2026-08-13
|
||||
|
||||
**A normative rule stated its own premise and then applied itself beyond it.**
|
||||
`spec/conformance-corpus.md` §7 justified the fixture-is-ground-truth ordering with *"**Two
|
||||
implementations** that return different verdicts…"* and then stated the rule with no scope at
|
||||
all. For `signatures/active-content.json` there is no second implementation — the seed runtime
|
||||
authored both the payloads and the table — and that runtime has stated that the classification
|
||||
behind it is calibration it does not freeze. §7 as written made a reserved change on their side
|
||||
into a bug on their side.
|
||||
|
||||
**Breaking in category, minor in number.** This changes disposition semantics, which the
|
||||
versioning note at the top of this file counts as contract. The repository is in 0.x, where a
|
||||
breaking change is a minor bump by the rules — the same reading `[0.3.0]` recorded: *read the
|
||||
entry, not the version number*.
|
||||
|
||||
### Changed
|
||||
|
||||
- **`spec/conformance-corpus.md` — new §7.1, *Where the second paragraph does not hold*.** The
|
||||
scope is keyed on a **structural property**, never on a table name: a case whose scope is a
|
||||
table only one runtime implements, whose payload that runtime authored. A rule naming
|
||||
`active-content` would rot the day a second runtime implements it. §7's own second paragraph
|
||||
already carried the premise; §7.1 makes it explicit and states the disposition for the case
|
||||
the premise excludes — the fixture is not rewritten on the divergence alone, the divergence is
|
||||
recorded against the version pinned, and re-pinning is a separate release. That is the
|
||||
disposition §5 already applies to a stale `observed_out_of_scope` entry, extended to the one
|
||||
place where it can reach a verdict.
|
||||
|
||||
**It creates no fourth verdict, and that constraint shaped the wording.**
|
||||
`schema/conformance-declaration.schema.json` closes `result` with `additionalProperties: false`
|
||||
over four counts plus two arithmetic invariants; a fifth verdict would have broken every
|
||||
consumer's parser, which is a worse break than the one intended. A case whose expected findings
|
||||
are not produced still **fails** and is still named in `failed_cases`. What §7.1 changes is what
|
||||
the failure licenses concluding, not what is reported.
|
||||
|
||||
Two limits stated in the section rather than left to be inferred: it does **not** reach a
|
||||
third-party implementer of the same table — against them the fixture is the contract, exactly
|
||||
as §7 says, and that is the only thing these cases can prove while one runtime is all there is
|
||||
— and it is **not** a licence for a runtime to self-declare, since the exemption is carried by
|
||||
the corpus's provenance record for the scope and not asserted per case by whoever failed.
|
||||
|
||||
Superseded text is named rather than edited away, following §6's own pattern: *"Through corpus
|
||||
version 0.8.1 this section stated the rule above with no scope at all."*
|
||||
|
||||
**The competing reading was tested and disposed of**, because it is the one that would have
|
||||
avoided this release: that §7's existing hatch (*"unless the fixture itself is proven wrong"*)
|
||||
already covered it. It does not. The hatch's consequence is that **the fixture changes**, and
|
||||
the manifest field asserts the opposite — pinned, not rewritten, re-pinning a separate
|
||||
decision. And a runtime recalibrating does not prove the earlier classification wrong: the
|
||||
fixture measured `de09711` / `0.4.0` correctly, and a later release does not reach back and
|
||||
falsify an earlier measurement. The case fits neither of §7's two dispositions, which is the
|
||||
defect.
|
||||
|
||||
- **`conformance/manifest.json` `0.6.1` → `0.6.2` —
|
||||
`active_content_provenance.pins_a_version_not_a_frozen_classification` no longer records an
|
||||
open question.** The retirement is **partial and it is quoted, not dropped**, per the house
|
||||
style this field established one release ago (*"a correction that does not say what it corrects
|
||||
cannot be audited"*). What falls is only the open-question status; the clause *"section 7 …
|
||||
is NOT amended by this block"* **stays true and is kept**, because §7 was amended by its own
|
||||
release and not by a data file. Value change only — read back from disk against `HEAD` with a
|
||||
flattened key diff: `added: 0, removed: 0, changed: 2` (the field and `version`), and the new
|
||||
string printed and read rather than inferred from the count, since a value edit reports
|
||||
`changed: 1` whatever it wrote.
|
||||
|
||||
Six prose dashes in the new text were written `--` and promoted to `—` before commit: `--` is
|
||||
the variant-suffix separator token of §6's case-id grammar, and every other occurrence of it in
|
||||
this file is that token, a real case id, or a CLI flag.
|
||||
|
||||
### Neighbours — measured, and the ones left alone are named
|
||||
|
||||
A sweep for the retired premise was run over the whole repository, widened past *"ground truth"*
|
||||
to the second paragraph's own wording (*"one of them has a bug"*, *"two implementations"*), since
|
||||
a restatement in that phrasing would have survived the first search.
|
||||
|
||||
- **`CONVENTIONS.md` — changed.** Carried the rule unscoped and called the proven-wrong hatch
|
||||
*"the one way that reverses"*. There are now two, and both are listed.
|
||||
- **`CLAUDE.md` — changed.** The Norwegian restatement that governs sessions in this repository
|
||||
carried the same unscoped rule; left alone, the next session here would have acted on it.
|
||||
- **`SECURITY.md` §2 — minimal cross-reference only.** Its claim is about a fixture that expects
|
||||
**too little**, and §7.1 narrows *who the rule reaches*, not that direction. The conclusion
|
||||
survives intact, so it was not rewritten.
|
||||
- **`SECURITY.md` "Why a confirmed defect is usually not fixed here first" — untouched.** Its
|
||||
*"two implementations answering differently"* is about extracted **data** diverging from its
|
||||
source, not about fixtures.
|
||||
- **`README.md` — untouched.** Its conformance row says *"Ground truth"* as a descriptor and does
|
||||
not restate the disagreement rule, and it already names the asymmetry it would otherwise hide:
|
||||
the seven active-content cases are *"measured against the one runtime that implements that
|
||||
table"*. Nothing there became false.
|
||||
- **`docs/extraction-plan.md` — untouched, and it is supporting evidence rather than a stale
|
||||
neighbour.** It already records that the calibration file *"inverts this repository's central
|
||||
rule"* — so this is the second place the unscoped rule was known not to hold, and the first was
|
||||
documented before this release.
|
||||
|
||||
### Not in this release
|
||||
|
||||
Whether the seven active-content cases still pass at the seed runtime's `v1.1.0` is **unmeasured**,
|
||||
and §7.1 is silent on it. No case was minted, no data file touched, no id string proposed.
|
||||
|
||||
## [0.8.1] — 2026-08-13
|
||||
|
||||
**The field 0.8.0 added to make the exposure precise stated it with a hand-derived count, and the
|
||||
count was wrong.** Caught in the same session, before any consumer read it, and corrected inside
|
||||
the field rather than by rewriting it. No measurement changed and no verdict moved.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`conformance/manifest.json` 0.6.0 → 0.6.1 —
|
||||
`active_content_provenance.pins_a_version_not_a_frozen_classification` now enumerates instead of
|
||||
totalling.** As published it read *"three of the four dimensions they name as calibration cannot
|
||||
move one of these cases at all. The fourth can: which `active:` ids a payload yields IS the
|
||||
classification"*. Two defects in one sentence. First, the total was derived by hand over a
|
||||
taxonomy the field had itself recategorized: the seed runtime's four calibration dimensions are
|
||||
severities, thresholds, **lexicon entries** and dispositions, and lexicon entries are *not* absent
|
||||
from these fixtures — `active__data-uri` carries `data-uri:executable` and `active__raw-html`
|
||||
carries `hybrid-xss:script-tag` in `observed_out_of_scope`, both verified as members of
|
||||
`lexicon/injection-lexicon.json` and non-members of `signatures/active-content.json`. Second,
|
||||
*"the fourth"* silently substituted the classification for lexicon entries as the fourth item of
|
||||
their sentence, which it is not — the classification is what they addressed separately.
|
||||
|
||||
- The replacement names three things and totals none of them: severities/thresholds/dispositions
|
||||
are absent and move no verdict; lexicon entries move no verdict either — spec section 5 forbids
|
||||
failing a runtime over `observed_out_of_scope` — but a lexicon calibration change **ages** those
|
||||
two entries as evidence, which is the exposure
|
||||
`active_content_measurement_0_7_0.movement_sweep.residue_is_the_field_no_test_protects` already
|
||||
names as a class, and this corpus pins a stale residue entry rather than rewriting it; and the
|
||||
active-content classification is the one thing that can move a verdict. The retired sentence is
|
||||
**quoted** in the field's `AMENDED IN 0.6.1` clause, not merely dropped, for the same reason
|
||||
`scope_planned.$comment` quotes what it retired: a correction that does not say what it corrects
|
||||
cannot be audited.
|
||||
|
||||
## [0.8.0] — 2026-08-13
|
||||
|
||||
**The seven active-content fixtures pin a VERSION of the seed runtime, and nothing said so.**
|
||||
That runtime tagged `v1.0.0` on 2026-08-13 and stated that the freeze covers its exported Python
|
||||
surface only, excluding detection behaviour: severities, thresholds, lexicon entries and
|
||||
dispositions are calibration there and move in minor and patch releases. The manifest already
|
||||
pinned commit and version per measurement block, but nowhere recorded that the thing pinned is a
|
||||
version rather than a frozen classification. No case is minted, no data file is touched, no id is
|
||||
proposed.
|
||||
|
||||
### Added
|
||||
|
||||
- **`conformance/manifest.json` 0.5.2 → 0.6.0 —
|
||||
`active_content_provenance.pins_a_version_not_a_frozen_classification`.** One field, scoping the
|
||||
neighbouring `asymmetry` rather than replacing it, and deliberately narrower than the runtime's
|
||||
own statement. The exposure is bounded by what the fixtures assert, which was read from all seven
|
||||
rather than assumed: every finding carries `pattern_id` and nothing else — no severity, no
|
||||
threshold, no disposition — so three of the four dimensions that runtime names as calibration
|
||||
cannot move one of these cases at all. The fourth can, because which `active:` ids a payload
|
||||
yields *is* the classification. The field names both pins (six at 0.4.0 / `de09711`, the seventh
|
||||
at 0.7.0 / `be9759b`) rather than one, since a single version would flatten two measurements into
|
||||
one header — the defect `superseded_for_one_case` exists to prevent. Their v1.0.0 statement is
|
||||
**attributed** to their coord message of 2026-08-13T20:40:31Z, not restated as a fact measured
|
||||
from this side.
|
||||
|
||||
- The field also names the disposition of a future divergence, so it is not left to be inferred: a
|
||||
later 1.x that classifies one of these payloads differently is not a breach by them and does not
|
||||
make the fixture wrong. The fixture stays ground truth at its pinned version, the divergence is
|
||||
measured and recorded, and re-pinning is a separate decision — the same disposition this corpus
|
||||
already applies to a stale `observed_out_of_scope` entry.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`docs/secret-egress-divergence.md:75-76` carried the same misquote `conformance/manifest.json`
|
||||
had corrected in 0.7.3**, named there as a deliberate omission and closed here. The field's value
|
||||
ends ``ascending `order` `` — the backticks are the field's own. The fix is *not* the one the
|
||||
omission note implied: those two lines are a single code span delimited by **single** backticks
|
||||
across a line break, so inserting the field's backticks inside it would have terminated the span
|
||||
at the first one and rendered the quote broken. The outer delimiter is promoted to double
|
||||
backticks instead, which is what lets the inner singles survive. The manifest's correction ported
|
||||
as a literal string because JSON has no backtick semantics; markdown does. Verified by extracting
|
||||
the span from the file on disk, unfolding the line break, and comparing to the decoded value in
|
||||
`signatures/secret-egress.json` — equal — and by confirming no backtick run of length ≥ 2 sits
|
||||
inside the span.
|
||||
|
||||
### Not done, and named rather than left silent
|
||||
|
||||
- **`spec/conformance-corpus.md` section 7 is untouched.** It states the disagreement rule without
|
||||
scope: *"The fixture is ground truth. A runtime that disagrees is wrong."* Read against the
|
||||
active-content scope, whose only implementing runtime has now said in writing that its
|
||||
classification may legitimately move, that rule would call a calibration change there a bug. The
|
||||
manifest field records the interaction and explicitly does not amend the spec. Whether the
|
||||
normative rule needs a scope is a decision for its own release.
|
||||
|
||||
## [0.7.3] — 2026-08-13
|
||||
|
||||
**The README still argued the premise 0.7.2 retired, and the two files sat on a public remote
|
||||
disagreeing.** `README.md` opened the egress gap with "It is not an id question at all"; the
|
||||
blocker it sends the reader to for authority now opens reason (1) with "NO ID SPACE ON THE
|
||||
COMMONS SIDE. This is the hard blocker." Before 0.7.2 the README was merely out of date. After
|
||||
it, our own commit had made it contradictory — the same defect class 0.7.1 existed to close. No
|
||||
data moves, no case is minted, no id is proposed.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **The README now carries the three measured reasons instead of the retired one.** (1) No id
|
||||
space on the commons side — the hard blocker, and the only one an answer can resolve; the
|
||||
answer belongs to the runtimes that own the seeds. (2) Match semantics disagree, and an id
|
||||
space would not close it. (3) Membership diverges in both directions and the divergence is
|
||||
inherited: the two sides hold 19 entries and 25, and they are ports of two *different* source
|
||||
tables in one source repository. The counts survived the falsification; only the causal claim
|
||||
fell, so `different tables` is kept and "cut at different granularities" is gone. The
|
||||
paragraph deliberately does **not** restate the outgoing question's status: that is true on
|
||||
the day it is written, nothing tests README prose, and `conformance/manifest.json` already
|
||||
carries the date. The standing `entry_points_by_scope` requirement is likewise left out rather
|
||||
than printed as a fourth reason.
|
||||
|
||||
- **`conformance/manifest.json` 0.5.1 → 0.5.2: the blocker misquoted the contract it cites.** It
|
||||
rendered the field as `match_semantics: "… evaluated in ascending order"`; the value in
|
||||
`signatures/secret-egress.json` ends ``ascending `order` `` — the backticks are the field's
|
||||
own. A blocker that misquotes the semantics it is blocking on invites a consumer to implement
|
||||
the wrong one. The data file is
|
||||
unchanged and was never wrong — only the quotation of it was, which `scope_planned.$comment`
|
||||
now records. Verified by reading the edited file back from disk and matching the decoded
|
||||
string against the data file that owns it; `json.tool` passes on wrong escaping.
|
||||
|
||||
Known and deliberately left: `docs/secret-egress-divergence.md` renders the same value without
|
||||
its backticks. That document is `Status: informative` and was outside this release's scope.
|
||||
|
||||
## [0.7.2] — 2026-08-13
|
||||
|
||||
**`scope_planned.blockers` named the premise that `docs/secret-egress-divergence.md`
|
||||
falsified.** The blocker read "19 entries … 25 at different cut points" — one table cut at two
|
||||
granularities, waiting on a reconciliation of two ports. Measured 2026-08-13: they are ports of
|
||||
**two different source tables** in the same source repository, so no reconciliation of the ports
|
||||
was ever going to close it. No data moves in this release, and no case is minted — only the
|
||||
recorded reason a case cannot be.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **The egress blocker now carries the three measured reasons, kept independent.** (1) Commons
|
||||
has no id space for this table: seed A (`hooks/scripts/pre-edit-secrets.mjs`) carries a name
|
||||
and a pattern per entry and nothing else, so entries are keyed by human-readable `name` while
|
||||
the guard emits `egress:<id>`, and a fixture names labels. This is the only one of the three
|
||||
an answer can resolve, and it is the outgoing question. (2) Match semantics disagree:
|
||||
`first match wins` with `ordering.normative: true` here, against `finditer` over all 25
|
||||
patterns there — one witness, an `Authorization` header holding a three-part JWT, produces
|
||||
**one** label under commons' declared contract and **two** from the guard. (3) Membership
|
||||
diverges both ways and is inherited from two different seeds (seed A 19 entries, seed B 33,
|
||||
the guard ported 25, 8 unported), so re-measuring either port cannot close it. The blocker
|
||||
points to `docs/secret-egress-divergence.md` for the method behind every number.
|
||||
|
||||
- **Two hand-carried numbers in the retired text are corrected in the same string.**
|
||||
`aws-access-key-id` was called "the one clean one-to-one": measured, only **2 of 19** commons
|
||||
patterns are byte-identical to a guard pattern after unescaping, and AWS is not among them —
|
||||
the guard anchors the same run as `\bAKIA[0-9A-Z]{16}\b`. `GitHub Token` was called "four ids
|
||||
there": measured on witnesses it maps to **three**, and leaves `ghu_` and `ghr_` covered by no
|
||||
guard id. Both were transcription, not measurement. What is retracted is quoted in place; the
|
||||
full retired text stands in git at `conformance/manifest.json` 0.5.0.
|
||||
|
||||
- **`scope_planned.$comment` said "a distinct unresolved question" — singular.** Left alone it
|
||||
would tell a reader the case becomes mintable when an answer arrives, which is true of one
|
||||
reason in three. Amended alongside the blocker rather than after it, since the two are read
|
||||
together.
|
||||
|
||||
### Measured
|
||||
|
||||
- **The standing requirement was measured here, not transcribed from the document.**
|
||||
`entry_points_by_scope.scopes` carries **no entry at all** for `signatures/secret-egress.json`
|
||||
— the three declared scopes are the lexicon, active-content and carriers. Entry point,
|
||||
findings accessor and fixture presentation must be filled for both runtimes before a first
|
||||
egress case, independently of the three reasons. It is recorded as a requirement, not as a
|
||||
fourth reason: it would stand even if all three were resolved tomorrow.
|
||||
|
||||
- **No id string is proposed, in this file or anywhere else.** Checked against the two outgoing
|
||||
coord messages of 2026-08-13 rather than assumed: both state in as many words that no id is
|
||||
being proposed. Naming an id in a shared space is the exception `carrier:*` established, it
|
||||
requires both runtimes asked first, and both are unanswered.
|
||||
|
||||
`conformance/manifest.json` 0.5.0 → 0.5.1. No case directory, no `expected.json` and no
|
||||
signature table changed; `git status` shows one file besides this changelog.
|
||||
|
||||
## [0.7.1] — 2026-08-13
|
||||
|
||||
Two loose ends from `0.7.0`, neither of which changes a contract.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **The README still named the manifest as the authority for the variant rule.** It read
|
||||
"see `case_id_derivation.variant_suffix` in the manifest" — true until `0.7.0`, when the
|
||||
rule became normative in `spec/conformance-corpus.md` §6 and the manifest's block became
|
||||
the *measurement* behind it rather than the contract. Left alone it would have reproduced
|
||||
in one line the same defect `0.7.0` closed: a reader sent to the wrong authority.
|
||||
|
||||
### Measured
|
||||
|
||||
- **The `__` half of the derivation was re-measured too, not just the `--` half.** `0.7.0`
|
||||
made a point of re-measuring `--` rather than copying the manifest's `0.3.0` numbers
|
||||
forward, while the adjacent sentence asserting that `__` "does not occur anywhere in the
|
||||
ratified id space" was inherited untested — a claim about this release's own soundness that
|
||||
the release did not check. Measured now across all five published id spaces: the 83 lexicon
|
||||
ids, the 7 `active:` ids, the 3 `carrier:` ids, the 7 malware rule ids and the 19
|
||||
secret-egress entry names carry **neither** `__` nor `--`. The one-to-one transform holds.
|
||||
No text changed; the sentence was true. It is now true *and* measured.
|
||||
|
||||
## [0.7.0] — 2026-08-13
|
||||
|
||||
**The normative spec forbade, in as many words, a case the corpus has shipped since
|
||||
|
|
|
|||
|
|
@ -65,6 +65,13 @@ Ingen. Data + prosa. Filformater: JSON (data + schema), Markdown (spec), rå tek
|
|||
- `expected.json` er ground truth. Er en runtime uenig med `expected.json`, er runtimen
|
||||
feil — med mindre fixturen selv bevises feil, og da endres fixturen i eget commit med
|
||||
begrunnelse.
|
||||
- **Regelen over er skopet, og skopet er bærende.** Er casens scope en tabell bare ÉN runtime
|
||||
implementerer, og den runtimen skrev payloaden, finnes ikke den andre implementasjonen
|
||||
regelen dømmer mellom. Da er en divergens fra *den* runtimen verken en bevist feil fixture
|
||||
eller nødvendigvis deres bug: fixturen skrives ikke om på divergensen alene, den føres mot
|
||||
versjonen som er pinnet, og re-pinning er en egen release. Mot en TREDJEPARTS-implementasjon
|
||||
av samme tabell gjelder §7 uendret. Til og med `v0.8.1` sto regelen uskopet. Se
|
||||
`spec/conformance-corpus.md` §7.1.
|
||||
- **En case er ikke mintbar uten inngangspunkt for sitt scope.** Korpuset pinner ikke
|
||||
lenger ett inngangspunkt per runtime for alt — `manifest.json` →
|
||||
`entry_points_by_scope` bærer inngangspunkt, **findings-accessor** og
|
||||
|
|
|
|||
|
|
@ -107,10 +107,16 @@ this document:
|
|||
- `<case-id>` is stable and descriptive. **Changing a case id is a breaking change** — a
|
||||
published conformance result names it.
|
||||
- `expected.json` is **ground truth**. If a runtime disagrees with it, the runtime is wrong.
|
||||
- The one way that reverses: the fixture is proven wrong. Then the fixture changes **in its own
|
||||
- One way that reverses: the fixture is proven wrong. Then the fixture changes **in its own
|
||||
commit, with the reason written down** — never folded into a change that does something else,
|
||||
because a fixture edit is the one edit that can make every conforming runtime wrong
|
||||
identically.
|
||||
- The other, added in `v0.9.0`: where a case's scope is a table only one runtime implements and
|
||||
that runtime authored the payload, a divergence by **that** runtime is neither a proven-wrong
|
||||
fixture nor necessarily its bug. The fixture is not rewritten on the divergence alone — it is
|
||||
recorded against the version pinned, and re-pinning is a separate release. Through `v0.8.1`
|
||||
this list carried only the first way. See
|
||||
[`spec/conformance-corpus.md` §7.1](spec/conformance-corpus.md).
|
||||
- A case declares the data files it is `scope`d to. A runtime that does not implement a scoped
|
||||
table reports the case `not-applicable` — a third verdict beside pass and fail, and one that
|
||||
must be reported rather than dropped from the denominator. See
|
||||
|
|
|
|||
38
README.md
38
README.md
|
|
@ -16,6 +16,18 @@ unicode-carrier smuggling or active content in untrusted text, on any runtime.
|
|||
|
||||
**It holds no runnable code.** Data, specifications and fixtures only.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Install](#install)
|
||||
- [Requirements](#requirements)
|
||||
- [What it does](#what-it-does)
|
||||
- [Non-goals](#non-goals)
|
||||
- [Known limitations](#known-limitations)
|
||||
- [Contributing](#contributing)
|
||||
- [Reporting a wrong entry](#reporting-a-wrong-entry)
|
||||
- [Changelog](#changelog)
|
||||
- [License](#license)
|
||||
|
||||
## Install
|
||||
|
||||
Nothing to install — this repository is **vendored into consumers**, not installed.
|
||||
|
|
@ -93,8 +105,9 @@ The corpus covers three tables, and they do not carry equal weight — treating
|
|||
number would misreport all three:
|
||||
|
||||
- `lexicon/injection-lexicon.json` — 84 cases over 83 patterns. Both seeding runtimes
|
||||
implement it and both ratified its id space. One pattern carries a second, variant case;
|
||||
see `case_id_derivation.variant_suffix` in the manifest.
|
||||
implement it and both ratified its id space. One pattern carries a second, variant case:
|
||||
the rule for when that is legal is normative in [§6](spec/conformance-corpus.md), and
|
||||
`case_id_derivation.variant_suffix` in the manifest carries the measurement behind it.
|
||||
- `signatures/active-content.json` — 7 cases, one per published id, the seventh added in
|
||||
v0.6.0 when the seed runtime split raw HTML into two carrier classes. One runtime
|
||||
implements it. For a runtime that
|
||||
|
|
@ -108,11 +121,22 @@ number would misreport all three:
|
|||
missing **name** rather than a missing capability. It lapses the moment that runtime names
|
||||
its label and the alias is added.
|
||||
|
||||
One case remains unshipped, for the secret-egress table, and it is not blocked on effort.
|
||||
It is not an id question at all — the two runtimes carry *different tables*, 19 entries
|
||||
against 25, cut at different granularities, and a shared id space presupposes a reconciliation
|
||||
nobody has performed. `conformance/manifest.json` records that blocker under
|
||||
`scope_planned.blockers`, measured, so the gap is visible rather than inferred.
|
||||
One case remains unshipped, for the secret-egress table, and it is not blocked on effort. The
|
||||
reasons are three, they were measured, and they are independent — none of them dissolves under
|
||||
anything this repository can run alone. **(1) There is no id space on the commons side.** The
|
||||
seed this table was ported from carries a name and a pattern per entry and nothing else, so its
|
||||
entries are keyed by human-readable name while the other runtime emits `egress:<id>` labels —
|
||||
and a fixture names labels. This is the hard blocker, and the only one of the three that an
|
||||
answer can resolve; the answer belongs to the runtimes that own the seeds, not to a name coined
|
||||
here. **(2) Match semantics disagree**, and an id space would not close it: this table declares
|
||||
first-match-wins with `ordering.normative: true`, the other runtime reports every match, and one
|
||||
witness — an `Authorization` header holding a three-part JWT — produces one label here and two
|
||||
there. That difference is exactly what an `expected.json` encodes. **(3) Membership diverges in
|
||||
both directions, and the divergence is inherited rather than introduced.** The two sides hold 19
|
||||
entries and 25, but they are ports of two *different* source tables in one source repository, so
|
||||
re-measuring either port cannot close it. `conformance/manifest.json` records all three under
|
||||
`scope_planned.blockers`, and the method behind every number is in
|
||||
[the divergence measurement](docs/secret-egress-divergence.md).
|
||||
|
||||
The carrier blocker closed in v0.5.0 and is kept, with its retired text, under
|
||||
`scope_planned.blockers_resolved` — including the correction one runtime volunteered against
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@ detector, and that is a working bypass against every consumer until it is closed
|
|||
|
||||
Report privately by email:
|
||||
|
||||
- **hello@fromaitochitta.com**, with `SECURITY` at the start of the subject.
|
||||
- **security@fromaitochitta.com**, with `SECURITY` at the start of the subject.
|
||||
|
||||
Pull requests are not the channel either — they are switched off on the canonical
|
||||
repository, and not as an oversight. This repository is vendored into independent runtimes
|
||||
|
|
@ -45,8 +45,9 @@ In scope — all of these are real reports:
|
|||
escaping is wrong for the declared dialect, missing or wrong flags, a pattern that fails
|
||||
to compile in a documented engine and gets skipped rather than reported.
|
||||
2. **A conformance fixture that sanctions a miss.** `expected.json` is ground truth: a
|
||||
runtime that disagrees with it is deemed wrong. A fixture that expects too little makes
|
||||
every conforming runtime wrong identically, and the corpus will not catch it.
|
||||
runtime that disagrees with it is deemed wrong (as scoped by `spec/conformance-corpus.md`
|
||||
§7.1, which narrows who that reaches and not this direction). A fixture that expects too
|
||||
little makes every conforming runtime wrong identically, and the corpus will not catch it.
|
||||
3. **A normative clause that mandates unsafe behaviour.** The `spec/` files bind the
|
||||
implementations that consume them, so a weak rule propagates to all of them.
|
||||
4. **A real secret or personal data in the repository or its history.** The history is
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
295
docs/secret-egress-divergence.md
Normal file
295
docs/secret-egress-divergence.md
Normal file
|
|
@ -0,0 +1,295 @@
|
|||
# Secret-egress divergence — commons vs the Python guard
|
||||
|
||||
**Status: informative.** Nothing here is normative and nothing here changes a data file. It
|
||||
records a measured disagreement between two tables that were believed to be two cuts of one
|
||||
source, and turns out not to be. Under this repository's behaviour-preservation invariant,
|
||||
a divergence found here is **reported, not fixed**.
|
||||
|
||||
Produced 2026-08-13. Every number below came from a command; the scripts live in the session
|
||||
scratchpad rather than in this repository, because executable code here would breach the
|
||||
charter. They are reproducible from the method column. The guard was read via
|
||||
`git archive v0.7.0`, never from its working copy.
|
||||
|
||||
**The premise this document was opened to test does not survive it.** The open question was
|
||||
recorded as "19 entries here against the guard's 25, cut at different granularity" — one
|
||||
table, two granularities. That is not what the two files are. They are ports of **two
|
||||
different source tables in the same source repository**, and the granularity difference is a
|
||||
consequence of that, not the cause. Everything below follows from correcting that premise.
|
||||
|
||||
## What was compared
|
||||
|
||||
| Side | Artefact | Version / coordinate |
|
||||
| --- | --- | --- |
|
||||
| commons | [`signatures/secret-egress.json`](../signatures/secret-egress.json) | file `version` 0.3.0, 19 entries |
|
||||
| guard | `llm-ingestion-pipeline-security` `src/llm_ingestion_guard/output.py` `_SECRET_PATTERNS` | tag `v0.7.0` = commit `be9759b`, 25 entries |
|
||||
| seed A | `llm-security` `hooks/scripts/pre-edit-secrets.mjs` `SECRET_PATTERNS` | commit `47905da`, 19 entries — what commons ported |
|
||||
| seed B | `llm-security` `knowledge/secrets-patterns.md` | commit `47905da`, blob `a7ed469`, 33 entries — what the guard ported |
|
||||
|
||||
**The guard's v0.7.0 is the guard's current behaviour.** `git diff v0.7.0..aff3511 -- src/`
|
||||
is empty, where `aff3511` was the guard's head when this was measured. Pinning at the tag
|
||||
therefore costs no currency; it is not a waypoint measurement.
|
||||
|
||||
**Seed B was read, not accepted.** The guard's module docstring asserts *"Ported from the
|
||||
`llm-security` `knowledge/secrets-patterns.md` seed"*. That assertion is a claim about a
|
||||
third repository and would be an attribution, not a finding, if it were relayed. It was
|
||||
measured instead: the file exists at the pinned commit on the public remote, and all 25 of
|
||||
the guard's ids appear in it verbatim — `0` guard ids are absent from seed B. The docstring
|
||||
is correct.
|
||||
|
||||
**Both seeds are named in commons' own file.** `signatures/secret-egress.json`'s `$comment`
|
||||
already says which of the two it took and that the other *"is a separate PCRE-flavoured
|
||||
agent-consumed variant that stays where it is"*. What was not known until now is that the
|
||||
guard ported the other one.
|
||||
|
||||
## Result
|
||||
|
||||
| Measure | Method | Result |
|
||||
| --- | --- | --- |
|
||||
| Entry count, both sides | count entries | 19 and 25 |
|
||||
| Seed B entry count | parse the `.md` at the pinned blob | 33 |
|
||||
| Guard ids present in seed B | set membership on `id` | **25/25** |
|
||||
| Seed B ids the guard did not port | set difference | **8** |
|
||||
| Guard vs seed B, field-identical | compare regex (after stripping seed B's `(?i)` inline rendering), flags and severity | **16/25** |
|
||||
| — of the 9 remaining, escaping-only | unescape the guard's `\"` (a Python raw-string artefact) and compare for string identity | **4/4 identical** |
|
||||
| — of the 9 remaining, behaviourally real | differential match comparison | **5** — 4 connection strings, 1 capture-group change |
|
||||
| commons vs guard, byte-identical patterns | unescape both sides' `\/` and `\"`, compare source + flags | **2/19** |
|
||||
| Differential probe corpus | one witness per guard id, plus each side's exclusive shapes and the semantics witness | 36 probes |
|
||||
| Shapes commons reports and the guard is silent on | differential | **7 witnesses, across 5 commons entries** |
|
||||
| Shapes the guard reports and commons is silent on | differential | **3** |
|
||||
| Match-semantics divergence | one witness matching two entries on both sides | **1 label vs 2 labels** |
|
||||
|
||||
16 field-identical + 4 escaping-only + 5 real = 25.
|
||||
|
||||
**The `2/19` is the number that says these are not two cuts of one table.** Only
|
||||
`GitHub Fine-Grained PAT` ↔ `github-pat-fine-grained` and `OpenAI Legacy API Key` ↔
|
||||
`openai-api-key-legacy` are byte-identical after unescaping. Even `AWS Access Key ID` is not:
|
||||
commons has `AKIA[0-9A-Z]{16}` and the guard has the same run anchored, `\bAKIA[0-9A-Z]{16}\b`.
|
||||
The earlier note calling that pair the one clean 1:1 was wrong, and was wrong by transcription
|
||||
rather than by measurement.
|
||||
|
||||
## Match semantics: the divergence that is not about membership
|
||||
|
||||
This is the finding a membership table would hide, and it is the one a consumer implementing
|
||||
from commons will get wrong first.
|
||||
|
||||
`signatures/secret-egress.json` declares ``match_semantics: "first match wins; patterns are
|
||||
evaluated in ascending `order`"``, marks `ordering.normative: true`, and names
|
||||
`last_entry_is_load_bearing: "JWT (three-part token)"` — the JWT entry is placed last
|
||||
precisely so a token inside an `Authorization` header is reported as the header, not as a
|
||||
bare JWT.
|
||||
|
||||
The guard's `scan_secret_egress` runs `finditer` over all 25 patterns and adds a finding for
|
||||
every match. Order carries **no** semantics there, and there is no first-match-wins layer.
|
||||
|
||||
Measured on one witness — an `Authorization` header whose value is a three-part JWT:
|
||||
|
||||
| Side | Finding set |
|
||||
| --- | --- |
|
||||
| commons, under its own declared contract | `Authorization header with token` — one label |
|
||||
| guard, `scan_secret_egress` at `v0.7.0` | `egress:bearer-token`, `egress:jwt-token` — two labels |
|
||||
|
||||
Both detect the credential. They disagree about what a report says, which is what a
|
||||
`conformance/expected.json` encodes. Two runtimes that both "pass" here would still produce
|
||||
different fixture files.
|
||||
|
||||
The commons side of this was not hand-rewritten: the evaluator compiles the patterns out of
|
||||
the JSON, in `order`, applying `re.I` exactly where the file's own `dialect.translation_notes`
|
||||
say to, and stops at the first hit. The guard side is the imported module. Neither table was
|
||||
transcribed.
|
||||
|
||||
## Membership, measured
|
||||
|
||||
Every row below comes from running a witness input through both sides, not from reading the
|
||||
two regexes side by side.
|
||||
|
||||
| commons `order` / name | guard ids observed | Relation |
|
||||
| --- | --- | --- |
|
||||
| 0 `AWS Access Key ID` | `aws-access-key-id` | 1:1, guard anchored |
|
||||
| 1 `AWS Secret Access Key` | — | **guard silent** |
|
||||
| 2 `Azure Connection String (AccountKey/SharedAccessKey/sig)` | `azure-storage-key` | overlap; see below |
|
||||
| 3 `Azure AD ClientSecret` | `azure-client-secret` | 1:1 |
|
||||
| 4 `Azure AI Services Key` | — | **guard silent** |
|
||||
| 5 `GitHub Token` | `github-pat-classic`, `github-oauth-token`, `github-server-token` | 1:3, **plus 2 prefixes neither guard id covers** |
|
||||
| 6 `npm Token` | `npm-token` | 1:1 |
|
||||
| 7 `Anthropic API Key` | `anthropic-api-key` | 1:1 |
|
||||
| 8 `OpenAI Project Key` | `openai-project-key` | 1:1 |
|
||||
| 9 `GitHub Fine-Grained PAT` | `github-pat-fine-grained` | 1:1, **byte-identical** |
|
||||
| 10 `Google API Key` | `gcp-api-key` | 1:1 |
|
||||
| 11 `Private Key PEM Block` | `rsa-private-key`, `ec-private-key`, `pkcs8-private-key` | 1:3, **minus one PEM label** |
|
||||
| 12 `JWT Secret` | — | **guard silent** |
|
||||
| 13 `Slack/Discord Webhook URL` | — | **guard silent** |
|
||||
| 14 `Generic credential assignment` | `generic-api-key`, `config-password`, `config-secret` | 1:3 |
|
||||
| 15 `Authorization header with token` | `bearer-token` (+ `jwt-token`, see semantics) | 1:1 |
|
||||
| 16 `Database connection string` | `postgres-connstr`, `mysql-connstr`, `redis-connstr` | 1:3, **minus the MongoDB SRV form** |
|
||||
| 17 `OpenAI Legacy API Key` | `openai-api-key-legacy` | 1:1, **byte-identical** |
|
||||
| 18 `JWT (three-part token)` | `jwt-token` | 1:1 |
|
||||
|
||||
Guard ids with no commons entry firing on their own witness: `gcp-service-account-json`,
|
||||
`mongodb-connstr` — and `ec-private-key` on the `ENCRYPTED` header.
|
||||
|
||||
`Azure Connection String` is listed as *overlap* rather than 1:1 deliberately. Commons'
|
||||
entry is `(?:AccountKey|SharedAccessKey|sig)=[A-Za-z0-9+/=]{20,}` — three alternatives, no
|
||||
length pin. The guard's `azure-storage-key` is `AccountKey=([A-Za-z0-9+/]{86}==)` — one
|
||||
alternative, exact length. The corpus witnessed only the `AccountKey` shape, where both fire.
|
||||
`SharedAccessKey=` and `sig=` were not witnessed; seed B carries them under separate ids
|
||||
(`azure-servicebus-connstr`, `azure-sas-token`) that the guard did not port. Read this row as
|
||||
"one witnessed overlap", not as a coverage claim.
|
||||
|
||||
## What each side misses that the other catches
|
||||
|
||||
**Commons reports, guard silent — 7 witnesses across 5 commons entries:**
|
||||
|
||||
| Witness shape | commons entry | Why the guard is silent |
|
||||
| --- | --- | --- |
|
||||
| `ghu_` prefixed token | `GitHub Token` | guard ported `ghp`/`gho`/`ghs`; no id for `ghu` |
|
||||
| `ghr_` prefixed token | `GitHub Token` | same |
|
||||
| `aws_secret_access_key = <40 chars>` | `AWS Secret Access Key` | seed B has `aws-secret-access-key`; guard did not port it |
|
||||
| `Ocp-Apim-Subscription-Key` assignment | `Azure AI Services Key` | absent from seed B entirely |
|
||||
| `JWT_SECRET` assignment | `JWT Secret` | absent from seed B entirely |
|
||||
| Slack webhook URL | `Slack/Discord Webhook URL` | absent from seed B entirely |
|
||||
| Discord webhook URL | `Slack/Discord Webhook URL` | same |
|
||||
|
||||
One row is one witness, so two commons entries appear twice: `GitHub Token` covers five
|
||||
prefixes behind one name, and `Slack/Discord Webhook URL` covers two hosts. Counting rows
|
||||
rather than entries would overstate how much of commons the guard is missing, and counting
|
||||
entries rather than rows would hide that `GitHub Token` is only *partly* uncovered — its
|
||||
`ghp`/`gho`/`ghs` prefixes map onto three guard ids just fine.
|
||||
|
||||
Three of the seven are the sharper finding: the `Ocp-Apim-Subscription-Key`, `JWT_SECRET` and
|
||||
webhook shapes are not entries the guard declined to port, they are entries **seed B does not
|
||||
have**. Seed A carries three shapes seed B never did.
|
||||
|
||||
**Guard reports, commons silent — 3 shapes:**
|
||||
|
||||
| Shape | guard id | Why commons is silent |
|
||||
| --- | --- | --- |
|
||||
| `"type": "service_account"` | `gcp-service-account-json` | seed A has no GCP service-account marker |
|
||||
| `-{5}BEGIN ENCRYPTED PRIVATE KEY-{5}` | `ec-private-key` | commons' PEM alternation is `(?:RSA \| EC \| DSA \| OPENSSH )?`; `ENCRYPTED` is not in it |
|
||||
| `mongodb+srv://user:pw@host` | `mongodb-connstr` | commons' scheme run is `(?:postgres\|mysql\|mongodb\|redis)://` — the `+srv` suffix breaks the literal |
|
||||
|
||||
The `mongodb+srv` miss is worth naming precisely: commons is not missing MongoDB, it is
|
||||
missing the **SRV** form, which is the form Atlas hands out. Plain `mongodb://` is caught.
|
||||
|
||||
**This asymmetry is not a scoreboard.** Each side is faithful to its own seed. Every shape in
|
||||
the left table is present in seed A and absent from seed B; every shape in the right table is
|
||||
the reverse. Neither port is wrong about its source. The seeds disagree.
|
||||
|
||||
## False-positive suppression: a layer commons has no field for
|
||||
|
||||
The guard applies value-based suppression to the five entries that capture a value
|
||||
(`_is_fp_value`): structural placeholders (`your-`, `<`, `>`, `***`), word-boundary
|
||||
placeholder words (`example`, `changeme`, `todo`, …), variable references (`${`, `$(`,
|
||||
`os.environ`, `process.env`, …), all-same-character values, and values under 8 characters.
|
||||
|
||||
Measured:
|
||||
|
||||
| Witness | commons (first match) | guard |
|
||||
| --- | --- | --- |
|
||||
| `password: 'your-password-here'` | `Generic credential assignment` | — suppressed |
|
||||
| `api_key: '${MY_API_KEY_VALUE}'` | `Generic credential assignment` | — suppressed |
|
||||
|
||||
Commons has no field that could carry this. `dialect.translation_notes` warns in prose that
|
||||
the generic entries are *"shape matches, not proofs of a live credential"* and assigns the
|
||||
trade-off to the consumer's policy — which is a correct statement of ownership and is also
|
||||
why two consumers reading commons will produce different reports on the same placeholder.
|
||||
Seed B carries the suppression semantics per entry in a `false_positive_notes` field; seed A
|
||||
carries name and pattern only, so commons had nothing to extract. This is a gap in the seed,
|
||||
not an omission in the extraction.
|
||||
|
||||
## The connection-string bound
|
||||
|
||||
The guard bounds the password run in all four connection-string patterns at
|
||||
`MAX_CONNSTR_VALUE = 256`, and its module explains why in full: an unbounded run in front of
|
||||
a required literal makes every start position rescan the tail when the literal never arrives.
|
||||
They measured 8.2 s at 100 000 characters on crafted `redis://:` input and extrapolated to
|
||||
hours at their own 1 000 000-character cap. Seed B's connection-string patterns are unbounded;
|
||||
this is one of the 5 real guard-vs-seed-B drifts, and it is a deliberate, documented one.
|
||||
|
||||
Commons' `Database connection string` is `(?:postgres|mysql|mongodb|redis):\/\/[^\s]+@[^\s]+`
|
||||
— **shape-analogous** to what the guard bounded. Measured against the exact boundary:
|
||||
|
||||
| Password length | commons | guard |
|
||||
| --- | --- | --- |
|
||||
| 12 | matches | `egress:postgres-connstr` |
|
||||
| 256 | matches | `egress:postgres-connstr` |
|
||||
| 257 | matches | — |
|
||||
| 300 | matches | — |
|
||||
|
||||
Read this as two facts, not one verdict. Commons has recall the guard traded away above 256
|
||||
characters. Commons also carries the runtime shape the guard's measurement was about — and
|
||||
carries it in an *unanchored* form (`[^\s]+@[^\s]+` rather than the guard's
|
||||
`[^:@\s]+:…@[^\s'"]+`), so the two are not the same pattern under load and no timing claim
|
||||
about commons is made here. **Nothing is changed on that basis.** The entry is faithful to
|
||||
seed A, the file that owns it is `llm-security`'s, and the behaviour-preservation invariant
|
||||
puts the decision there. It is reported, and the guard's measurement is cited so the owner
|
||||
does not have to redo it.
|
||||
|
||||
## Severity and ids: what commons does not carry
|
||||
|
||||
Seed B carries `id` and `severity` per entry; the guard preserved both, and all 25 severities
|
||||
are field-identical to the seed. Seed A carries neither, so commons carries neither, and its
|
||||
`evidence_limits` says so explicitly: *"No severity, and no per-entry disposition, was
|
||||
supplied … so neither is invented here."*
|
||||
|
||||
That restraint was right and it has a consequence: **commons has no id space for this table.**
|
||||
Its entries are keyed by human-readable `name` (`"GitHub Token"`), while the guard emits
|
||||
`egress:<id>` labels. A `conformance/expected.json` scoped to secret egress cannot be written
|
||||
against commons today, because a fixture names labels and commons has none to name.
|
||||
|
||||
The 25 guard ids are **not guard-internal labels**. They are seed B's ids, adopted verbatim,
|
||||
which was measured above (25/25 present in the seed). That makes the id space question a
|
||||
question for `llm-security` first — they own both seeds and the id space in one of them — and
|
||||
for the guard second. Per this repository's naming rule, **no id is proposed here.** The rule
|
||||
that `carrier:*` established applies exactly: naming an id in a shared space is the exception,
|
||||
it requires both runtimes asked first, and publishing `aliases.<runtime>` is irreversible at
|
||||
file granularity.
|
||||
|
||||
## What this does not show
|
||||
|
||||
- **It does not show that either table is wrong.** Both are faithful ports. The disagreement
|
||||
is between seed A and seed B, inside `llm-security`, and only that repository can say
|
||||
whether two tables is intentional (one engine-consumed, one agent-consumed) or whether one
|
||||
supersedes the other.
|
||||
- **It does not measure seed A's current state.** Commons' fidelity to seed A was verified at
|
||||
commit `47905da` and this document adds nothing to that.
|
||||
*(Seed B was read at the same commit, which is a shared coordinate and not the commit the
|
||||
guard ported from. That was going to be a caveat — a seed-B entry that moved between the
|
||||
guard's port and `47905da` would show up here as guard drift. It is dissolved by measurement
|
||||
instead: `git log -- knowledge/secrets-patterns.md` in a deepened mirror returns exactly one
|
||||
commit at or before `47905da`, `f153f96`, dated 2026-04-08, and the guard's `output.py` was
|
||||
first committed 2026-07-04. The seed had been still for three months when the port was
|
||||
written and has not moved since. Reading it at `47905da` reads what the guard ported from,
|
||||
so the 5 real drifts are guard-side by measurement rather than by inference.)*
|
||||
- **It does not compare coverage.** The probe corpus has one witness per guard id plus each
|
||||
side's exclusive shapes — 36 inputs. It is built to expose membership and semantics, not to
|
||||
estimate recall. `SharedAccessKey=` and `sig=` Azure shapes, and seed B's 8 unported ids,
|
||||
have no witness here.
|
||||
- **It does not measure the runtimes' entry points.** Both sides were driven at table level:
|
||||
commons through an evaluator compiled from its own JSON under its own declared contract, the
|
||||
guard through `scan_secret_egress` directly. What `scan_output` composes around it —
|
||||
decode-and-rescan re-labelling findings as `decoded:egress:*`, the oversize cap — is not in
|
||||
scope and would change the finding sets.
|
||||
- **It does not touch `manifest.json`.** `scope_planned.blockers` still names this divergence
|
||||
as the blocker for egress cases. Whether this document dissolves that blocker or merely
|
||||
describes it is a separate decision, and it depends on answers this document does not have.
|
||||
|
||||
## Consequence for `conformance/`
|
||||
|
||||
An egress case is not mintable today, and the reason has changed. It was recorded as "the two
|
||||
tables are cut at different granularity". The measured reasons are three, and they are
|
||||
independent:
|
||||
|
||||
1. **No id space on the commons side.** A fixture names labels. Commons has names, not ids.
|
||||
This is the hard blocker and it is the subject of the outgoing question to both runtimes.
|
||||
2. **Match semantics disagree.** Even with an id space, the Bearer-plus-JWT witness produces a
|
||||
one-label expectation under commons' declared contract and a two-label one from the guard.
|
||||
A fixture would have to encode one of them.
|
||||
3. **Membership disagrees in both directions**, and the disagreement is inherited from two
|
||||
different seeds rather than from a porting error — so it cannot be closed by re-measuring
|
||||
either port.
|
||||
|
||||
None of the three is dissolved by a measurement this repository can run alone. Per
|
||||
`conformance/manifest.json` → `entry_points_by_scope`, a new scope also needs an entry point,
|
||||
a findings accessor and a fixture presentation for every runtime before its first case, and
|
||||
those three slots are empty for egress on both runtimes. That requirement stands independently
|
||||
of everything above.
|
||||
|
|
@ -284,6 +284,47 @@ This ordering is the whole point of the repository. Two implementations that ret
|
|||
different verdicts on the same input are not holding different opinions; one of them has a
|
||||
bug.
|
||||
|
||||
### 7.1 Where the second paragraph does not hold
|
||||
|
||||
**Through corpus version 0.8.1 this section stated the rule above with no scope at all**, and
|
||||
the scope was load-bearing: the justification names *two* implementations. Where a case's
|
||||
scope is a table only one runtime implements, and that runtime authored the payload the case
|
||||
was extracted from, there is no second implementation whose disagreement the paragraph could
|
||||
adjudicate. Which cases those are is recorded in the corpus, not asserted per run — see
|
||||
`active_content_provenance.asymmetry` in
|
||||
[`conformance/manifest.json`](../conformance/manifest.json).
|
||||
|
||||
For such a case, a disagreement by the **seed runtime itself** is a third thing, and it is
|
||||
neither of the two the paragraph offers:
|
||||
|
||||
- The fixture is not proven wrong. It recorded that runtime's behaviour correctly at the
|
||||
commit and version its own measurement block pins, and a later classification does not
|
||||
reach back and falsify an earlier measurement.
|
||||
- The runtime does not necessarily have a bug. Where the seed runtime has stated that the
|
||||
classification behind such a table is calibration it does not freeze, a release that
|
||||
classifies the payload differently is a change it reserved, not a defect.
|
||||
|
||||
So: the fixture MUST NOT be rewritten on the strength of the divergence alone; the divergence
|
||||
SHOULD be recorded against the version pinned; and re-pinning the case to a later version of
|
||||
the seed runtime is a separate decision, taken deliberately and released on its own. This is
|
||||
the disposition §5 already applies to a stale `observed_out_of_scope` entry, extended to the
|
||||
one place where it can reach a verdict — and a divergence recorded here is the signal that
|
||||
the re-pinning decision is due, not a reason to leave it open.
|
||||
|
||||
Three things this does **not** do.
|
||||
|
||||
- **It creates no fourth verdict.** The counts of §1.1 and
|
||||
[`schema/conformance-declaration.schema.json`](../schema/conformance-declaration.schema.json)
|
||||
are unchanged: a case whose expected findings are not produced still **fails**, and is still
|
||||
named in `failed_cases`. What changes is what the failure licenses concluding, not what is
|
||||
reported.
|
||||
- **It does not reach a third-party implementer** of the same table. Against them the fixture
|
||||
is the contract, exactly as §7 states — which is what these cases were minted to provide,
|
||||
and the only thing they can prove while one runtime is all there is.
|
||||
- **It is not a licence to self-declare.** The exemption is carried by the corpus's own
|
||||
provenance record for the scope. A runtime MUST NOT claim it for a case by asserting that
|
||||
its own divergence is calibration.
|
||||
|
||||
## 8. What conformance does and does not prove
|
||||
|
||||
Passing this corpus proves that a runtime agrees with the other runtimes that pass it, on
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue