docs(knowledge): the judgment axis is knowledge, not a detector (B2)
Article rule 1 ("give Claude judgement instead of rules") gets its register
entry, and deliberately no detector. `BP-JUDG-001` carries `lensCheck: null`.
The cut between deterministic prefilter and prose judge was the open design
decision. It was settled by measurement, and the measurement declined both
halves:
- 409 real CLAUDE.md files (38488 lines, 8689 prose blocks): the caging class
fires 7 times, and all 7 are false positives ("rendered prose-side",
"naming is a flag on the class"). Verified along an independent grep path
that bypasses block-splitting and sentence-splitting entirely, in both word
orders: 5 lines and 1 line, none an instruction.
- The narrow variant (absolute + form noun + numeric cap) fired 8 times —
one duplicated block seen seven times across plugin caches, precision 0 %.
The pre-committed rule required 90 % over 20 distinct fires.
- Where the shape does occur — 45 lines across 4755 skill/agent/command files
— it is the author's editorial policy (emoji, sentence length, slide
titles). Nothing in the text separates that from a vendor's over-tight
guardrail, and the article's reasoning does not transfer: the model is not
the author of a user's config.
So no CA-OPT-002; finding-codes keeps OPT next-free = 2. The numbers live in
the entry's own `note`, so the next session does not re-derive the question.
Two premises the chunk falsified. The brief justified a separate axis by
saying these blocks sit inside `floor-exclusion`'s floor — but the article's
own canonical line carries no floor marker at all, so "inside the floor"
cannot define the axis (the corpus tendency is 76 %, which is a tendency, not
a mechanism). And the fasit's own form-noun vocabulary was wrong: `name` and
`format` alone drove 97 % of fires.
Not folded into `--subtract`: a third "loosen instead of delete" verdict in
the subtraction judge is the AS#5 mixing STATE forbids, and with the corrected
vocabulary there are 0 collisions to arbitrate anyway.
Guards, both seen red against their own defect first: the entry must exist,
be confirmed, date its source and name NO lensCheck; and every lensCheck in
the register must be backed by a real detector.
No behaviour changed — no new finding, no output change, nothing consumes the
entry yet — hence `docs`, not `feat`. Suite 1701 -> 1703/0; frozen v5.0.0 and
default-output baselines 0 changed files. Fasit:
docs/b2-judgment-lens-fasit.local.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017mCkx9wGywqNQzsXkBMzJ1
This commit is contained in:
parent
dbb6a6a3cf
commit
6bb100f2e0
3 changed files with 86 additions and 8 deletions
|
|
@ -46,12 +46,19 @@ scanner *input*, so the axes compose instead of competing.
|
||||||
**Deviation from brief, verified 2026-08-03:** the article contains NO mechanism-choice or
|
**Deviation from brief, verified 2026-08-03:** the article contains NO mechanism-choice or
|
||||||
size-limit content → it was NOT added to BP-MECH-*/BP-SIZE-001 (that would be false
|
size-limit content → it was NOT added to BP-MECH-*/BP-SIZE-001 (that would be false
|
||||||
provenance). If a future read finds real coverage, add it then.
|
provenance). If a future read finds real coverage, add it then.
|
||||||
- **B2 — new lens axis (own chunk):** `BP-JUDG-001` + `CA-OPT-002` for instructions that are
|
- **B2 — DONE, and the detector was DECLINED by measurement.** `BP-JUDG-001` ships as
|
||||||
local and specific but **over-specify and cage judgment** (article rule 1). NOT an extension
|
register knowledge with `lensCheck: null`; **no `CA-OPT-002`** (OPT next-free stays 2).
|
||||||
of `--subtract` — the floor (`floor-exclusion.mjs`) rightly protects these blocks from
|
Measured over **409 real CLAUDE.md files** (38 488 lines, 8 689 prose blocks): the caging
|
||||||
deletion; this axis says *keep the content, loosen the phrasing*. Same precision gate as
|
class fires **7 times, all 7 false positives**, confirmed along an independent grep path in
|
||||||
`optimization-lens-agent` (cite rule + source, stay silent when unsure). Mixing the axes is
|
both word orders (5 lines / 1 line, none an instruction). Where the shape *does* occur — 45
|
||||||
the ÅS#5 defect class.
|
lines across 4 755 skill/agent/command files — it is the author's **editorial policy** (emoji,
|
||||||
|
sentence length, slide titles), and nothing in the text separates that from a vendor's
|
||||||
|
over-tight guardrail: the article's reasoning does not transfer, because the model is not the
|
||||||
|
author of a user's config. Precision-first ⇒ silence. Two premises the chunk falsified: the
|
||||||
|
brief's «these blocks are inside the floor» (the article's own example carries **no** floor
|
||||||
|
marker; the corpus tendency is 76 %, which is not a mechanism), and the §4 form-noun
|
||||||
|
vocabulary (`name`/`format` alone were 97 % of fires). Full record:
|
||||||
|
`docs/b2-judgment-lens-fasit.local.md` §9.
|
||||||
- **B3 — two-layer duplication/contradiction (CNF extension):** article rule 4 + `/doctor`'s
|
- **B3 — two-layer duplication/contradiction (CNF extension):** article rule 4 + `/doctor`'s
|
||||||
measured Sjekk 2 catch (global CLAUDE.md model-policy ↔ agent frontmatter) prove the class
|
measured Sjekk 2 catch (global CLAUDE.md model-policy ↔ agent frontmatter) prove the class
|
||||||
exists and is catchable. Extend `conflict-detector` with cross-layer checks where the pair is
|
exists and is catchable. Extend `conflict-detector` with cross-layer checks where the pair is
|
||||||
|
|
@ -201,8 +208,11 @@ Check 3) and fails the binding `/doctor` positioning; and regenerating destroys
|
||||||
- **C1–C5:** criteria as specified 2026-07-14 (C2 red-first nearest-match; C3 fixture
|
- **C1–C5:** criteria as specified 2026-07-14 (C2 red-first nearest-match; C3 fixture
|
||||||
missing-path fires / URL-glob-placeholder silent; C4 authored-agent matrix; C5 failure-modes
|
missing-path fires / URL-glob-placeholder silent; C4 authored-agent matrix; C5 failure-modes
|
||||||
section present).
|
section present).
|
||||||
- **B2:** fixture with a precise-but-caging instruction → CA-OPT-002 with rule+source citation;
|
- **B2 (superseded by the measurement above):** the criterion was written for a detector that
|
||||||
floor-protected block WITHOUT caging phrasing → silent (axis separation proven).
|
measurement declined. What was verified instead: `BP-JUDG-001` present, `confirmed`, primary
|
||||||
|
source dated `2026-07-24`, `lensCheck` **absent** — plus a guard that every `lensCheck` in the
|
||||||
|
register is backed by a real detector, so no later session can "complete" the entry by wiring
|
||||||
|
one. Both assertions seen RED against their own defect before landing.
|
||||||
- **B3:** fixture with same instruction in CLAUDE.md + rule → CNF finding; single-layer only →
|
- **B3:** fixture with same instruction in CLAUDE.md + rule → CNF finding; single-layer only →
|
||||||
silent ([[guard-can-be-green-on-its-own-defect]]: assert the blanket invariant).
|
silent ([[guard-can-be-green-on-its-own-defect]]: assert the blanket invariant).
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -263,6 +263,22 @@
|
||||||
"note": "Verbatim: 'Claude will be more predisposed to double-checking additional hypotheses or verifying correctness at higher effort levels, but it generally won't artificially inflate usage for simple tasks at higher effort levels.'; 'In fact, our team pays close attention to \"overthinking\" during model training as it degrades effectiveness.'"
|
"note": "Verbatim: 'Claude will be more predisposed to double-checking additional hypotheses or verifying correctness at higher effort levels, but it generally won't artificially inflate usage for simple tasks at higher effort levels.'; 'In fact, our team pays close attention to \"overthinking\" during model training as it degrades effectiveness.'"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "BP-JUDG-001",
|
||||||
|
"claim": "An instruction that fixes a form decision — how code or prose should look (comment density, docstring length, naming shape, sentence or paragraph count) — as an absolute rule buys a guardrail current models no longer need, and is wrong for the cases the rule did not anticipate. Anthropic removed its own example from the Claude Code system prompt: \"Never write multi-paragraph docstrings or multi-line comment blocks — one short line max\" was replaced by \"Write code that reads like the surrounding code: match its comment density, naming, and idiom.\"",
|
||||||
|
"appliesTo": "claude-md",
|
||||||
|
"recommendation": "Where an absolute governs a form decision rather than a local fact, state the outcome you want and let the model judge the instance. This does not apply to safety rules, tool or version facts, or a house style you hold deliberately — those are the reason the claim is not machine-checkable.",
|
||||||
|
"confidence": "confirmed",
|
||||||
|
"category": "judgment-fit",
|
||||||
|
"lensCheck": null,
|
||||||
|
"note": "KNOWLEDGE ONLY — no detector, by measurement (docs/b2-judgment-lens-fasit.local.md §9). Across 409 real CLAUDE.md files (38488 lines, 8689 prose blocks) the class fired 7 times and all 7 were false positives; an independent grep in both word orders found 5 lines and 1 line respectively, none an instruction. Where the shape does occur — 45 matching lines across 4755 skill/agent/command files — it is the author's editorial policy (emoji, sentence length, slide titles), which nothing in the text separates from a vendor's over-tight guardrail. Precision-first: no CA-OPT code was allocated and OPT next-free stays 2.",
|
||||||
|
"source": {
|
||||||
|
"url": "https://claude.com/blog/the-new-rules-of-context-engineering-for-claude-5-generation-models",
|
||||||
|
"title": "The new rules of context engineering for Claude 5 generation models",
|
||||||
|
"published": "2026-07-24",
|
||||||
|
"verified": "2026-08-12"
|
||||||
|
}
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -7,6 +7,8 @@ import {
|
||||||
CONFIDENCE_LEVELS,
|
CONFIDENCE_LEVELS,
|
||||||
REGISTER_PATH,
|
REGISTER_PATH,
|
||||||
} from '../../scanners/lib/best-practices-register.mjs';
|
} from '../../scanners/lib/best-practices-register.mjs';
|
||||||
|
import { LENS_DETECTORS } from '../../scanners/lib/lens-prefilter.mjs';
|
||||||
|
import { SUBTRACT_DETECTORS } from '../../scanners/lib/subtraction-prefilter.mjs';
|
||||||
|
|
||||||
// A minimal well-formed entry; negative tests clone + mutate this.
|
// A minimal well-formed entry; negative tests clone + mutate this.
|
||||||
const validEntry = () => ({
|
const validEntry = () => ({
|
||||||
|
|
@ -107,6 +109,56 @@ describe('bundled register integrity (Verifiseringsplikt)', () => {
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// B2. The judgment entry is knowledge WITHOUT a detector, and that is the
|
||||||
|
// measured outcome, not an omission: the caging class the article names fired
|
||||||
|
// 7 times across 409 real CLAUDE.md files and all 7 were false positives
|
||||||
|
// (docs/b2-judgment-lens-fasit.local.md §9.4). Asserting the absent lensCheck
|
||||||
|
// is what stops a later session from "completing" the entry by wiring a
|
||||||
|
// detector the corpus refused.
|
||||||
|
it('carries the judgment entry (B2) as detector-less knowledge', () => {
|
||||||
|
const e = getEntry(reg, 'BP-JUDG-001');
|
||||||
|
assert.ok(e, 'BP-JUDG-001 missing from the bundled register');
|
||||||
|
assert.equal(e.confidence, 'confirmed', 'BP-JUDG-001 must be confirmed');
|
||||||
|
assert.equal(e.category, 'judgment-fit', 'BP-JUDG-001 wrong category');
|
||||||
|
assert.equal(
|
||||||
|
e.lensCheck ?? null,
|
||||||
|
null,
|
||||||
|
'BP-JUDG-001 must NOT name a lensCheck — no detector survived measurement'
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
e.source.url,
|
||||||
|
'https://claude.com/blog/the-new-rules-of-context-engineering-for-claude-5-generation-models',
|
||||||
|
'BP-JUDG-001 primary source must be the article that states rule 1'
|
||||||
|
);
|
||||||
|
assert.equal(e.source.published, '2026-07-24', 'BP-JUDG-001 must date its primary source');
|
||||||
|
assert.match(
|
||||||
|
String(e.note),
|
||||||
|
/409/,
|
||||||
|
'BP-JUDG-001 must carry the measured negative result, so it is not re-derived'
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
// The register's own consumers key on lensCheck; an entry that names one it
|
||||||
|
// does not have would reach a payload with no detector behind it.
|
||||||
|
it('every lensCheck in the register is backed by a detector', () => {
|
||||||
|
const detectors = new Set([
|
||||||
|
...LENS_DETECTORS.map((d) => d.lensCheck),
|
||||||
|
...SUBTRACT_DETECTORS.map((d) => d.lensCheck),
|
||||||
|
// Scanner-side checks that are their own detector.
|
||||||
|
'procedure-in-claude-md',
|
||||||
|
'CA-OST-001',
|
||||||
|
'CA-CML-001',
|
||||||
|
'CA-SKL-002',
|
||||||
|
]);
|
||||||
|
for (const e of reg.entries) {
|
||||||
|
if (e.lensCheck == null) continue;
|
||||||
|
assert.ok(
|
||||||
|
detectors.has(e.lensCheck),
|
||||||
|
`${e.id} names lensCheck "${e.lensCheck}" with no detector behind it`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe('validateRegister (negative cases)', () => {
|
describe('validateRegister (negative cases)', () => {
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue