Scores committed agent-run fixtures against the golden corpus at
(file, rule_key) granularity, building on the deterministic coordinator
contract (4a). Offline: committed reviewer payloads, no live agent spawn,
no LLM, no network (the LLM-in-the-loop grading is the separate 4c tier).
- lib/review/gold-scorer.mjs: scoreFindings (precision/recall/f1 at
(file,rule_key) granularity, line+severity ignored) + scoreVerdict; pure,
with documented vacuous-set conventions.
- tests/fixtures/bakeoff-rich/runs/run-perfect.json: committed run that
reproduces all 5 seeded gold findings through runContract.
- tests/lib/gold-eval.test.mjs: the scoring RUN (precision/recall/f1 = 1.0,
verdict == expected_verdict BLOCK, nothing suppressed/skipped).
- lib/util/test-census.mjs: third census category (goldEval) — a scoring run
is neither behavior coverage nor a doc-pin; honest-count invariant now 3-way.
- docs/eval-corpus/README.md: 4b moved from Future hardening to implemented.
Suite 809 -> 822 (820/0/2). gold-scorer covers TP+FP+FN+degenerate paths.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BJQYC5vpkJWxndS55vQQZ6
Run the full T2 §5 prose-vs-Workflow /trekreview bake-off (operator GO,
choice "a"): 3 runs/arm on a rich-finding JWT-auth fixture, resolving the
smoke's 0-finding limitation.
Deliverables:
- tests/fixtures/bakeoff-rich/ — JWT-auth brief + diff with 5 seeded blatant,
brief-traceable issues (varied severity/rule_key, one dual-flaggable).
- scripts/bakeoff-armA-merge.mjs — Arm A (prose) validate (NW1) + triplet-dedup,
matching Arm B's dedup exactly.
- scripts/bakeoff-fidelity.mjs — cross-arm + within-arm + granularity-ladder
fidelity analysis over the structured arm outputs.
- docs/T2-bakeoff-results.md §Full run — the T2 §5 verdict.
Result (3 runs/arm, both arms ran the coordinator):
- Verdict fidelity EQUIVALENT — all 6 runs BLOCK, cross-arm verdict-match 1.0.
- Finding-set: substrate is fidelity-neutral. Cross-arm jaccard 0.41 (triplet)
→ 0.71 (file,rule_key) → 1.0 (file); cross-arm ≈ within-arm at every
granularity. Issue coverage 5/5 in 6/6 runs. Low triplet jaccard is
line-citation noise shared by both arms, not a substrate effect.
- Token +4.4% (Arm B vs A; <=+15%). Classifier interference 0 at 9-agent
concurrency. JSON-robustness: Arm B schema-forced; Arm A 6/6 valid via NW1.
- VERDICT POSITIVE → S11 proceeds with opt-in --workflow flag.
Caveat (per plan posture): strict triplet-jaccard>=0.7 flag is 0/9, a
metric-calibration artifact (both arms sub-0.7 against themselves), not a
regression. Residual: F4 auto/bypass explicit-mode check (mode not settable
in-session).
Suite green (662/660 pass, 2 skip); plugin validate clean (modulo the
pre-existing root-CLAUDE.md warning). No production code changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
Build the prose-vs-Workflow bake-off machinery for /trekreview Phase 5-6 and
run a 1-run/arm smoke to de-risk before the full measurement (operator posture:
build + smoke, then pause for go/no-go on the full >=3-runs/arm run).
New:
- lib/review/fidelity-diff.mjs (+ tests) — the PRIMARY metric: parse two
review.md (or two structured arm outputs) and compare verdict + jaccard over
(file,line,rule_key)-IDs + per-finding severity/rule_key. Reuses jaccard +
frontmatter + NW1 findings-schema + finding-id. fidelityDiffStructured avoids
rendering review.md per run.
- scripts/trekreview-armB.workflow.mjs — Arm B: Phase 5-6 as a Workflow
(parallel([conformance, correctness]) schema-forced -> JS dedup-by-triplet ->
agent(review-coordinator) verdict schema). Path-based input via args (reviewers
carry Read). Inlines dedup + the 12-key rule_key enum (scripts have no imports).
- tests/fixtures/bakeoff/ — committable fixture: real diff of b149538 (NW1) +
brief reconstructed from plan S9. Both arms review the same pinned input.
- docs/T2-bakeoff-results.md — smoke results + verdict + go/no-go recommendation.
Smoke result: SMOKE PASS. Arm B runs the full pipeline (3 agents) with ZERO
classifier interference; fidelity EQUIVALENT to Arm A at the verdict level
(both ALLOW; jaccard 1.0). Caveat: the clean TDD'd fixture yielded ~0 findings,
so finding-SET fidelity was not stressed (only verdict fidelity proven). A
reviewer-level divergence appeared (Arm B raised 1 raw finding, coordinator
filtered it; Arm A raised 0) — to be quantified in the full run on a
richer-finding-surface fixture. NOT the T2 §5 POSITIVE/NEGATIVE verdict.
Suite 647 -> 662 (660 pass / 2 skip / 0 fail; +15 fidelity-diff). claude plugin
validate clean (known root-CLAUDE.md warning only). Plan: docs/W1-narrow-wins-plan.md S10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
The v4.2/v4.3 bespoke playground SPA (~388 KB), the /trekrevise command,
Handover 8 (annotation → revision), the supporting lib/ modules
(anchor-parser, annotation-digest, markdown-write, revision-guard), the
Playwright e2e suite, and the @playwright/test / @axe-core/playwright
devDeps are removed. A browser walkthrough found the playground borderline
unusable, and it duplicated the official /playground plugin's
document-critique / diff-review templates.
In their place: scripts/render-artifact.mjs — a small, zero-dependency
renderer that turns a brief/plan/review .md into a self-contained,
design-system-styled, zero-network .html (frontmatter folded into a
<details> block). /trekbrief, /trekplan, and /trekreview call it on their
last step and print the file:// link; to annotate, run /playground
(document-critique) on the .md and paste the generated prompt back.
Resolves the v4.3.1-deferred findings as moot (their target files are
deleted). npm test green: 509 tests, 507 pass, 0 fail, 2 skipped.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Step 20 of v4.1 — implements drift detection in plan-validator.mjs per
brief Assumptions block 7: "Mismatch (e.g. korrupt manuell endring)
emitterer MANIFEST_PROFILE_DRIFT-warning fra plan-validator i --strict-modus."
Logic (after validateAllManifests in validatePlanContent):
1. Strict-mode only — soft mode never emits drift warnings.
2. Plan frontmatter must declare 'profile: <name>' to establish baseline.
3. For each step manifest, if profile_used is set AND differs from plan
profile, emit warning (NOT error) with code MANIFEST_PROFILE_DRIFT
and location 'step N: profile_used = X, plan profile = Y'.
Forward-compat preserved: drift is a warning, plan remains valid:true.
Operators see the drift in --strict mode without parsing breaking.
New files:
tests/validators/plan-validator-profile-drift.test.mjs — 4 tests
tests/fixtures/plan-profile-drift.md — drift fixture
Tests verify:
1. drift detected in strict mode → MANIFEST_PROFILE_DRIFT in warnings
2. drift NOT detected in soft mode → strict gate honored
3. matching profile → no drift warning
4. no plan-level profile → drift detection silent (no baseline)
Tests: 479 pass + 2 skipped (Docker not installed).
Synthetic plan.md fixture with source_findings: block-style YAML list of 3
40-char hex IDs in frontmatter, plus minimal plan structure (Title +
Implementation Plan + 1 Step + Manifest). 3 tests verify:
1. plan-validator accepts a plan with source_findings (additive optional field)
2. frontmatter parser extracts source_findings as array of strings
3. each ID matches the 40-char lowercase hex format from finding-id.mjs
Closes the SC3(b) gap flagged by adversarial review (scope-guardian Gap 2).
LLM-level behavior (planner emitting source_findings) remains non-testable
without live invocation; this covers the structural contract.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
review-run-A.md (5 findings) and review-run-B.md (6 findings, A ⊂ B) form a
known-Jaccard fixture pair: |A ∩ B| = 5, |A ∪ B| = 6, Jaccard = 5/6 = 0.833,
above the SC4 threshold of 0.70. IDs are real 40-char SHA1s computed via
lib/parsers/finding-id.mjs from realistic (file, line, rule_key) triplets.
Both fixtures pass review-validator --strict (frontmatter + body sections +
findings shape). Real-LLM determinism measurement deferred to v1.1.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Extract `/ultra-cc-architect-local` and `/ultra-skill-author-local` plus all 7
supporting agents, the `cc-architect-catalog` skill (13 files), the
`ngram-overlap.mjs` IP-hygiene script, and the skill-factory test fixtures
from `ultraplan-local` v2.4.0 into a new `ultra-cc-architect` plugin v0.1.0.
Why: ultraplan-local had drifted into containing two distinct domains — a
universal planning pipeline (brief → research → plan → execute) and a
Claude-Code-specific architecture phase. Keeping them together forced users
to inherit an unfinished CC-feature catalog (~11 seeds) when they only
wanted the planning pipeline, and locked the catalog and the pipeline into
the same release cadence. The architect was already optional and decoupled
at the code level — only one filesystem touchpoint remained
(auto-discovery of `architecture/overview.md`), which already handles
absence gracefully.
Plugin manifests:
- ultraplan-local: 2.4.0 → 3.0.0 (description + keywords updated)
- ultra-cc-architect: new at 0.1.0 (pre-release; catalog is thin, Fase 2/3
of skill-factory unbuilt, decision-layer empty, fallback list still
needed)
What stays in ultraplan-local: brief/research/plan/execute commands, all
19 planning agents, security hooks, plan auto-discovery of
`architecture/overview.md` (filesystem-level contract, not code-level).
What moved (28 files via git mv, R100 — full history preserved):
- 2 commands, 8 agents, 1 skill catalog (13 files), 2 scripts, 8 fixtures
Documentation updates: plugin CLAUDE.md and README.md for both plugins,
root README.md (added ultra-cc-architect section, updated ultraplan-local
section), root CLAUDE.md (added ultra-cc-architect to repo-struktur),
marketplace.json (registered ultra-cc-architect), ultraplan-local
CHANGELOG.md (v3.0.0 entry with migration guidance).
Test verification: ngram-overlap.test.mjs passes 23/23 from new location.
Memory updated: feedback_no_architect_until_v3.md now points at the new
plugin and reframes the threshold around catalog maturity rather than an
ultraplan-local milestone.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
skill-drafter now reads {catalog_root}/<slug>.md before writing its
draft and prepends a warning block to its confirmation output when
an existing skill would be overwritten during manual `mv` promotion.
The draft is still written to .drafts/<slug>.md — the check is a
hint, not a block.
Closes v2.3.0 dogfood finding (post_dogfood_findings[0]): the
drafter produced .drafts/hooks-pattern.md when an approved
hooks-pattern.md seed already existed, giving no signal that `mv`
during promotion would silently overwrite the seed. v2.3.1
introduced the qualified-slug mechanism to resolve such collisions;
v2.3.2 surfaces them at the right moment — before promotion.
Changes:
- agents/skill-drafter.md — new Step 2 between slug computation and
source reading. Reads {catalog_root}/<slug>.md, inspects
review_status, derives a kebab-case qualifier from the concept
handle (or source basename fallback). Subsequent steps renumbered
3→7. Output format gains Collision: field and optional warning
block. New Hard Rule.
- tests/fixtures/skill-drafter/slug-collision-expected.md — reference
fixture documenting expected confirmation shape across four
scenarios (no collision, approved collision, soft pending
collision, collision with no good qualifier). Skill-drafter is
prompt-driven; fixture anchors shape for human verification and
downstream parsers.
- CHANGELOG [2.3.2], plugin.json 2.3.1→2.3.2, README badge, plugin
CLAUDE.md slug-convention Collision-hint bullet, marketplace root
README summary, marketplace root CLAUDE.md plugin table.
Non-breaking. No frontmatter/drafts-layout/tool-scope/regex changes.
Existing pipelines see one extra field and an optional warning —
both purely additive.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
