fix(docs): rett README-statusen — to usanne paastander paa offentlig flate (ORDRE 20260825T122454Z)
"no live-model run yet" var falskt siden 14.08 (docs/2026-08-14-fase1b-forste-levende-kjoring.md): en live koersel mot gpt-4.1-mini konkluderte "rejected" — korrekt utfall, men modellen fant opp en kostkode. "sibling built in parallel" stemte ikke: po-claude er PARKERT (STATE.md). Begge punkt verifisert mot primaerkilde foer retting (Verifiseringsloven ansikt 3). Statusavsnittet er ogsaa gjort mer noekternt: ingen validerte forslag mot levende modell enna, ingen ekte ekspertdommer i treet (kun AI-forfattede froe, merket som saadan). Utvidet test_public_surface_claims_loadbearing.py med to nye rode->groenne gater (Iron Law): begge paastandene kan ikke gjeninnfores usett. Fant og fikset en linjebrudd-felle i egen test underveis (blockquote-wrap gjorde "no live-model run yet" usynlig for et raatt substring-soek). Ingen kodeendring i src/. Ingen push til open (Azure-gatet, egen ordre). 1025 passed / 5 skipped. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RF8QXXBWJ917Uj9zMf18nz
This commit is contained in:
parent
98cbb80a45
commit
932ece345b
2 changed files with 89 additions and 18 deletions
24
README.md
24
README.md
|
|
@ -261,16 +261,24 @@ per amount, before anything is summed.
|
||||||
- **Not a model benchmark.** The end-to-end proof runs offline against a scripted stand-in client:
|
- **Not a model benchmark.** The end-to-end proof runs offline against a scripted stand-in client:
|
||||||
it shows that the loop closes, not how well a given LLM proposes or judges.
|
it shows that the loop closes, not how well a given LLM proposes or judges.
|
||||||
|
|
||||||
> **Status:** the full 8-step agentic loop is wired and proven with load-bearing tests, and the
|
> **Status:** the full 8-step agentic loop is wired and proven with load-bearing tests. The
|
||||||
> end-to-end proof is an **offline simulation** with a scripted stand-in client — no live-model
|
> end-to-end proof is primarily an **offline simulation** with a scripted stand-in client, but one
|
||||||
> run yet. The **ingest layer** (real data sources) is implemented — file/CSV and SQL on both
|
> **live run** against a real endpoint (`gpt-4.1-mini`, 2026-08-14) has also completed: it ended
|
||||||
> stacks with bit-identical golden extractions from the shared spec, plus HTTP as a MAF-only
|
> in a correct `rejected` outcome — the deterministic validator caught a cost line the model had
|
||||||
> demonstrated extension point against a local mock — but exercised only against committed
|
> invented outright (a code absent from the knowledge base), on the tolerance gate rather than the
|
||||||
> fixtures: no bundle has yet been materialized from a live source. Ingested content passes a
|
> stricter existence gate, because the bundle it ran against ships no cost baseline to anchor
|
||||||
> **content gate** (`materialize_gated`) that scans every generated concept with
|
> against. No run has yet produced a **validated** proposal against a live model, and every
|
||||||
|
> human-in-the-loop verdict currently seeded into the knowledge base is a synthetic, AI-authored
|
||||||
|
> seed marked as such — no genuine expert verdict has entered the tree yet. The **ingest layer**
|
||||||
|
> (real data sources) is implemented — file/CSV and SQL on both stacks with bit-identical golden
|
||||||
|
> extractions from the shared spec, plus HTTP as a MAF-only demonstrated extension point against a
|
||||||
|
> local mock — but exercised only against committed fixtures: no bundle has yet been materialized
|
||||||
|
> from a live source. Ingested content passes a **content gate** (`materialize_gated`) that scans
|
||||||
|
> every generated concept with
|
||||||
> [`llm-ingestion-guard`](https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security)
|
> [`llm-ingestion-guard`](https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security)
|
||||||
> before any of it reaches the bundle; a refused run writes nothing. A sibling implementation of
|
> before any of it reaches the bundle; a refused run writes nothing. A sibling implementation of
|
||||||
> the same method on the **Claude Agents SDK** is built in parallel from the same shared spec.
|
> the same method on the **Claude Agents SDK** exists in a separate repository but is currently
|
||||||
|
> **parked**, not developed in parallel.
|
||||||
|
|
||||||
> **Disclaimer — technical framework only.** Deploying organizations own their processing
|
> **Disclaimer — technical framework only.** Deploying organizations own their processing
|
||||||
> purposes and assessments (DPIA, risk/ROS, security review). The framework ships the technical
|
> purposes and assessments (DPIA, risk/ROS, security review). The framework ships the technical
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
"""Load-bearing gates on two claims the PUBLISHED surface makes about itself.
|
"""Load-bearing gates on claims the PUBLISHED surface makes about itself.
|
||||||
|
|
||||||
AAA+ criterion A5 is that no claim on the public surface is untrue. Two of this repo's claims are
|
AAA+ criterion A5 is that no claim on the public surface is untrue. Four of this repo's claims are
|
||||||
made in prose that no test could see, and both drift silently:
|
made in prose that no test could see, and all four drift silently:
|
||||||
|
|
||||||
1. ``env.template`` tells the reader which credential the AZURE profile resolves. It said
|
1. ``env.template`` tells the reader which credential the AZURE profile resolves. It said
|
||||||
``DefaultAzureCredential`` while :mod:`portfolio_optimiser.backends` has never constructed one —
|
``DefaultAzureCredential`` while :mod:`portfolio_optimiser.backends` has never constructed one —
|
||||||
|
|
@ -10,12 +10,22 @@ made in prose that no test could see, and both drift silently:
|
||||||
2. ``README.md`` publishes a wheel-install command that spells the wheel's FILENAME, and a wheel
|
2. ``README.md`` publishes a wheel-install command that spells the wheel's FILENAME, and a wheel
|
||||||
filename carries the version. A version bump moves the file the build produces without touching
|
filename carries the version. A version bump moves the file the build produces without touching
|
||||||
the README, leaving a stranger with an install command for a file that does not exist.
|
the README, leaving a stranger with an install command for a file that does not exist.
|
||||||
|
3. ``README.md``'s status paragraph said "no live-model run yet". Measured false as of the F1
|
||||||
|
review (2026-08-25): a live run against ``gpt-4.1-mini`` completed 2026-08-14 (see
|
||||||
|
``docs/2026-08-14-fase1b-forste-levende-kjoring.md``) and ended in a correct ``rejected``
|
||||||
|
outcome.
|
||||||
|
4. ``README.md``'s status paragraph said the Claude Agents SDK sibling is "built in parallel".
|
||||||
|
Measured false the same day: ``STATE.md`` records it as parked.
|
||||||
|
|
||||||
Both gates read the source artefacts as RAW TEXT, because that is the only thing that can see prose.
|
Most gates read the source artefacts as RAW TEXT, because that is the only thing that can see
|
||||||
Both are LINE-ANCHORED rather than substring-matched: ``backends.py`` NAMES ``DefaultAzureCredential``
|
prose. Gates 1 and 2 are LINE-ANCHORED rather than substring-matched: ``backends.py`` NAMES
|
||||||
four times in the comments that explain why it is not used, so a whole-file substring check would be
|
``DefaultAzureCredential`` four times in the comments that explain why it is not used, so a
|
||||||
red on exactly the prose it protects (this repo's 08-09 defect class, and the reason the handover
|
whole-file substring check would be red on exactly the prose it protects (this repo's 08-09 defect
|
||||||
package's python-only gate matches archive member NAMES rather than document prose).
|
class, and the reason the handover package's python-only gate matches archive member NAMES rather
|
||||||
|
than document prose). Gates 3 and 4 are literal regression guards on the specific wording measured
|
||||||
|
false — ``STATE.md`` is local-only (gitignored, not shipped to a fresh clone) and cannot serve as a
|
||||||
|
portable ground-truth source for gate 4, so that gate cannot verify the CURRENT sibling status,
|
||||||
|
only refuse the specific false claim already made once.
|
||||||
|
|
||||||
Each positive assertion is paired with a CONTROL that the thing being searched for is actually
|
Each positive assertion is paired with a CONTROL that the thing being searched for is actually
|
||||||
present. An extractor that silently finds nothing makes a gate that can only ever be green, which
|
present. An extractor that silently finds nothing makes a gate that can only ever be green, which
|
||||||
|
|
@ -34,6 +44,7 @@ _ENV_TEMPLATE = _REPO_ROOT / "env.template"
|
||||||
_README = _REPO_ROOT / "README.md"
|
_README = _REPO_ROOT / "README.md"
|
||||||
_PYPROJECT = _REPO_ROOT / "pyproject.toml"
|
_PYPROJECT = _REPO_ROOT / "pyproject.toml"
|
||||||
_BACKENDS = _REPO_ROOT / "src" / "portfolio_optimiser" / "backends.py"
|
_BACKENDS = _REPO_ROOT / "src" / "portfolio_optimiser" / "backends.py"
|
||||||
|
_LIVE_RUN_EVIDENCE = _REPO_ROOT / "docs" / "2026-08-14-fase1b-forste-levende-kjoring.md"
|
||||||
|
|
||||||
# The credential is chosen on ONE assignment statement. Reading the credential names off that line —
|
# The credential is chosen on ONE assignment statement. Reading the credential names off that line —
|
||||||
# rather than off the whole module — is what keeps the explanatory comments out of the measurement.
|
# rather than off the whole module — is what keeps the explanatory comments out of the measurement.
|
||||||
|
|
@ -43,6 +54,9 @@ _CREDENTIAL_CALL = re.compile(r"(\w*Credential)\s*\(")
|
||||||
# A wheel filename spells the distribution, the version and the tags. The version is the drifting part.
|
# A wheel filename spells the distribution, the version and the tags. The version is the drifting part.
|
||||||
_WHEEL_FILENAME = re.compile(r"portfolio_optimiser-(?P<version>[0-9][^-\s]*)-py3-none-any\.whl")
|
_WHEEL_FILENAME = re.compile(r"portfolio_optimiser-(?P<version>[0-9][^-\s]*)-py3-none-any\.whl")
|
||||||
|
|
||||||
|
# The completed-live-run record on the evidence doc, tolerant of the exact column width used there.
|
||||||
|
_LIVE_RUN_OUTCOME = re.compile(r"outcome_type:\s+rejected")
|
||||||
|
|
||||||
# `[project]`'s own version line: the value hatchling stamps into the wheel filename.
|
# `[project]`'s own version line: the value hatchling stamps into the wheel filename.
|
||||||
_PROJECT_VERSION = re.compile(r'^version\s*=\s*"(?P<version>[^"]+)"', re.MULTILINE)
|
_PROJECT_VERSION = re.compile(r'^version\s*=\s*"(?P<version>[^"]+)"', re.MULTILINE)
|
||||||
|
|
||||||
|
|
@ -133,7 +147,56 @@ def test_readme_wheel_command_cites_the_built_version() -> None:
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("artefact", [_ENV_TEMPLATE, _README, _PYPROJECT, _BACKENDS])
|
# --------------------------------------------------------------------------------------------
|
||||||
|
# F1 (2026-08-25) — the README status paragraph does not repeat two claims measured false
|
||||||
|
# --------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_live_run_evidence_records_a_completed_run() -> None:
|
||||||
|
"""CONTROL. The negative assertion below is only meaningful while this run is on record."""
|
||||||
|
text = _LIVE_RUN_EVIDENCE.read_text(encoding="utf-8")
|
||||||
|
assert _LIVE_RUN_OUTCOME.search(text), (
|
||||||
|
f"{_LIVE_RUN_EVIDENCE} no longer records a completed live run with outcome_type: "
|
||||||
|
"rejected — the claim the status paragraph must not contradict has nothing to guard it"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _readme_prose() -> str:
|
||||||
|
"""``README.md``'s status paragraph is a markdown blockquote wrapped across several lines,
|
||||||
|
each continued with ``"> "``. Rendered markdown collapses those soft line breaks into one
|
||||||
|
paragraph, so a raw substring search must normalise the same way first — otherwise a claim
|
||||||
|
that happens to wrap across a line boundary is invisible to the gate (measured: "no
|
||||||
|
live-model run yet" wraps as "no live-model\\n> run yet" in the raw file).
|
||||||
|
"""
|
||||||
|
text = _README.read_text(encoding="utf-8")
|
||||||
|
return re.sub(r"\n>\s*", " ", text)
|
||||||
|
|
||||||
|
|
||||||
|
def test_readme_status_does_not_claim_no_live_run() -> None:
|
||||||
|
"""A live run against a real endpoint completed 2026-08-14 (see ``_LIVE_RUN_EVIDENCE``); the
|
||||||
|
status paragraph may not say otherwise, even though no run has yet produced a *validated*
|
||||||
|
proposal against a live model.
|
||||||
|
"""
|
||||||
|
assert "no live-model run yet" not in _readme_prose(), (
|
||||||
|
"README.md claims no live-model run has happened, but one completed on 2026-08-14 "
|
||||||
|
f"(see {_LIVE_RUN_EVIDENCE.name})"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_readme_status_does_not_claim_sibling_built_in_parallel() -> None:
|
||||||
|
"""The Claude Agents SDK sibling is parked (``STATE.md``: 'po-claude PARKERT'); the status
|
||||||
|
paragraph may not claim active parallel development. ``STATE.md`` is local-only and is not
|
||||||
|
read here — this is a literal regression guard on the specific claim measured false on
|
||||||
|
2026-08-25, not a check against the sibling's current status.
|
||||||
|
"""
|
||||||
|
assert "built in parallel" not in _readme_prose(), (
|
||||||
|
"README.md claims the Claude Agents SDK sibling is built in parallel, but it is parked"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"artefact", [_ENV_TEMPLATE, _README, _PYPROJECT, _BACKENDS, _LIVE_RUN_EVIDENCE]
|
||||||
|
)
|
||||||
def test_guarded_artefacts_exist(artefact: Path) -> None:
|
def test_guarded_artefacts_exist(artefact: Path) -> None:
|
||||||
"""CONTROL. A missing artefact must fail here rather than turn a gate into a no-op."""
|
"""CONTROL. A missing artefact must fail here rather than turn a gate into a no-op."""
|
||||||
assert artefact.is_file(), f"{artefact} is missing; the gates above would read nothing"
|
assert artefact.is_file(), f"{artefact} is missing; the gates above would read nothing"
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue