feat(ms-ai-architect): add plugin to open marketplace (v1.5.0 baseline)

Initial addition of ms-ai-architect plugin to the open-source marketplace.
Private content excluded: orchestrator/ (Linear tooling), docs/utredning/
(client investigation), generated test reports and PDF export script.
skill-gen tooling moved from orchestrator/ to scripts/skill-gen/.

Security scan: WARNING (risk 20/100) — no secrets, no injection found.
False positive fixed: added gitleaks:allow to Python variable reference
in output-validation-grounding-verification.md line 109.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-04-07 17:17:17 +02:00
commit 6a7632146e
490 changed files with 213249 additions and 2 deletions

View file

@ -0,0 +1,41 @@
#!/bin/bash
# capture-fixture.sh — Extract agent output fixture from a document
# Usage: bash tests/capture-fixture.sh <source-file> <section-header> <output-dir>
#
# Example:
# bash tests/capture-fixture.sh docs/utredning/.../utredning.md "S5: Sikkerhetsvurdering" tests/fixtures/security-assessment/
set -euo pipefail
if [ $# -lt 3 ]; then
echo "Usage: $0 <source-file> <section-header> <output-dir>"
echo ""
echo "Extracts a section from source file and saves as fixture.md"
echo ""
echo "Example:"
echo " $0 docs/utredning/file.md 'S5: Sikkerhetsvurdering' tests/fixtures/security-assessment/"
exit 1
fi
SOURCE="$1"
HEADER="$2"
OUTPUT_DIR="$3"
if [ ! -f "$SOURCE" ]; then
echo "Error: Source file not found: $SOURCE"
exit 1
fi
mkdir -p "$OUTPUT_DIR"
# Extract section from ## header to next ## header (or EOF)
awk -v header="$HEADER" '
BEGIN { found=0 }
/^## / {
if (found) exit
if (index($0, header)) found=1
}
found { print }
' "$SOURCE" > "$OUTPUT_DIR/fixture.md"
LINES=$(wc -l < "$OUTPUT_DIR/fixture.md" | tr -d ' ')
echo "Captured $LINES lines to $OUTPUT_DIR/fixture.md"

View file

@ -0,0 +1,93 @@
## EU AI Act — Vurdering: AutomatiskSaksbehandler
**Dato:** 2026-02-22
**Vurdert av:** AI Act Assessor
**Organisasjon:** Statens vegvesen
### 1. Risikoklassifisering
| Attributt | Verdi |
|-----------|-------|
| **Risikonivå** | Høyrisiko |
| **Annex III-kategori** | Kategori 5: Tilgang til og bruk av essensielle offentlige tjenester og ytelser |
| **GPAI-status** | Ja — basert på GPT-4o via Azure OpenAI |
| **Klassifiseringsgrunnlag** | Systemet automatiserer vurdering av helsekrav ved søknad om førerkort (klasse B). Direkte påvirkning på borgeres rett til førerkort — en essensiell offentlig tjeneste. |
| **Konfidens** | Høy |
#### Steg 1: Forbudt-sjekk (Art. 5)
Ingen forbudte praksiser identifisert. Systemet scorer ikke individer sosialt, og beslutninger kan overprøves av saksbehandler.
#### Steg 2: Annex III høyrisiko-sjekk
**Treffer kategori 5 (a):** AI-systemer som brukes av offentlige myndigheter for å vurdere berettigelse til offentlige ytelser og tjenester, inkludert tildelingsbeslutninger.
Førerkort er en essensiell offentlig tjeneste i norsk kontekst. Automatisert vurdering av helsekrav påvirker direkte borgeres tilgang til denne tjenesten.
**Grensevurdering:** Det er ingen tvil om at dette er høyrisiko. Systemet tar beslutninger som direkte påvirker enkeltpersoners rettigheter og muligheter.
#### Steg 3: GPAI-sjekk
GPT-4o er en GPAI-modell. Microsoft er provider av grunnmodellen. Statens vegvesen er deployer av det tilpassede systemet. Ettersom systemet har vesentlig tilpasning (fine-tuning på norske helseattest-vurderinger), kan Statens vegvesen også anses som provider av det tilpassede høyrisiko-systemet.
#### Steg 4: Ikke relevant — allerede klassifisert som høyrisiko
### 2. Rolle
| Attributt | Verdi |
|-----------|-------|
| **Organisasjonens rolle** | Deployer (primært) + mulig Provider (ved vesentlig tilpasning) |
| **Begrunnelse** | Som deployer har Statens vegvesen alle Art. 26-27 forpliktelser. Ved fine-tuning av modellen kan organisasjonen også få provider-forpliktelser for det tilpassede systemet (Art. 25). |
| **Provider (grunnmodell)** | Microsoft (Azure OpenAI Service) |
### 3. Forpliktelser
| # | Artikkel | Krav | Status | Gap |
|---|----------|------|--------|-----|
| 1 | Art. 26(1) | Bruk i samsvar med bruksanvisning | Delvis | Bruksanvisning fra Microsoft, men ikke tilpasset norsk kontekst |
| 2 | Art. 26(2) | Menneskelig tilsyn (effektiv kontroll) | Delvis | Saksbehandler kan overprøve, men prosedyre ikke formalisert |
| 3 | Art. 26(5) | FRIA gjennomført for offentlig sektor | Ikke oppfylt | Ingen FRIA utført |
| 4 | Art. 26(6) | Loggoppbevaring minimum 6 måneder | Ikke oppfylt | Logger settes til 90 dager i Application Insights |
| 5 | Art. 27 | FRIA for offentlig myndighet-deployer | Ikke oppfylt | Obligatorisk — ikke startet |
| 6 | Art. 13 | Transparens: bruksinstruksjon tilgjengelig | Ikke oppfylt | Ingen Art. 13-dokumentasjon |
| 7 | Art. 14 | Menneskelig tilsyn: override-mekanismer | Delvis | Override mulig men ikke systematisk |
| 8 | Art. 50(1) | Informer personer om AI-bruk | Ikke oppfylt | Borgere informeres ikke om at AI vurderer helseattester |
| 9 | Art. 9 | Risikostyringssystem | Ikke oppfylt | Ingen formell risikostyring for AI-systemet |
| 10 | Art. 12 | Automatisk loggføring | Delvis | Logger finnes men retention er for kort |
### 4. Tiltaksplan
| # | Tiltak | Prioritet | Frist | Ansvarlig |
|---|--------|-----------|-------|-----------|
| T1 | Gjennomfør FRIA (Art. 27) — bruk `/architect:frimpact` | Kritisk | 2026-05-01 | Personvernombud + AI-rådgiver |
| T2 | Etabler risikostyringssystem (Art. 9) | Kritisk | 2026-06-01 | Seksjonsleder |
| T3 | Øk log retention til minimum 6 måneder (Art. 12/26) | Kritisk | 2026-04-01 | IT-drift |
| T4 | Utvikle transparensnotis til borgere (Art. 50) | Høy | 2026-05-01 | Kommunikasjonsavdeling |
| T5 | Formalisér override-prosedyre for saksbehandlere (Art. 14) | Høy | 2026-05-15 | Fagleder |
| T6 | Gjennomfør DPIA (GDPR Art. 35) — overlapper med FRIA | Høy | 2026-05-01 | Personvernombud |
| T7 | Utarbeid Art. 13 bruksinstruksjon | Middels | 2026-06-15 | AI-rådgiver |
| T8 | Forbered samsvarsvurdering (Annex IV, Art. 43) | Middels | 2026-07-01 | Kvalitetsansvarlig |
| T9 | Vurdér behov for ekstern samsvarsvurdering | Lav | 2026-07-15 | Juridisk avdeling |
### 5. Neste steg
1. **Umiddelbart:** `/architect:frimpact` — FRIA er obligatorisk og bør prioriteres høyest
2. **Innen 30 dager:** `/architect:dpia` — Personvernkonsekvensanalyse (utdyper personverndimensjonen)
3. **Innen 60 dager:** `/architect:ros` — ROS-analyse med AI Act-dimensjon (dimensjon 6)
4. **Innen 90 dager:** `/architect:conformity` — Start samsvarsvurdering
5. **Dokumentér:** `/architect:adr` — Dokumenter klassifiseringsbeslutningen
### Viktige frister
| Frist | Krav | Relevans |
|-------|------|----------|
| 2025-02-02 | Forbudte AI-praksiser (Art. 5) | Gjelder ikke |
| 2025-08-02 | Governance og sanksjoner (Art. 99) | Gjelder — governance-struktur kreves |
| 2026-08-02 | GPAI-krav + Annex III høyrisiko | **Gjelder direkte — 161 dager** |
| 2027-08-02 | Alle høyrisiko-krav (full compliance) | Gjelder — full Art. 9-27 compliance |
### Referanser
- `ai-act-classification-methodology.md` — Klassifiseringsmetodikk
- `ai-act-annex-iii-checklist.md` — Annex III kategori 5 vurdering
- `ai-act-deployer-obligations.md` — Art. 26-27 forpliktelser
- `ai-act-fria-template.md` — FRIA-mal referanse
- `ai-act-provider-obligations.md` — Art. 9-15 (ved provider-status)
- `ai-act-compliance-guide.md` — Generell veileder
- Microsoft Learn: Azure OpenAI responsible AI practices

View file

@ -0,0 +1,79 @@
## EU AI Act — Vurdering: FartsPrediksjonsagent
**Dato:** 2026-02-22
**Vurdert av:** AI Act Assessor
**Organisasjon:** Statens vegvesen
### 1. Risikoklassifisering
| Attributt | Verdi |
|-----------|-------|
| **Risikonivå** | Minimal risiko |
| **Annex III-kategori** | Ikke Annex III |
| **GPAI-status** | Ja — basert på GPT-4o, men ikke systemisk risiko |
| **Klassifiseringsgrunnlag** | Systemet predikerer gjennomsnittsfart på vegstrekninger basert på historiske trafikkdata. Ingen direkte påvirkning på individer, ingen biometrisk identifikasjon, ikke kritisk infrastrukturstyring. |
| **Konfidens** | Høy |
#### Steg 1: Forbudt-sjekk (Art. 5)
Ingen av de forbudte praksisene er relevante. Systemet scorer ikke individer sosialt, manipulerer ikke sårbare grupper, og bruker ikke biometrisk fjernidentifisering.
#### Steg 2: Annex III høyrisiko-sjekk
Systemet treffer ingen av de 8 Annex III-kategoriene:
- Ikke biometrisk identifisering
- Ikke styring av kritisk infrastruktur (predikerer, styrer ikke)
- Ikke utdanning/opplæring
- Ikke ansettelse/personal
- Ikke essensielle offentlige tjenester
- Ikke rettshåndhevelse
- Ikke migrasjon/grensekontroll
- Ikke rettsforvaltning
#### Steg 3: GPAI-sjekk
Systemet bruker Azure OpenAI GPT-4o som grunnmodell. GPT-4o er en GPAI-modell, men FartsPrediksjonsagent er en downstream-applikasjon — provider-forpliktelser for GPAI hviler på Microsoft som modell-provider.
#### Steg 4: Begrenset/Minimal
Systemet har ingen direkte brukerinteraksjon med borgere. Resultater vises kun til trafikkplanleggere internt. Klassifiseres som **minimal risiko**.
### 2. Rolle
| Attributt | Verdi |
|-----------|-------|
| **Organisasjonens rolle** | Deployer |
| **Begrunnelse** | Statens vegvesen bruker et AI-system utviklet internt med Azure OpenAI. Ettersom systemet ikke markedsføres til andre, og bruker standard Azure-tjenester uten vesentlig tilpasning av modellen, er rollen deployer. |
| **Provider (ekstern)** | Microsoft (Azure OpenAI Service) |
### 3. Forpliktelser
| # | Artikkel | Krav | Status | Gap |
|---|----------|------|--------|-----|
| 1 | Art. 50(1) | Transparens: informer brukere om AI-bruk | Oppfylt | Interne brukere informert |
| 2 | Art. 4 | AI-kompetanse: sikre tilstrekkelig kunnskap | Delvis | Opplæringsplan ikke formalisert |
| 3 | Frivillig | Code of Conduct (Art. 95) | Ikke startet | Anbefales men ikke påkrevd |
### 4. Tiltaksplan
| # | Tiltak | Prioritet | Frist | Ansvarlig |
|---|--------|-----------|-------|-----------|
| T1 | Formalisér AI-kompetanseplan for trafikkplanleggere | Lav | 2026-12-31 | Seksjonsleder |
| T2 | Vurdér frivillig Code of Conduct-tilslutning | Lav | 2027-06-30 | AI-rådgiver |
### 5. Neste steg
1. Ingen regulatoriske blokkeringer — systemet kan brukes uten ytterligere tiltak
2. Anbefaler `/architect:ros` for generell risikovurdering (god praksis)
3. Vurdér `/architect:transparency` for å generere intern AI-bruksnotis
### Viktige frister
| Frist | Krav | Relevans |
|-------|------|----------|
| 2025-02-02 | Forbudte AI-praksiser (Art. 5) | Gjelder ikke |
| 2025-08-02 | Governance og sanksjoner (Art. 99) | Gjelder ikke direkte |
| 2026-08-02 | GPAI-krav + Annex III høyrisiko | Gjelder ikke (minimal risiko) |
| 2027-08-02 | Alle høyrisiko-krav (full compliance) | Gjelder ikke |
### Referanser
- `ai-act-classification-methodology.md` — Klassifiseringsmetodikk (steg 1-4)
- `ai-act-compliance-guide.md` — Generell veileder
- `ai-act-annex-iii-checklist.md` — Annex III-sjekkliste
- Microsoft Learn: Azure OpenAI EU Data Boundary compliance

View file

@ -0,0 +1,58 @@
## S6: Kostnadsvurdering
### S6.1: TCO per alternativ (3 år)
| Kostnadspost | Alt 0 | Alt 1 | Alt 2 | Alt 2B | Alt 3 |
|-------------|-------|-------|-------|--------|-------|
| **Etablering** | 0 | 0 | 800K | 2 035K | 3 500K |
| Prosjektkostnader | 0 | 0 | 200K | 500K | 800K |
| Utvikling (interne) | 0 | 0 | 400K | 885K | 1 500K |
| QA-konsulent | 0 | 0 | 0 | 200K | 400K |
| Opplæring | 0 | 0 | 100K | 300K | 300K |
| Buffer | 0 | 0 | 100K | 150K | 500K |
| **Årlig drift** | 0 | 0 | 600K | 1 350K | 1 800K |
| Lisenser (Copilot Studio) | 0 | 0 | 230K | 230K | 0 |
| Azure OpenAI (tokens) | 0 | 0 | 120K | 210K | 210K |
| Azure AI Search | 0 | 0 | 110K | 200K | 200K |
| Infrastruktur (øvrig) | 0 | 0 | 50K | 120K | 300K |
| Drift/vedlikehold | 0 | 0 | 50K | 150K | 350K |
| Overvåking | 0 | 0 | 20K | 80K | 100K |
| Embedding-refresh | 0 | 0 | 20K | 60K | 60K |
| Regional støtte/opplæring | 0 | 0 | 0 | 150K | 150K |
| Risk buffer (drift) | 0 | 0 | 0 | 150K | 230K |
| **3-års TCO** | **0** | **0** | **2 000K** | **6 335K** | **9 100K** |
Alle beløp i NOK. P50-estimat. Valutakurs: 11 NOK/USD.
### S6.2: Konfidensgradering
| Kostnadspost | Konfidens | Kilde |
|-------------|-----------|-------|
| Azure OpenAI token-priser | 🟢 Høy | MCP-verifisert (microsoft-learn) |
| Azure AI Search S1 | 🟢 Høy | MCP-verifisert |
| Copilot Studio capacity | 🟡 Middels | Fra kunnskapsbase (kan endre seg med ny prismodell) |
| Intern utviklerkostnad | 🟡 Middels | Estimert 700 NOK/time, 80% dedikasjon |
| Gevinstrealisering | 🔴 Lav | Antatt, basert på generelle produktivitetsestimater |
### S6.3: Gevinstrealisering (justert 30-50% realisering)
| År | Brutto gevinst | Realiserings-grad | Netto gevinst | AI-kostnad | Netto |
|----|---------------|-------------------|---------------|------------|-------|
| År 1 | 77M | 20% | 15.4M | 2.035M | +13.4M |
| År 2 | 77M | 40% | 30.8M | 1.35M | +29.5M |
| År 3 | 77M | 50% | 38.5M | 1.35M | +37.2M |
**NNV (3 år, 4% diskonteringsrente):** ~+80M NOK (konservativt estimat)
**Tilbakebetalingstid:** ~2 måneder (Fase 1-drift)
> **Merknad:** Gevinstestimatene er konservative (30-50% realisering vs. teoretisk 100%). Faktisk realisering avhenger av adopsjon, datakvalitet og endringsledelse. Anbefaler gevinstmåling fra måned 3.
### S6.4: Copilot Studio-lisensstrategi
Start med 1 capacity pack (25K meldinger/mnd). Skaler basert på faktisk bruk:
- MVP-pilotfase: 1 capacity pack (~230K NOK/år)
- Full utrulling: 2-3 capacity packs basert på bruksvolumet
- Overvåk via Copilot Studio Analytics og Azure Monitor
---

View file

@ -0,0 +1,63 @@
## S5: Sikkerhetsvurdering
### S5.1: Sikkerhetsscoring
**Totalscore: 2.80/5.00 — BETINGET AKSEPTABEL**
| Dimensjon | Score | Status | Viktigste funn |
|-----------|-------|--------|----------------|
| Identity & Access | 3.5/5 | 🟡 Adekvat | Entra ID + Managed Identity. Mangler: PIM for admin, break-glass prosedyrer. |
| Network Security | 2.5/5 | 🟡 Adekvat | VNet-integrasjon planlagt. Mangler: Private Endpoints for alle endepunkter. |
| Data Protection | 2.0/5 | 🔴 Svak | Sweden Central for Azure OpenAI. Presidio PII planlagt. Mangler: CMK, DLP-policyer. |
| Content Safety | 3.5/5 | 🟡 Adekvat | Content Safety + Prompt Shields planlagt. Mangler: norskspesifikke policyer, Groundedness Detection tuning. |
| Compliance & Governance | 2.5/5 | 🟡 Adekvat | AI Act-klassifisering utført. Mangler: DPIA, AI-register, Schrems II TIA. |
| Monitoring & Response | 2.8/5 | 🟡 Adekvat | Application Insights + Sentinel planlagt. Mangler: AI-spesifikke deteksjonsregler, incident response-plan. |
### Kritiske funn
**P0 (Blokkerende):**
1. **DPIA ikke gjennomført** — Obligatorisk (GDPR art. 35) pga. PII i avviksrapporter
2. **Schrems II TIA mangler** — Azure OpenAI i Sweden Central krever formell risikovurdering
3. **Incident Response-plan mangler** — Ingen prosedyre for AI-spesifikke hendelser (prompt injection breach, PII-lekkasje)
**P1 (Høy prioritet):**
4. **Red team-testing av security trimming** — Må verifisere at regionbasert tilgangskontroll fungerer korrekt
5. **AI-spesifikk alerting mangler** — Ingen varsling for anomalier i prompt-mønstre eller uvanlig token-forbruk
6. **PIM ikke konfigurert** — Admin-tilgang til AI-ressurser bør styres via Privileged Identity Management
### S5.2: DPIA-status
| Spørsmål | Svar |
|----------|------|
| Behandles personopplysninger? | Ja — PII i avviksrapporter, chatlogger |
| Er DPIA påkrevd? | Ja (GDPR art. 35) |
| Er DPIA gjennomført? | ⬜ Ikke startet |
| Personvernombud involvert? | Planlagt for Fase 1 |
| Konsultasjon med Datatilsynet? | Vurderes etter DPIA |
**Tidsestimat:** 1-2 uker for gjennomføring, inkl. personvernombuds review.
### S5.3: ROS-analyse
| # | Risiko | S | K | Nivå | Tiltak | Restrisiko |
|---|--------|---|---|------|--------|------------|
| 1 | Hallusinering i faglige svar | 4 | 3 | HØY | Groundedness detection 0.8, kildehenvisninger, HITL-disclaimer | MIDDELS |
| 2 | PII-lekkasje via AI-svar | 2 | 4 | HØY | Presidio pre-indexing + runtime PII-filter + security trimming | LAV |
| 3 | Prompt injection | 3 | 3 | MIDDELS | Prompt Shields + Content Safety + system message-hardening | LAV |
| 4 | Uautorisert dokumenttilgang | 2 | 4 | MIDDELS | Security trimming med SharePoint ACL-mapping + Entra ID | LAV |
| 5 | Schrems II — data residency | 3 | 3 | MIDDELS | Formell TIA + risikoaksept. Data-at-rest i Norway East. | MIDDELS |
| 6 | Modell-degradering over tid | 2 | 2 | LAV | Kvartalsvis eval + drift-monitoring + automatisk rollback | LAV |
### S5.4: Dataklassifisering
| Datatype | Klassifisering | Behandlingsgrunnlag | Lagringssted |
|----------|----------------|---------------------|--------------|
| Håndbøker (N-serien) | Intern | Berettiget interesse | Azure AI Search (Norway East) |
| Kontrakter | Fortrolig | Berettiget interesse | Azure AI Search (Norway East), security trimmed |
| Inspeksjonsrapporter | Intern | Berettiget interesse | Azure AI Search (Norway East) |
| Avviksrapporter (PII) | Fortrolig | Samtykke/Lovhjemmel | Azure AI Search (Norway East), PII-maskert |
| Chatlogger | Intern | Berettiget interesse | Application Insights (Sweden Central), 90d retention |
| NVDB-data | Åpen | Åpne data | Ikke lagret — live API-oppslag |
---

View file

@ -0,0 +1,98 @@
## S1: Sammendrag
### 1.1 Teknisk sammendrag
Denne utredningen vurderer innføring av en AI-assistert kunnskapssøk-løsning for drift- og vedlikeholdsavdelinger i en norsk statlig veietat med ~1500 ansatte fordelt på 5 regioner. Løsningen skal erstatte dagens fragmenterte, manuelle dokumentsøk på tvers av 65 000+ dokumenter i 5 separate systemer (SharePoint, Doculive, Landbruks-IT, lokale filservere, NVDB).
**Anbefalt plattform:** Alternativ 2B — Hybrid (Copilot Studio + Azure AI Foundry RAG) (S2.5, S8.1)
Copilot Studio fungerer som Teams-native UI-lag («Veihjelper AI»), mens Azure AI Foundry gir full kontroll over RAG-pipeline med custom chunking, Presidio PII-filtrering og SharePoint ACL-basert security trimming. Multi-modell-strategi med GPT-4o for komplekse fagspørsmål og GPT-4o-mini for enkel gjenfinning (80/20 cost routing) sikrer kostnadseffektivitet (S4.2). Embedding med text-embedding-3-large (3072 dim) i Azure AI Search S1 med hybrid search og semantic reranking gir best ytelse for norsk fagterminologi (S4.3).
**Arkitektur (S8.2):** Brukerinteraksjon via Teams (mobil/desktop) → Entra ID-autentisering → Copilot Studio Agent → Azure AI Foundry RAG-pipeline (Azure OpenAI GPT-4o/mini i Sweden Central, Azure AI Search S1 i Norway East) + NVDB REST API via function calling. Presidio PII-filter (pre-indexing) og Azure AI Content Safety (runtime) sikrer personvern og innholdssikkerhet.
**Sikkerhetsstatus (S5):** Totalscore 2.80/5.00 — BETINGET AKSEPTABEL. Tre P0-blokkere er identifisert: (1) DPIA ikke gjennomført (GDPR art. 35), (2) Schrems II TIA mangler for Azure OpenAI i Sweden Central, (3) Incident Response-plan mangler. Alle tre er innarbeidet som obligatoriske aktiviteter i Fase 0 av implementeringsplanen (S9.4). AI Act-klassifisering: Begrenset risiko — transparenskrav oppfylt med AI-merking og kildehenvisninger (S4.1).
**Kostnad (S6):** Etableringskostnad 2.035M NOK (Fase 1, innenfor 2M-budsjettramme). Årlig driftskostnad 1.35M NOK inkludert Copilot Studio capacity pack (230K), Azure OpenAI tokens (210K), Azure AI Search (200K), infrastruktur (120K), drift/vedlikehold (150K), overvåking (80K), embedding-refresh (60K), regional støtte (150K) og risk buffer (150K). 3-års TCO: 6.335M NOK. Fase 2 (Doculive + Landbruks-IT) budsjetteres separat til 850K.
**Implementeringsplan (S9):** Fasevis utrulling over 48 uker. Fase 0 (uke 1-4): Forberedelse, DPIA, Schrems II TIA, Azure OpenAI-søknad. Fase 1 (uke 5-32): MVP med SharePoint (39K dok) + NVDB, inkludert infrastruktur, RAG-pipeline, Copilot Studio agent, sikkerhetstesting, pilot med 50 brukere, opplæring (5 regioner) og region-for-region utrulling. Fase 2 (uke 33-48): Full dekning med Doculive (13K) og Landbruks-IT (10K). 8 milepæler definert (M0-M8), inkludert gevinstevaluering i uke 56 (S9.2).
**Arkitekturprinsipper (S3):** 3 av 7 Digdir-prinsipper fullt oppfylt, 4 delvis oppfylt. Hovedavvik: Schrems II-risiko (P4 Tillit), begrenset ekstern datadeling (P2/P7). Trade-off vedtatt: Sweden Central aksepteres med formell risikoaksept da Norway East ikke tilbyr Azure OpenAI.
**Digital samhandling (S7):** Juridisk samhandling er oppfylt (berettiget interesse, DPA med Microsoft, ingen vedtaksfatning). Organisatorisk samhandling er delvis under arbeid (RACI-matrise, AI-styringsgruppe). Semantisk samhandling er oppfylt (OData, Dublin Core, standardisert embedding). Teknisk samhandling er oppfylt (REST API, OAuth 2.0, SLA 99.8%). Styring planlagt med kvartalsvis AI-styringsgruppe og definerte KPI-er.
5 ADR-er er utarbeidet (S8.3): Copilot Studio som UI (ADR-001), custom RAG i Azure AI Foundry (ADR-002), Sweden Central med risikoaksept (ADR-003), batch-import fra Landbruks-IT (ADR-004), og Security Trimming med SharePoint ACL-mapping (ADR-005).
---
### 1.2 Beslutningsnotat (Executive Summary)
**Anbefaling:** Iverksett Alternativ 2B — Hybrid Copilot Studio + Azure AI Foundry RAG for AI-assistert dokumentsøk i drift- og vedlikeholdsavdelingene.
**Bakgrunn:** Driftspersonell bruker 3-5 timer per uke på manuelt dokumentsøk i 5 ulike systemer. Produktivitetstapet er estimert til ~77M NOK/år for hele organisasjonen. Kunnskapen er fragmentert, eksisterende søk forstår ikke norsk vei-fagterminologi, og det finnes ingen felles søkeflate.
**Hva vi anbefaler:** En Teams-basert AI-assistent («Veihjelper AI») som gir umiddelbar tilgang til hele dokumentkorpuset (65 000+ dokumenter) med norskspråklig semantisk søk, kildehenvisninger og sanntids veidata fra NVDB. Løsningen bruker Microsofts AI-plattform med full kontroll over sikkerhet og personvern.
**Hvorfor dette alternativet:** Alt 2B gir den beste balansen mellom brukeropplevelse (Teams-native), sikkerhet (PII-filtrering, security trimming) og kostnad (innenfor 2M budsjett for Fase 1). Alt 1 (SharePoint AI Search) dekker kun 60% av dokumentene. Alt 2 (innebygd RAG) mangler kontroll over PII og security trimming. Alt 3 (full custom) er overingeniørt og sprenger budsjettet.
**Investering:** 2.035M NOK etableringskostnad (Fase 1), 1.35M NOK årlig drift. 3-års TCO: 6.335M NOK. Fase 2 (full dokumentdekning): 850K NOK separat.
**Forventet gevinst:** Med konservativ 30-50% realisering av produktivitetsgevinst gir løsningen en NNV på ~+80M NOK over 3 år. Tilbakebetalingstid: ~2 måneder etter produksjonssetting. Gevinstmåling starter fra måned 3.
**Sikkerhet og compliance:** Sikkerhetsstatus er BETINGET AKSEPTABEL (2.80/5). Tre blokkerende funn må lukkes før produksjon: DPIA, Schrems II TIA og Incident Response-plan. Alle er planlagt gjennomført i Fase 0 (uke 1-4). AI Act: Begrenset risiko — transparenskrav oppfylt.
**Tidsplan:** Fase 0 (forberedelse): 4 uker. Fase 1 (MVP i produksjon): 7-8 måneder. Fase 2 (full dekning): 3-4 måneder. Total: ~12 måneder til full dekning.
**Kritiske forutsetninger:**
1. Azure OpenAI-tilgang i Sweden Central må innvilges
2. DPIA må godkjennes av personvernombud
3. 2 Azure-utviklere må dedikeres minimum 80%
4. Schrems II-risikoaksept må signeres av behandlingsansvarlig
**Risiko:** Hovedrisikoen er hallusinering i faglige svar (HØY), som mitigeres med groundedness detection (terskel 0.8), kildehenvisninger og tydelig AI-disclaimer. PII-lekkasje (HØY) mitigeres med Presidio pre-indexing og runtime PII-filter.
**Anbefaling til ledelsen:** Godkjenn prosjektoppstart med Go/No-Go-beslutning etter Fase 0 (4 uker). Bevilg 2.035M NOK for Fase 1. Utpek prosjekteier med mandat i KI-seksjonen. Igangsett DPIA umiddelbart.
---
### 1.3 Nøkkeltall
| Parameter | Verdi |
|-----------|-------|
| **Anbefalt alternativ** | 2B: Hybrid (Copilot Studio + Azure AI Foundry RAG) |
| **AI Act-risikoklasse** | Begrenset risiko |
| **Kompleksitetsscore** | 14/18 (KOMPLEKS) |
| **Sikkerhetsscore** | 2.80/5.00 (Betinget akseptabel) |
| **P0-blokkere** | 3 (DPIA, Schrems II TIA, Incident Response) |
| **Etableringskostnad (Fase 1)** | 2.035M NOK |
| **Årlig driftskostnad** | 1.35M NOK |
| **3-års TCO** | 6.335M NOK |
| **Fase 2-kostnad** | 850K NOK (separat) |
| **NNV (3 år, 4% diskontering)** | ~+80M NOK (konservativt) |
| **Tilbakebetalingstid** | ~2 måneder (etter Fase 1-drift) |
| **Brutto produktivitetsgevinst** | ~77M NOK/år |
| **Gevinstraliseringsgrad** | 30-50% (konservativt) |
| **Dokumentkorpus** | 65 000+ dokumenter |
| **Datakilder** | 5 (SharePoint, Doculive, Landbruks-IT, filservere, NVDB) |
| **Primærbrukere** | ~750 (potensielt 1500) |
| **Time-to-value (MVP)** | 7-8 måneder (Fase 1) |
| **Full dekning** | 12 måneder (Fase 1 + 2) |
| **AI-modeller** | GPT-4o + GPT-4o-mini, text-embedding-3-large |
| **Azure-regioner** | Sweden Central (OpenAI), Norway East (AI Search) |
| **SLA** | ~99.8% (samlet) |
| **ADR-er** | 5 (alle Accepted) |
| **Digdir-prinsipper** | 3/7 fullt oppfylt, 4/7 delvis |
| **Antall milepæler** | 8 (M0-M8, uke 1-56) |
---
### 1.4 Konfidenstabell
| Dimensjon | Konfidens | Begrunnelse |
|-----------|-----------|-------------|
| Teknisk gjennomførbarhet | 🟢 Høy | RAG-teknologi er veletablert (Azure AI Search GA siden 2014, Azure OpenAI GA siden 2023). Copilot Studio GA med AI-kapabiliteter. 2 interne Azure-utviklere finnes. |
| Kostnadsestimat | 🟡 Middels | Azure-priser MCP-verifisert (høy). Intern utviklerkostnad estimert (middels). Gevinstestimater konservative men usikre (lav). P50 med 150K buffer. |
| Regulatorisk compliance | 🟡 Middels | AI Act avklart (Begrenset risiko). Schrems II mitigerbar med TIA. DPIA ikke gjennomført — blokkerende men standard prosess (1-2 uker). |
| Organisatorisk gjennomførbarhet | 🟡 Middels | Kompetansegap mitigeres med 300K opplæring + 200K QA-konsulent. Avhenger av 80% dedikering fra 2 utviklere. Endringsledelse budsjettert med superbrukermodell. |
---

View file

@ -0,0 +1,187 @@
#!/bin/bash
# e2e-helpers.sh — Shared validation functions for E2E regression tests
set -euo pipefail
# Counters
PASS=0
FAIL=0
WARN=0
SUITE_NAME=""
# Colors
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m'
init_suite() {
SUITE_NAME="$1"
PASS=0
FAIL=0
WARN=0
echo -e "${BLUE}═══════════════════════════════════════${NC}"
echo -e "${BLUE} E2E Suite: $SUITE_NAME${NC}"
echo -e "${BLUE}═══════════════════════════════════════${NC}"
echo ""
}
pass() {
PASS=$((PASS + 1))
echo -e " ${GREEN}PASS${NC} $1"
}
fail() {
FAIL=$((FAIL + 1))
echo -e " ${RED}FAIL${NC} $1"
}
warn() {
WARN=$((WARN + 1))
echo -e " ${YELLOW}WARN${NC} $1"
}
print_summary() {
local total=$((PASS + FAIL))
echo ""
echo -e "${BLUE}───────────────────────────────────────${NC}"
echo -e " ${SUITE_NAME}: ${PASS}/${total} PASS, ${FAIL} FAIL, ${WARN} WARN"
echo -e "${BLUE}───────────────────────────────────────${NC}"
echo ""
return $FAIL
}
# --- Assertions ---
assert_has_section() {
local file="$1"
local header="$2"
local description="${3:-Section: $header}"
if grep -qE "^#{1,4} .*${header}" "$file"; then
pass "$description"
else
fail "$description — header '$header' not found"
fi
}
assert_min_lines() {
local file="$1"
local min="$2"
local description="${3:-Minimum $min lines}"
local count
count=$(wc -l < "$file" | tr -d ' ')
if [ "$count" -ge "$min" ]; then
pass "$description ($count lines)"
else
fail "$description — only $count lines (min: $min)"
fi
}
assert_encoding_ok() {
local file="$1"
local description="${2:-UTF-8 encoding valid}"
# Check for common broken UTF-8 sequences
if file "$file" | grep -qi "utf-8\|ascii\|unicode"; then
pass "$description"
else
fail "$description — encoding not UTF-8/ASCII"
fi
}
assert_no_ascii_approximation() {
local file="$1"
local description="${2:-No ASCII approximation of Norwegian chars}"
# Check for ae/oe/aa used where æ/ø/å should be (Norwegian words)
# Only flag common Norwegian words that should have æøå
local violations=0
for pattern in '\bvaere\b' '\bfoerste\b' '\bhoey\b' '\bgjoere\b' '\baarlig\b' '\bsaerlig\b' '\bnoedvendig\b'; do
if grep -qiE "$pattern" "$file" 2>/dev/null; then
violations=$((violations + 1))
fi
done
if [ "$violations" -eq 0 ]; then
pass "$description"
else
fail "$description$violations ASCII approximation patterns found"
fi
}
assert_scores_in_range() {
local file="$1"
local description="${2:-Scores in X/5 format within range}"
# Match X/5 or X.X/5 patterns and verify range
local bad_scores=0
while IFS= read -r match; do
local score="${match%%/*}"
score=$(echo "$score" | tr -d ' ')
# Check if it's a valid number between 0 and 5
if echo "$score" | grep -qE '^[0-5](\.[0-9]+)?$'; then
:
else
bad_scores=$((bad_scores + 1))
fi
done < <(grep -oE '[0-9]+\.?[0-9]*/5' "$file" 2>/dev/null || true)
local total_scores
total_scores=$(grep -cE '[0-9]+\.?[0-9]*/5' "$file" 2>/dev/null || echo "0")
if [ "$total_scores" -gt 0 ] && [ "$bad_scores" -eq 0 ]; then
pass "$description ($total_scores scores found)"
elif [ "$total_scores" -eq 0 ]; then
fail "$description — no X/5 scores found"
else
fail "$description$bad_scores out-of-range scores"
fi
}
assert_has_nok_amounts() {
local file="$1"
local min="${2:-1}"
local description="${3:-NOK amounts present}"
local count
count=$(grep -cE '([0-9]+[.,]?[0-9]*(M|K)\s*NOK|[0-9]+\s*NOK|NOK\s*[0-9]+|[0-9]+\s*000\s*NOK)' "$file" 2>/dev/null || echo "0")
if [ "$count" -ge "$min" ]; then
pass "$description ($count found)"
else
fail "$description — only $count NOK amounts (min: $min)"
fi
}
assert_min_tables() {
local file="$1"
local min="$2"
local description="${3:-Minimum $min markdown tables}"
# Count lines starting with | that contain | somewhere after the first char (table rows)
local table_separators
table_separators=$(grep -cE '^\|.*\|.*\|' "$file" 2>/dev/null || echo "0")
# Rough estimate: each table has header + separator + at least 1 row = 3 lines minimum
local estimated_tables=$((table_separators / 3))
if [ "$estimated_tables" -ge "$min" ]; then
pass "$description (~$estimated_tables tables)"
else
fail "$description — only ~$estimated_tables tables (min: $min)"
fi
}
assert_matches_pattern() {
local file="$1"
local pattern="$2"
local description="${3:-Pattern match: $pattern}"
if grep -qE "$pattern" "$file"; then
pass "$description"
else
fail "$description"
fi
}
assert_has_dimensions() {
local file="$1"
local min="$2"
local description="${3:-Minimum $min security dimensions}"
local count
count=$(grep -cE '^\| .+ \| [0-9]+\.?[0-9]*/5' "$file" 2>/dev/null || echo "0")
if [ "$count" -ge "$min" ]; then
pass "$description ($count dimensions)"
else
fail "$description — only $count dimensions (min: $min)"
fi
}

View file

@ -0,0 +1,78 @@
#!/bin/bash
# run-e2e.sh — Run E2E regression tests for agent outputs
# Usage: bash tests/run-e2e.sh [--security] [--cost] [--summary] [--all]
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
RED='\033[0;31m'
GREEN='\033[0;32m'
CYAN='\033[0;36m'
NC='\033[0m'
# Parse arguments
RUN_SECURITY=false
RUN_COST=false
RUN_SUMMARY=false
RUN_ROS=false
RUN_AI_ACT=false
if [ $# -eq 0 ] || [ "${1:-}" = "--all" ]; then
RUN_SECURITY=true
RUN_COST=true
RUN_SUMMARY=true
RUN_ROS=true
RUN_AI_ACT=true
else
while [ $# -gt 0 ]; do
case "$1" in
--security) RUN_SECURITY=true ;;
--cost) RUN_COST=true ;;
--summary) RUN_SUMMARY=true ;;
--ros) RUN_ROS=true ;;
--ai-act) RUN_AI_ACT=true ;;
*)
echo "Usage: bash tests/run-e2e.sh [--security] [--cost] [--summary] [--ros] [--ai-act] [--all]"
exit 1
;;
esac
shift
done
fi
echo -e "${CYAN}╔══════════════════════════════════════════════╗${NC}"
echo -e "${CYAN}║ MS AI Architect — E2E Regression Tests ║${NC}"
echo -e "${CYAN}╚══════════════════════════════════════════════╝${NC}"
echo ""
FAILURES=0
if $RUN_SECURITY; then
bash "$SCRIPT_DIR/test-security-output.sh" || FAILURES=$((FAILURES + 1))
fi
if $RUN_COST; then
bash "$SCRIPT_DIR/test-cost-output.sh" || FAILURES=$((FAILURES + 1))
fi
if $RUN_SUMMARY; then
bash "$SCRIPT_DIR/test-summary-output.sh" || FAILURES=$((FAILURES + 1))
fi
if $RUN_ROS; then
bash "$SCRIPT_DIR/test-ros-output.sh" || FAILURES=$((FAILURES + 1))
fi
if $RUN_AI_ACT; then
bash "$SCRIPT_DIR/test-ai-act-output.sh" || FAILURES=$((FAILURES + 1))
fi
echo -e "${CYAN}══════════════════════════════════════════════${NC}"
if [ "$FAILURES" -eq 0 ]; then
echo -e "${GREEN} All E2E suites passed${NC}"
else
echo -e "${RED} $FAILURES suite(s) had failures${NC}"
fi
echo -e "${CYAN}══════════════════════════════════════════════${NC}"
exit $FAILURES

View file

@ -0,0 +1,81 @@
#!/bin/bash
# test-ai-act-output.sh — Validate ai-act-assessor agent output structure
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
source "$SCRIPT_DIR/lib/e2e-helpers.sh"
FIXTURE="$SCRIPT_DIR/fixtures/ai-act/fixture.md"
FIXTURE_HR="$SCRIPT_DIR/fixtures/ai-act/fixture-high-risk.md"
if [ ! -f "$FIXTURE" ]; then
echo "ERROR: Fixture not found: $FIXTURE"
echo "Run: bash tests/capture-fixture.sh to generate fixtures"
exit 1
fi
if [ ! -f "$FIXTURE_HR" ]; then
echo "ERROR: High-risk fixture not found: $FIXTURE_HR"
exit 1
fi
init_suite "AI Act Assessor Agent"
# === Minimal risk fixture ===
echo ""
echo " --- Minimal Risk Fixture ---"
# Structure checks
assert_has_section "$FIXTURE" "Risikoklassifisering" "Has risk classification section"
assert_has_section "$FIXTURE" "Rolle" "Has role section"
assert_has_section "$FIXTURE" "Forpliktelser" "Has obligations section"
# Content quality
assert_min_lines "$FIXTURE" 20 "Minimum 20 lines"
# Encoding
assert_encoding_ok "$FIXTURE" "UTF-8 encoding valid (minimal)"
assert_no_ascii_approximation "$FIXTURE" "No ASCII approximation (minimal)"
# Domain-specific: classification result
assert_matches_pattern "$FIXTURE" "(Minimal risiko|Begrenset risiko|Høyrisiko|Forbudt)" "Contains risk level classification"
assert_matches_pattern "$FIXTURE" "(Provider|Deployer)" "Contains role determination"
# Domain-specific: article references
assert_matches_pattern "$FIXTURE" "Art\. [0-9]" "References specific AI Act articles"
# === High risk fixture ===
echo ""
echo " --- High Risk Fixture ---"
# Structure checks
assert_has_section "$FIXTURE_HR" "Risikoklassifisering" "HR: Has risk classification section"
assert_has_section "$FIXTURE_HR" "Rolle" "HR: Has role section"
assert_has_section "$FIXTURE_HR" "Forpliktelser" "HR: Has obligations section"
assert_has_section "$FIXTURE_HR" "Tiltaksplan" "HR: Has action plan section"
assert_has_section "$FIXTURE_HR" "Neste steg" "HR: Has next steps section"
assert_has_section "$FIXTURE_HR" "Viktige frister" "HR: Has deadline section"
# Content quality
assert_min_lines "$FIXTURE_HR" 40 "HR: Minimum 40 lines"
assert_min_tables "$FIXTURE_HR" 1 "HR: Has at least 1 table"
# Encoding
assert_encoding_ok "$FIXTURE_HR" "HR: UTF-8 encoding valid"
assert_no_ascii_approximation "$FIXTURE_HR" "HR: No ASCII approximation"
# Domain-specific: high-risk classification
assert_matches_pattern "$FIXTURE_HR" "Høyrisiko" "HR: Classified as high-risk"
assert_matches_pattern "$FIXTURE_HR" "Annex III" "HR: References Annex III"
assert_matches_pattern "$FIXTURE_HR" "FRIA" "HR: References FRIA requirement"
# Domain-specific: deadlines
assert_matches_pattern "$FIXTURE_HR" "202[5-7]-[0-9]{2}-[0-9]{2}" "HR: Contains compliance deadline dates"
# Domain-specific: article references
assert_matches_pattern "$FIXTURE_HR" "Art\. (26|27|9|13|14|50)" "HR: References key deployer/provider articles"
# Domain-specific: action plan
assert_matches_pattern "$FIXTURE_HR" "(Kritisk|Høy|Middels|Lav)" "HR: Action plan has priority levels"
print_summary

View file

@ -0,0 +1,39 @@
#!/bin/bash
# test-cost-output.sh — Validate cost-estimation-agent output structure
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
source "$SCRIPT_DIR/lib/e2e-helpers.sh"
FIXTURE="$SCRIPT_DIR/fixtures/cost-estimation/fixture.md"
if [ ! -f "$FIXTURE" ]; then
echo "ERROR: Fixture not found: $FIXTURE"
echo "Run: bash tests/capture-fixture.sh to generate fixtures"
exit 1
fi
init_suite "Cost Estimation Agent"
# Structure checks
assert_has_section "$FIXTURE" "Kostnad" "Has cost section header"
assert_has_section "$FIXTURE" "TCO" "Has TCO section"
assert_has_section "$FIXTURE" "Konfidens" "Has confidence grading section"
assert_has_section "$FIXTURE" "Gevinst" "Has benefit realization section"
# Content quality
assert_min_lines "$FIXTURE" 30 "Minimum 30 lines"
assert_min_tables "$FIXTURE" 2 "Minimum 2 tables (TCO, confidence)"
assert_has_nok_amounts "$FIXTURE" 3 "At least 3 NOK amounts"
# Encoding
assert_encoding_ok "$FIXTURE" "UTF-8 encoding valid"
assert_no_ascii_approximation "$FIXTURE" "No ASCII approximation of Norwegian chars"
# Domain-specific
assert_matches_pattern "$FIXTURE" "(Alt|Alternativ)\s*[0-9]" "References numbered alternatives"
assert_matches_pattern "$FIXTURE" "(Etablering|etablering)" "Has establishment cost section"
assert_matches_pattern "$FIXTURE" "(drift|Drift)" "Has operational cost section"
assert_matches_pattern "$FIXTURE" "(MCP-verifisert|Verifisert|verifisert)" "Has verification markers"
print_summary

View file

@ -0,0 +1,132 @@
#!/bin/bash
# test-hooks.sh — Unit tests for ms-ai-architect hook scripts
# Usage: bash tests/test-hooks.sh
set -euo pipefail
PLUGIN_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
SCRIPTS_DIR="$PLUGIN_ROOT/hooks/scripts"
PASS=0
FAIL=0
pass() { echo -e "\033[0;32m ✓ $1\033[0m"; PASS=$((PASS + 1)); }
fail() { echo -e "\033[0;31m ✗ $1\033[0m"; FAIL=$((FAIL + 1)); }
echo "=== Hook Script Tests ==="
echo ""
# -------------------------------------------------------
# pre-edit-secrets.mjs
# -------------------------------------------------------
echo "--- pre-edit-secrets.mjs ---"
# Test 1: clean content passes
echo '{"tool_input":{"content":"Hello world","file_path":"readme.md"}}' \
| node "$SCRIPTS_DIR/pre-edit-secrets.mjs" 2>/dev/null \
&& pass "Clean content: exit 0" \
|| fail "Clean content: expected exit 0"
# Test 2: Azure Storage key blocked (constructed at runtime to avoid hook self-trigger)
AZURE_KEY_PAYLOAD=$(printf '{"tool_input":{"content":"%s","file_path":"config.ts"}}' \
"DefaultEndpointsProtocol=https;AccountName=test;AccountKey=$(printf 'a%.0s' {1..44})=")
echo "$AZURE_KEY_PAYLOAD" \
| node "$SCRIPTS_DIR/pre-edit-secrets.mjs" 2>/dev/null \
&& fail "Azure Storage key: expected exit 2 (blocked)" \
|| pass "Azure Storage key: blocked correctly"
# Test 3: GitHub token blocked (constructed at runtime)
GH_TOKEN="ghp_$(printf 'a%.0s' {1..40})"
printf '{"tool_input":{"content":"token: %s","file_path":"ci.yml"}}' "$GH_TOKEN" \
| node "$SCRIPTS_DIR/pre-edit-secrets.mjs" 2>/dev/null \
&& fail "GitHub token: expected exit 2 (blocked)" \
|| pass "GitHub token: blocked correctly"
# Test 4: empty content passes
echo '{"tool_input":{"content":"","file_path":"empty.md"}}' \
| node "$SCRIPTS_DIR/pre-edit-secrets.mjs" 2>/dev/null \
&& pass "Empty content: exit 0" \
|| fail "Empty content: expected exit 0"
# Test 5: test files skipped (constructed at runtime)
printf '{"tool_input":{"content":"api_key = \\"%s\\"","file_path":"auth.test.ts"}}' \
"$(printf 'x%.0s' {1..25})" \
| node "$SCRIPTS_DIR/pre-edit-secrets.mjs" 2>/dev/null \
&& pass "Test file: skipped (exit 0)" \
|| fail "Test file: should be skipped"
echo ""
# -------------------------------------------------------
# session-start-context.mjs
# -------------------------------------------------------
echo "--- session-start-context.mjs ---"
OUTPUT=$(CLAUDE_PLUGIN_ROOT="$PLUGIN_ROOT" node "$SCRIPTS_DIR/session-start-context.mjs" 2>/dev/null)
EXIT_CODE=$?
if [ $EXIT_CODE -eq 0 ]; then
pass "Runs without error (exit 0)"
else
fail "Expected exit 0, got $EXIT_CODE"
fi
if echo "$OUTPUT" | grep -q "Architect:"; then
pass "Output contains 'Architect:' prefix"
else
fail "Output missing 'Architect:' prefix"
fi
echo ""
# -------------------------------------------------------
# stop-assessment-reminder.mjs
# -------------------------------------------------------
echo "--- stop-assessment-reminder.mjs ---"
# Test from a temp dir without .work/
TMPDIR_TEST=$(mktemp -d)
OUTPUT=$(cd "$TMPDIR_TEST" && node "$SCRIPTS_DIR/stop-assessment-reminder.mjs" 2>/dev/null)
EXIT_CODE=$?
if [ $EXIT_CODE -eq 0 ]; then
pass "No .work/: exit 0"
else
fail "No .work/: expected exit 0, got $EXIT_CODE"
fi
if [ "$OUTPUT" = "{}" ]; then
pass "No .work/: returns {}"
else
fail "No .work/: expected {}, got: $OUTPUT"
fi
# Test with a fresh .work/ session
mkdir -p "$TMPDIR_TEST/.work/test-session"
touch "$TMPDIR_TEST/.work/test-session/state.json"
OUTPUT=$(cd "$TMPDIR_TEST" && node "$SCRIPTS_DIR/stop-assessment-reminder.mjs" 2>/dev/null)
if echo "$OUTPUT" | grep -q "systemMessage"; then
pass "Fresh .work/: returns systemMessage"
else
fail "Fresh .work/: expected systemMessage in output"
fi
if echo "$OUTPUT" | grep -q "architect:adr"; then
pass "Fresh .work/: suggests /architect:adr"
else
fail "Fresh .work/: missing /architect:adr suggestion"
fi
rm -rf "$TMPDIR_TEST"
echo ""
# -------------------------------------------------------
# Summary
# -------------------------------------------------------
TOTAL=$((PASS + FAIL))
echo "=== Results: $PASS/$TOTAL passed, $FAIL failed ==="
if [ "$FAIL" -gt 0 ]; then
exit 1
fi

View file

@ -0,0 +1,110 @@
#!/bin/bash
# test-kb-integrity.sh — Cross-reference agent KB paths against actual files
# Also finds orphaned KB files not referenced by any agent or SKILL.md
# Usage: bash tests/test-kb-integrity.sh
set -euo pipefail
PLUGIN_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
PASS=0
FAIL=0
WARN=0
pass() { echo -e "\033[0;32m ✓ $1\033[0m"; PASS=$((PASS + 1)); }
fail() { echo -e "\033[0;31m ✗ $1\033[0m"; FAIL=$((FAIL + 1)); }
warn() { echo -e "\033[1;33m ⚠ $1\033[0m"; WARN=$((WARN + 1)); }
echo "=== KB Integrity Test ==="
echo ""
# -------------------------------------------------------
# Check 1: Agent file references resolve
# -------------------------------------------------------
echo "--- Check 1: Agent File References ---"
for agent_file in "$PLUGIN_ROOT"/agents/*.md; do
[ -f "$agent_file" ] || continue
basename_agent="$(basename "$agent_file")"
# Extract explicit file paths (references/, skills/) from agent
while IFS= read -r ref_path; do
ref_path=$(echo "$ref_path" | sed 's/`//g; s/"//g; s/^[[:space:]]*//; s/[[:space:]]*$//')
[ -z "$ref_path" ] && continue
full_path="$PLUGIN_ROOT/$ref_path"
if [ -e "$full_path" ]; then
pass "$basename_agent -> $ref_path"
else
fail "$basename_agent -> $ref_path (NOT FOUND)"
fi
done < <(grep -oE '(references|skills)/[a-zA-Z0-9_./-]+\.md' "$agent_file" 2>/dev/null || true)
done
echo ""
# -------------------------------------------------------
# Check 2: SKILL.md file references resolve
# -------------------------------------------------------
echo "--- Check 2: SKILL.md File References ---"
for skill_file in "$PLUGIN_ROOT"/skills/*/SKILL.md; do
[ -f "$skill_file" ] || continue
skill_name="$(basename "$(dirname "$skill_file")")"
while IFS= read -r ref_path; do
ref_path=$(echo "$ref_path" | sed 's/`//g; s/"//g; s/^[[:space:]]*//; s/[[:space:]]*$//')
[ -z "$ref_path" ] && continue
# Resolve: references/ paths are relative to skill dir
if [[ "$ref_path" == references/* ]]; then
full_path="$PLUGIN_ROOT/skills/$skill_name/$ref_path"
else
full_path="$PLUGIN_ROOT/$ref_path"
fi
if [ -e "$full_path" ]; then
pass "SKILL:$skill_name -> $ref_path"
else
fail "SKILL:$skill_name -> $ref_path (NOT FOUND)"
fi
done < <(grep -oE '(references|skills)/[a-zA-Z0-9_./-]+\.md' "$skill_file" 2>/dev/null || true)
done
echo ""
# -------------------------------------------------------
# Check 3: Orphaned KB files
# -------------------------------------------------------
echo "--- Check 3: Orphaned KB Files ---"
ORPHAN_COUNT=0
while IFS= read -r kb_file; do
rel_path="${kb_file#$PLUGIN_ROOT/}"
kb_basename="$(basename "$kb_file")"
# Check if this filename appears in any agent or SKILL.md
if grep -rql "$kb_basename" "$PLUGIN_ROOT/agents/" "$PLUGIN_ROOT"/skills/*/SKILL.md 2>/dev/null; then
: # Referenced, ok
else
warn "Orphaned: $rel_path"
ORPHAN_COUNT=$((ORPHAN_COUNT + 1))
fi
done < <(find "$PLUGIN_ROOT/skills" -path "*/references/*.md" -type f 2>/dev/null)
if [ "$ORPHAN_COUNT" -eq 0 ]; then
pass "No orphaned KB files found"
fi
echo ""
# -------------------------------------------------------
# Summary
# -------------------------------------------------------
TOTAL=$((PASS + FAIL))
echo "=== Results: $PASS/$TOTAL passed, $FAIL failed, $WARN warnings ==="
if [ "$FAIL" -gt 0 ]; then
exit 1
fi

View file

@ -0,0 +1,161 @@
#!/bin/bash
# test-plugin-discovery.sh — Smoke test for auto-discovery chain
# Validates that plugin.json, hooks.json, and script references are consistent
# Usage: bash tests/test-plugin-discovery.sh
set -euo pipefail
PLUGIN_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
PASS=0
FAIL=0
pass() { echo -e "\033[0;32m ✓ $1\033[0m"; PASS=$((PASS + 1)); }
fail() { echo -e "\033[0;31m ✗ $1\033[0m"; FAIL=$((FAIL + 1)); }
echo "=== Plugin Discovery Smoke Test ==="
echo ""
# -------------------------------------------------------
# Check 1: plugin.json
# -------------------------------------------------------
echo "--- Check 1: plugin.json ---"
PLUGIN_JSON="$PLUGIN_ROOT/.claude-plugin/plugin.json"
if [ -f "$PLUGIN_JSON" ]; then
pass "plugin.json exists"
else
fail "plugin.json missing at .claude-plugin/plugin.json"
fi
if grep -q '"auto_discover": true' "$PLUGIN_JSON" 2>/dev/null || grep -q '"auto_discover":true' "$PLUGIN_JSON" 2>/dev/null; then
pass "auto_discover: true"
else
fail "auto_discover is not true"
fi
# plugin.json must NOT have a "hooks" key
if grep -q '"hooks"' "$PLUGIN_JSON" 2>/dev/null; then
fail "plugin.json contains 'hooks' key (should be in hooks/hooks.json only)"
else
pass "plugin.json does not contain 'hooks' key"
fi
echo ""
# -------------------------------------------------------
# Check 2: hooks.json
# -------------------------------------------------------
echo "--- Check 2: hooks.json ---"
HOOKS_JSON="$PLUGIN_ROOT/hooks/hooks.json"
if [ -f "$HOOKS_JSON" ]; then
pass "hooks.json exists"
else
fail "hooks.json missing at hooks/hooks.json"
fi
if node -e "JSON.parse(require('fs').readFileSync('$HOOKS_JSON', 'utf-8'))" 2>/dev/null; then
pass "hooks.json is valid JSON"
else
fail "hooks.json is invalid JSON"
fi
# Check structure: hooks key is an object
if node -e "
const h = JSON.parse(require('fs').readFileSync('$HOOKS_JSON', 'utf-8'));
if (!h.hooks || typeof h.hooks !== 'object') process.exit(1);
" 2>/dev/null; then
pass "hooks.json has 'hooks' object at root"
else
fail "hooks.json missing root 'hooks' object"
fi
echo ""
# -------------------------------------------------------
# Check 3: Script references
# -------------------------------------------------------
echo "--- Check 3: Script References ---"
# Extract script paths from hooks.json
SCRIPT_PATHS=$(node -e "
const h = JSON.parse(require('fs').readFileSync('$HOOKS_JSON', 'utf-8'));
for (const [event, handlers] of Object.entries(h.hooks)) {
for (const handler of handlers) {
for (const hook of (handler.hooks || [])) {
if (hook.command) {
const match = hook.command.match(/\\\$\\{CLAUDE_PLUGIN_ROOT\\}\\/(.+)/);
if (match) console.log(match[1]);
}
}
}
}
" 2>/dev/null)
for script_path in $SCRIPT_PATHS; do
full_path="$PLUGIN_ROOT/$script_path"
if [ -f "$full_path" ]; then
pass "Script exists: $script_path"
else
fail "Script missing: $script_path"
fi
done
echo ""
# -------------------------------------------------------
# Check 4: Event names
# -------------------------------------------------------
echo "--- Check 4: Event Names ---"
VALID_EVENTS="SessionStart UserPromptSubmit PreToolUse PostToolUse Stop"
HOOK_EVENTS=$(node -e "
const h = JSON.parse(require('fs').readFileSync('$HOOKS_JSON', 'utf-8'));
for (const event of Object.keys(h.hooks)) console.log(event);
" 2>/dev/null)
for event in $HOOK_EVENTS; do
if echo "$VALID_EVENTS" | grep -qw "$event"; then
pass "Valid event: $event"
else
fail "Invalid event: $event"
fi
done
echo ""
# -------------------------------------------------------
# Check 5: Documentation
# -------------------------------------------------------
echo "--- Check 5: Documentation ---"
CLAUDE_MD="$PLUGIN_ROOT/CLAUDE.md"
if grep -q "## Hooks" "$CLAUDE_MD" 2>/dev/null; then
pass "CLAUDE.md has Hooks section"
else
fail "CLAUDE.md missing Hooks section"
fi
for script in pre-edit-secrets session-start-context stop-assessment-reminder; do
if grep -q "$script" "$CLAUDE_MD" 2>/dev/null; then
pass "CLAUDE.md documents $script"
else
fail "CLAUDE.md missing $script documentation"
fi
done
echo ""
# -------------------------------------------------------
# Summary
# -------------------------------------------------------
TOTAL=$((PASS + FAIL))
echo "=== Results: $PASS/$TOTAL passed, $FAIL failed ==="
if [ "$FAIL" -gt 0 ]; then
exit 1
fi

View file

@ -0,0 +1,74 @@
#!/bin/bash
# test-ros-output.sh — Validate ros-analysis-agent output structure
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
source "$SCRIPT_DIR/lib/e2e-helpers.sh"
FIXTURE="$SCRIPT_DIR/fixtures/ros-analysis/fixture.md"
if [ ! -f "$FIXTURE" ]; then
echo "ERROR: Fixture not found: $FIXTURE"
echo "Run: bash tests/capture-fixture.sh to generate fixtures"
exit 1
fi
init_suite "ROS Analysis Agent"
# Structure checks
assert_has_section "$FIXTURE" "ROS-analyse" "Has ROS analysis header"
assert_has_section "$FIXTURE" "Risikoregister" "Has risk register section"
assert_has_section "$FIXTURE" "Risikomatrise" "Has risk matrix section"
assert_has_section "$FIXTURE" "Tiltaksplan" "Has measures plan section"
assert_has_section "$FIXTURE" "Restrisiko" "Has residual risk section"
assert_has_section "$FIXTURE" "Dimensjonsvurdering" "Has dimension assessment section"
# Content quality
assert_min_lines "$FIXTURE" 60 "Minimum 60 lines"
assert_min_tables "$FIXTURE" 4 "Minimum 4 tables (register, matrix, dimensions, measures)"
assert_scores_in_range "$FIXTURE" "ROS scores in valid X/5 range"
assert_has_dimensions "$FIXTURE" 6 "At least 6 risk dimensions scored"
# Encoding
assert_encoding_ok "$FIXTURE" "UTF-8 encoding valid"
assert_no_ascii_approximation "$FIXTURE" "No ASCII approximation of Norwegian chars"
# Domain-specific: methodology references
assert_matches_pattern "$FIXTURE" "(NS 5814|ISO 31000)" "References NS 5814 or ISO 31000 methodology"
# Domain-specific: threat and risk IDs
assert_matches_pattern "$FIXTURE" "T-[A-Z]{3}-[0-9]{2}" "Contains threat IDs (T-xxx-NN format)"
assert_matches_pattern "$FIXTURE" "R-[0-9]" "Contains risk IDs (R-NN format)"
# Domain-specific: risk dimensions
assert_matches_pattern "$FIXTURE" "(Modellsikkerhet|Dataintegritet|Bias|Tilgjengelighet|Forklarbarhet|Juridisk|Organisatorisk)" "Covers ROS risk dimensions"
# Domain-specific: regulatory references
assert_matches_pattern "$FIXTURE" "(AI Act|GDPR|OWASP)" "References key regulations/standards"
# Structure: check all 8 phases (Full ROS)
assert_has_section "$FIXTURE" "Fase 1" "Has Phase 1 header"
assert_has_section "$FIXTURE" "Fase 2" "Has Phase 2 header"
assert_has_section "$FIXTURE" "Fase 3" "Has Phase 3 header"
assert_has_section "$FIXTURE" "Fase 4" "Has Phase 4 header"
assert_has_section "$FIXTURE" "Fase 5" "Has Phase 5 header"
assert_has_section "$FIXTURE" "Fase 6" "Has Phase 6 header"
assert_has_section "$FIXTURE" "Fase 7" "Has Phase 7 header"
assert_has_section "$FIXTURE" "Fase 8" "Has Phase 8 header"
assert_has_section "$FIXTURE" "Ledelsessammendrag" "Has executive summary"
# Measure IDs (M-xxx)
assert_matches_pattern "$FIXTURE" "M-[0-9]" "Contains measure IDs (M-NN format)"
# Minimum threat count for full ROS
threat_count=$(grep -cE "T-[A-Z]{3}-[0-9]{2}" "$FIXTURE" || echo 0)
if [ "$threat_count" -ge 8 ]; then
pass "Minimum 8 threats identified ($threat_count found)"
else
fail "Minimum 8 threats — only $threat_count found"
fi
# Vedlegg O coverage (for systems with agents/MCP)
assert_matches_pattern "$FIXTURE" "(MAESTRO|forsyningskjede|MCP|supply chain)" "References supply chain/MAESTRO (Vedlegg O coverage)"
print_summary

View file

@ -0,0 +1,43 @@
#!/bin/bash
# test-security-output.sh — Validate security-assessment-agent output structure
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
source "$SCRIPT_DIR/lib/e2e-helpers.sh"
FIXTURE="$SCRIPT_DIR/fixtures/security-assessment/fixture.md"
if [ ! -f "$FIXTURE" ]; then
echo "ERROR: Fixture not found: $FIXTURE"
echo "Run: bash tests/capture-fixture.sh to generate fixtures"
exit 1
fi
init_suite "Security Assessment Agent"
# Structure checks
assert_has_section "$FIXTURE" "Sikkerhetsvurdering" "Has security assessment header"
assert_has_section "$FIXTURE" "Sikkerhetsscoring" "Has scoring section"
assert_has_section "$FIXTURE" "Kritiske funn" "Has critical findings section"
assert_has_section "$FIXTURE" "DPIA" "Has DPIA section"
assert_has_section "$FIXTURE" "ROS-analyse" "Has risk analysis section"
assert_has_section "$FIXTURE" "Dataklassifisering" "Has data classification section"
# Content quality
assert_min_lines "$FIXTURE" 40 "Minimum 40 lines"
assert_min_tables "$FIXTURE" 3 "Minimum 3 tables (scoring, ROS, data classification)"
assert_scores_in_range "$FIXTURE" "Security scores in valid X/5 range"
assert_has_dimensions "$FIXTURE" 5 "At least 5 security dimensions scored"
# Encoding
assert_encoding_ok "$FIXTURE" "UTF-8 encoding valid"
assert_no_ascii_approximation "$FIXTURE" "No ASCII approximation of Norwegian chars"
# Domain-specific
assert_matches_pattern "$FIXTURE" "(GDPR|DPIA|personvern)" "References GDPR/DPIA"
assert_matches_pattern "$FIXTURE" "(AI Act|AI-Act)" "References AI Act"
assert_matches_pattern "$FIXTURE" "(Schrems II|Schrems)" "References Schrems II"
assert_matches_pattern "$FIXTURE" "P0|P1|Blokkerende" "Has priority classifications (P0/P1)"
assert_matches_pattern "$FIXTURE" "(Identity|Network|Data Protection|Content Safety|Compliance|Monitoring)" "Covers standard security dimensions"
print_summary

View file

@ -0,0 +1,39 @@
#!/bin/bash
# test-summary-output.sh — Validate summary-agent output structure
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
source "$SCRIPT_DIR/lib/e2e-helpers.sh"
FIXTURE="$SCRIPT_DIR/fixtures/summary/fixture.md"
if [ ! -f "$FIXTURE" ]; then
echo "ERROR: Fixture not found: $FIXTURE"
echo "Run: bash tests/capture-fixture.sh to generate fixtures"
exit 1
fi
init_suite "Summary Agent"
# Structure checks
assert_has_section "$FIXTURE" "Sammendrag" "Has summary header"
assert_has_section "$FIXTURE" "Beslutningsnotat" "Has decision note section"
assert_has_section "$FIXTURE" "Nøkkeltall" "Has key figures section"
# Content quality
assert_min_lines "$FIXTURE" 50 "Minimum 50 lines"
assert_min_tables "$FIXTURE" 1 "At least 1 table (key figures)"
assert_has_nok_amounts "$FIXTURE" 3 "At least 3 NOK amounts"
# Encoding
assert_encoding_ok "$FIXTURE" "UTF-8 encoding valid"
assert_no_ascii_approximation "$FIXTURE" "No ASCII approximation of Norwegian chars"
# Cross-references
assert_matches_pattern "$FIXTURE" "S[0-9]" "References other sections (S-numbers)"
assert_matches_pattern "$FIXTURE" "(Sikkerhet|sikkerhet)" "Cross-references security"
assert_matches_pattern "$FIXTURE" "(Kostnad|kostnad|TCO)" "Cross-references cost"
assert_matches_pattern "$FIXTURE" "(Go|NO-GO|anbefaling|Anbefaling)" "Contains recommendation"
assert_matches_pattern "$FIXTURE" "(Fase|fase)\s*[0-9]" "References implementation phases"
print_summary

View file

@ -0,0 +1,293 @@
#!/bin/bash
# validate-plugin.sh — Static validation for ms-ai-architect plugin
# Usage: bash tests/validate-plugin.sh
set -euo pipefail
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'
PLUGIN_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
PASS=0
FAIL=0
WARN=0
pass() { echo -e "${GREEN}$1${NC}"; PASS=$((PASS + 1)); }
fail() { echo -e "${RED}$1${NC}"; FAIL=$((FAIL + 1)); }
warn() { echo -e "${YELLOW}$1${NC}"; WARN=$((WARN + 1)); }
echo "=== ms-ai-architect Plugin Validation ==="
echo "Plugin root: $PLUGIN_ROOT"
echo ""
# -------------------------------------------------------
# Check 1: Agent Frontmatter
# -------------------------------------------------------
echo "--- Check 1: Agent Frontmatter ---"
VALID_MODELS="opus sonnet haiku"
VALID_COLORS="blue green yellow purple cyan red orange magenta white"
for agent_file in "$PLUGIN_ROOT"/agents/*.md; do
[ -f "$agent_file" ] || continue
basename_file="$(basename "$agent_file")"
# Must have --- on line 1
first_line="$(head -n 1 "$agent_file")"
if [ "$first_line" != "---" ]; then
fail "$basename_file: missing frontmatter delimiter (---) on line 1"
continue
fi
# Extract frontmatter (between first and second ---)
frontmatter="$(sed -n '1,/^---$/{ /^---$/d; p; }' "$agent_file" | sed '1d')"
# sed '1d' removes the first --- captured; we actually need lines between first and second ---
# Redo: extract lines between line 2 and next ---
frontmatter="$(awk 'NR==1{next} /^---$/{exit} {print}' "$agent_file")"
# Check required fields
for field in "name:" "description:" "model:" "color:" "tools:"; do
if echo "$frontmatter" | grep -q "^${field}"; then
pass "$basename_file: has $field"
elif echo "$frontmatter" | grep -q "^ *${field}"; then
# indented (part of multiline) - still counts for description
pass "$basename_file: has $field"
else
# description can be multi-line with |
if [ "$field" = "description:" ] && echo "$frontmatter" | grep -q "description:"; then
pass "$basename_file: has $field"
else
fail "$basename_file: missing $field"
fi
fi
done
# Validate model value
model_value="$(echo "$frontmatter" | grep "^model:" | sed 's/^model: *//' | tr -d '[:space:]')"
if [ -n "$model_value" ]; then
model_valid=false
for m in $VALID_MODELS; do
if [ "$model_value" = "$m" ]; then
model_valid=true
break
fi
done
if $model_valid; then
pass "$basename_file: model '$model_value' is valid"
else
fail "$basename_file: model '$model_value' is not valid (expected: $VALID_MODELS)"
fi
fi
# Validate color value
color_value="$(echo "$frontmatter" | grep "^color:" | sed 's/^color: *//' | tr -d '[:space:]')"
if [ -n "$color_value" ]; then
color_valid=false
for c in $VALID_COLORS; do
if [ "$color_value" = "$c" ]; then
color_valid=true
break
fi
done
if $color_valid; then
pass "$basename_file: color '$color_value' is valid"
else
fail "$basename_file: color '$color_value' is not valid (expected: $VALID_COLORS)"
fi
fi
# Validate tools is a JSON array (starts with [)
tools_line="$(echo "$frontmatter" | grep "^tools:" || true)"
if [ -n "$tools_line" ]; then
tools_value="$(echo "$tools_line" | sed 's/^tools: *//')"
if echo "$tools_value" | grep -q '^\['; then
pass "$basename_file: tools is a JSON array"
else
fail "$basename_file: tools is not a JSON array (got: $tools_value)"
fi
fi
done
echo ""
# -------------------------------------------------------
# Check 2: Command Frontmatter
# -------------------------------------------------------
echo "--- Check 2: Command Frontmatter ---"
for cmd_file in "$PLUGIN_ROOT"/commands/*.md; do
[ -f "$cmd_file" ] || continue
basename_file="$(basename "$cmd_file")"
basename_noext="${basename_file%.md}"
# Must have --- on line 1
first_line="$(head -n 1 "$cmd_file")"
if [ "$first_line" != "---" ]; then
fail "$basename_file: missing frontmatter delimiter (---) on line 1"
continue
fi
# Extract frontmatter
frontmatter="$(awk 'NR==1{next} /^---$/{exit} {print}' "$cmd_file")"
# Check required fields: name, description
for field in "name:" "description:"; do
if echo "$frontmatter" | grep -q "${field}"; then
pass "$basename_file: has $field"
else
fail "$basename_file: missing $field"
fi
done
# Check allowed-tools (warn if missing)
if echo "$frontmatter" | grep -q "allowed-tools:"; then
pass "$basename_file: has allowed-tools"
else
warn "$basename_file: missing allowed-tools (recommended)"
fi
# Validate name matches filename pattern
name_value="$(echo "$frontmatter" | grep "^name:" | sed 's/^name: *//' | tr -d '[:space:]')"
if [ -n "$name_value" ] && [ "$name_value" = "$basename_noext" ]; then
pass "$basename_file: name matches filename"
elif [ -n "$name_value" ]; then
fail "$basename_file: name '$name_value' does not match filename '$basename_noext'"
fi
done
echo ""
# -------------------------------------------------------
# Check 3: Encoding Validation
# -------------------------------------------------------
echo "--- Check 3: Encoding Validation ---"
encoding_issues=0
for dir in agents commands skills; do
dir_path="$PLUGIN_ROOT/$dir"
[ -d "$dir_path" ] || continue
while IFS= read -r -d '' mdfile; do
basename_file="$(basename "$mdfile")"
rel_path="${mdfile#$PLUGIN_ROOT/}"
# Check for broken UTF-8 sequences
if grep -ql 'æ\|ø\|Ã¥\|Ã\†\|Ø\|Ã…' "$mdfile" 2>/dev/null; then
fail "$rel_path: broken æ/ø/å encoding detected"
encoding_issues=$((encoding_issues + 1))
fi
if grep -ql 'â€"' "$mdfile" 2>/dev/null; then
fail "$rel_path: broken em-dash/en-dash encoding detected"
encoding_issues=$((encoding_issues + 1))
fi
done < <(find "$dir_path" -name '*.md' -print0)
done
if [ "$encoding_issues" -eq 0 ]; then
pass "No encoding issues found in agents/, commands/, skills/"
fi
echo ""
# -------------------------------------------------------
# Check 4: KB Reference Validation
# -------------------------------------------------------
echo "--- Check 4: KB Reference Validation ---"
for agent_file in "$PLUGIN_ROOT"/agents/*.md; do
[ -f "$agent_file" ] || continue
basename_file="$(basename "$agent_file")"
# Extract lines referencing references/ paths
ref_paths="$(grep -o 'references/[a-zA-Z0-9_-]*/\?' "$agent_file" | sort -u || true)"
if [ -z "$ref_paths" ]; then
continue
fi
while IFS= read -r ref_path; do
# Normalize: remove trailing slash, build full path relative to skill references
ref_dir="$(echo "$ref_path" | sed 's|/$||')"
# Check across all skill directories
full_path=""
for skill_dir in "$PLUGIN_ROOT"/skills/*/; do
if [ -d "${skill_dir}${ref_dir}" ]; then
full_path="${skill_dir}${ref_dir}"
break
fi
done
if [ -z "$full_path" ]; then
full_path="$PLUGIN_ROOT/skills/ms-ai-engineering/$ref_dir"
fi
if [ -d "$full_path" ]; then
# Check if directory has files
file_count="$(find "$full_path" -maxdepth 1 -name '*.md' -type f | wc -l | tr -d ' ')"
if [ "$file_count" -gt 0 ]; then
pass "$basename_file: $ref_dir/ exists ($file_count files)"
else
warn "$basename_file: $ref_dir/ exists but is empty"
fi
else
fail "$basename_file: referenced $ref_dir/ does not exist at $full_path"
fi
done <<< "$ref_paths"
done
echo ""
# -------------------------------------------------------
# Check 5: Plugin.json Validation
# -------------------------------------------------------
echo "--- Check 5: Plugin.json Validation ---"
plugin_json="$PLUGIN_ROOT/.claude-plugin/plugin.json"
if [ ! -f "$plugin_json" ]; then
fail "plugin.json not found at .claude-plugin/plugin.json"
else
pass "plugin.json exists"
# Check required fields
for field in "name" "version" "description"; do
if grep -q "\"$field\"" "$plugin_json"; then
pass "plugin.json: has \"$field\""
else
fail "plugin.json: missing \"$field\""
fi
done
# Check auto_discover: true
if grep -q '"auto_discover"' "$plugin_json"; then
auto_val="$(grep '"auto_discover"' "$plugin_json" | grep -o 'true\|false')"
if [ "$auto_val" = "true" ]; then
pass "plugin.json: auto_discover is true"
else
fail "plugin.json: auto_discover is not true (got: $auto_val)"
fi
else
fail "plugin.json: missing auto_discover"
fi
fi
echo ""
# -------------------------------------------------------
# Summary
# -------------------------------------------------------
echo "=== Results ==="
echo -e "${GREEN}PASS: $PASS${NC}"
echo -e "${RED}FAIL: $FAIL${NC}"
echo -e "${YELLOW}WARN: $WARN${NC}"
if [ $FAIL -gt 0 ]; then
echo -e "${RED}VALIDATION FAILED${NC}"
exit 1
else
echo -e "${GREEN}VALIDATION PASSED${NC}"
exit 0
fi