feat(ms-ai-architect): R7.3 start — 49 filer ekstrahert (1008 claims), payloads bygget, 5 dømt+ingestet (ledger 94→99) [skip-docs]
Ekstraksjon komplett for hele R7.3-batchen (neste 49 av 149 pending, worklist- rekkefølge): 1008 claims over 49 filer, null valideringsfeil. Manifest og 49 v3.1-payloads bygget deterministisk. Judge-pass startet: 5 filer dømt blindt mot live MS Learn (alle flagged, 33 nye flagg) og ingestet serielt etter dry-run. Ingen pass-records → verken re-judge- gate eller stempling utløst. Ledger: 99 records (R7.1 45 + R7.2 49 + R7.3 5), 88 flagged / 11 pass, 287 flagg. Suite 947/947.
This commit is contained in:
parent
2935039f44
commit
80bfd02726
1 changed files with 452 additions and 1 deletions
|
|
@ -4,7 +4,7 @@
|
|||
"derived_from": "scripts/kb-update/data/full-pass-worklist.json (243 due, all never-verified)",
|
||||
"cadence": "R7–R10 (5 økter × ~49, 8–10 samtidige)",
|
||||
"batch": "R7.1",
|
||||
"count": 94,
|
||||
"count": 99,
|
||||
"generated": "2026-07-18"
|
||||
},
|
||||
"files": [
|
||||
|
|
@ -3104,6 +3104,457 @@
|
|||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/data-drift-monitoring-detection.md",
|
||||
"batch": "R7.3",
|
||||
"judged_at": "2026-07-25",
|
||||
"per_file_verdict": "flagged",
|
||||
"claim_count": 22,
|
||||
"verified": null,
|
||||
"verified_by": null,
|
||||
"flags": [
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#4",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R2",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/machine-learning/reference-yaml-monitor?view=azureml-api-2",
|
||||
"evidence_quote": "Allowed categorical metric names: `jensen_shannon_distance`, `chi_squared_test`, `population_stability_index`",
|
||||
"reason": "Pearson's Chi-Squared Test stemmer, men Euclidean Distance er fraværende fra den kanoniske kategoriske metrikklisten for v2 model monitoring — den er en v1 Dataset Monitor-metrikk (how-to-monitor-datasets), ikke en v2 drift detection-metrikk.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/data-drift-monitoring-detection.md",
|
||||
"line": 50,
|
||||
"claim": "For kategoriske features bruker Azure ML drift detection disse metrikkene: Pearson's Chi-Squared Test | Euclidean Distance (beregnet på empiriske fordelinger av kategoriske kolonner).",
|
||||
"disposition": "outdated"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#9",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R4",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2",
|
||||
"evidence_quote": "spark_compute = ServerlessSparkCompute( instance_type=\"standard_e4s_v3\", runtime_version=\"3.4\" )",
|
||||
"reason": "Den siterte siden bruker runtime_version \"3.4\" i samtlige SDK- og CLI-eksempler; 3.3 er en superseded verdi som kun henger igjen i skjematabellen på reference-yaml-monitor, hvis egne YAML-eksempler også er oppdatert til 3.4.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/data-drift-monitoring-detection.md",
|
||||
"line": 112,
|
||||
"claim": "ServerlessSparkCompute for model monitoring konfigureres med runtime_version \"3.3\" (Spark runtime-versjon).",
|
||||
"disposition": "outdated"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#10",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R4",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/machine-learning/how-to-monitor-datasets?view=azureml-api-1",
|
||||
"evidence_quote": "Data drift (preview) was retired on September 1, 2025. Migrate to Model Monitor for your data drift tasks.",
|
||||
"reason": "Migreringsanbefalingen stemmer, men statusen er ikke «deprecated»: funksjonen er retired per 1. september 2025 — et senere og handlingsendrende stadium enn deprecation, og eksakt-verdi-regelen gjelder med særlig kraft for status.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/data-drift-monitoring-detection.md",
|
||||
"line": 144,
|
||||
"claim": "Azure ML Dataset Monitors (preview, v1 SDK) er deprecated, og migrering til Model Monitor (v2 SDK) er anbefalt.",
|
||||
"disposition": "outdated"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#14",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R8",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2",
|
||||
"evidence_quote": "Schedule model monitoring jobs to run on serverless Spark compute pools.",
|
||||
"reason": "Workspace (v2) og Blob Storage stemmer, men den bærende compute-delen er feil: både how-to-siden og skjemaet krever Spark pool («Description of compute resources for Spark pool to run monitoring job»), «managed compute cluster» er ikke et alternativ, og Application Insights står ikke i prerequisites for v2 model monitoring.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/data-drift-monitoring-detection.md",
|
||||
"line": 191,
|
||||
"claim": "Data drift monitoring krever: Azure ML workspace (v2 API) | compute (serverless Spark eller managed compute cluster) | datastore for production inference data (Azure Blob Storage eller ADLS Gen2) | valgfritt Application Insights for custom metrics logging.",
|
||||
"disposition": "outdated"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#18",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R8",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/foundry/concepts/observability",
|
||||
"evidence_quote": "Scheduled evaluation: Scheduled quality and safety evaluation using test datasets to detect system drift",
|
||||
"reason": "Første del holder (Foundry har egen gen-AI-monitorering med groundedness, relevance og fluency), men den kanoniske observability-siden lister kun Evaluation, Monitoring og Tracing og «system drift» via planlagt evaluering på testdatasett — drift detection for grounding data i RAG er fraværende.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/data-drift-monitoring-detection.md",
|
||||
"line": 218,
|
||||
"claim": "Microsoft Foundry har egen monitoring for generativ AI med generation quality metrics: groundedness | relevance | fluency, og støtter drift detection for grounding data i RAG-scenarier.",
|
||||
"disposition": "outdated"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#19",
|
||||
"judge_verdict": "source_silent",
|
||||
"rule": "",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/machine-learning/concept-plan-manage-cost?view=azureml-api-2",
|
||||
"evidence_quote": "",
|
||||
"reason": "Ingen learn.microsoft.com-side sier at model monitoring er inkludert uten separat lisens eller at Azure ML-workspacet i seg selv er kostnadsfritt; kostnadssiden beskriver kun at medfølgende ressurser (ACR, Blob Storage, Key Vault, Azure Monitor) og compute påløper etter forbruk.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/data-drift-monitoring-detection.md",
|
||||
"line": 286,
|
||||
"claim": "Data drift monitoring er inkludert i Azure Machine Learning uten separat lisens; Azure ML workspace har ingen kostnad i seg selv, mens compute og storage faktureres separat (consumption-based).",
|
||||
"disposition": "unsourced"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#20",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R2",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/machine-learning/reference-yaml-monitor?view=azureml-api-2",
|
||||
"evidence_quote": "**Optional** for basic model monitoring that uses recent past production data as comparison baseline and has 3 monitoring signals: data drift, prediction drift, and data quality.",
|
||||
"reason": "Den kanoniske opplistingen gir out-of-box nøyaktig tre signaler (data drift, prediction drift, data quality) — feature attribution drift og custom signals er fraværende og krever advanced- eller custom-oppsett, hvilket how-to-siden bekrefter.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/data-drift-monitoring-detection.md",
|
||||
"line": 388,
|
||||
"claim": "Out-of-box model monitoring-signaler for online endpoints er: Data quality | Data drift | Prediction drift | Feature attribution drift | Custom signals (brukerdefinerte metrics via Python-skript).",
|
||||
"disposition": "outdated"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#21",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R8",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2",
|
||||
"evidence_quote": "You can monitor models that you deploy to Azure Machine Learning batch endpoints or models you deploy outside Azure Machine Learning.",
|
||||
"reason": "Siden definerer «advanced» som flere signaler, trenings-/valideringsdata som referanse og topp-N-features, mens modeller utenfor Azure ML og batch endpoints hører til et eget oppsett («Set up model monitoring for production data») — claimets kategorimapping av Advanced er dermed feil.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/data-drift-monitoring-detection.md",
|
||||
"line": 395,
|
||||
"claim": "Oppsettsalternativene for model monitoring er: Out-of-box (automatisk konfigurert for Azure ML online endpoints, ingen konfigurasjon påkrevd) | Advanced (custom monitoring for modeller deployet utenfor Azure ML, batch endpoints eller eksternt) | Azure Event Grid-integrasjon for ruting av alerts.",
|
||||
"disposition": "outdated"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#22",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R8",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/machine-learning/reference-yaml-monitor?view=azureml-api-2",
|
||||
"evidence_quote": "Allowed numerical metric names: `jensen_shannon_distance`, `normalized_wasserstein_distance`, `population_stability_index`, `two_sample_kolmogorov_smirnov_test`",
|
||||
"reason": "jensen_shannon_distance er tillatt for både numeriske og kategoriske features (ikke kategoriske alene), metrikken heter «Jensen-Shannon Distance» og ikke divergence, og «Normalized Wasserstein Distance» er det korrekte navnet — mappingen claimet setter opp finnes ikke i dokumentasjonen og motsier claim #3 i samme fil.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/data-drift-monitoring-detection.md",
|
||||
"line": 400,
|
||||
"claim": "Statistiske metoder brukt av Azure ML model monitoring: Jensen-Shannon divergence for kategoriske features | Wasserstein distance (Earth Mover's Distance) for numeriske features | Population Stability Index (PSI) for feature-stabilitet.",
|
||||
"disposition": "outdated"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/feedback-loops-continuous-improvement.md",
|
||||
"batch": "R7.3",
|
||||
"judged_at": "2026-07-25",
|
||||
"per_file_verdict": "flagged",
|
||||
"claim_count": 20,
|
||||
"verified": null,
|
||||
"verified_by": null,
|
||||
"flags": [
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#3",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R2",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/foundry/concepts/built-in-evaluators",
|
||||
"evidence_quote": "General purpose evaluators | Coherence | Fluency ... RAG evaluators | Retrieval | Document Retrieval | Groundedness | Groundedness Pro (preview) | Relevance | Response Completeness (preview)",
|
||||
"reason": "Jeg hentet den kanoniske enumereringen (Built-in evaluators reference) — relevance, groundedness og safety-evaluatorer finnes, men INGEN «Correctness»-evaluator er oppført i noen kategori (Correctness er en MLflow-scorer, ikke en Foundry-evaluator); fraværet er bevis mot eksistenspåstanden.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/feedback-loops-continuous-improvement.md",
|
||||
"line": 93,
|
||||
"claim": "Foundry Agent Evaluation utfører evaluering med LLM judges for correctness | relevance | groundedness | safety.",
|
||||
"disposition": "outdated"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#6",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R8",
|
||||
"evidence_url": "https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2",
|
||||
"evidence_quote": "spark_compute = ServerlessSparkCompute(instance_type=\"standard_e4s_v3\", runtime_version=\"3.4\")",
|
||||
"reason": "instance_type standard_e4s_v3 stemmer, men siden angir runtime_version \"3.4\" i alle eksempler (YAML og SDK) — claimets 3.3 er en gammel verdi; én bærende del er feil.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/feedback-loops-continuous-improvement.md",
|
||||
"line": 176,
|
||||
"claim": "Azure ML model monitoring kan kjøre på ServerlessSparkCompute med instance_type standard_e4s_v3 og runtime_version 3.3.",
|
||||
"disposition": "outdated"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#7",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R2",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/databricks/mlflow3/genai/",
|
||||
"evidence_quote": "MLflow 3 for GenAI provides these key pieces for efficient development, deployment, and continuous improvement: Tracing ... Built-in and custom LLM judges and scorers ... Review apps for expert feedback ... Automated evaluation and monitoring ... App and prompt versioning",
|
||||
"reason": "Den siterte kanoniske siden definerer forbedringssyklusen med FEM nøkkelelementer; ingen Learn-side definerer en «10-stegs» syklus med de ti navngitte stegene claimet lister.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/feedback-loops-continuous-improvement.md",
|
||||
"line": 241,
|
||||
"claim": "MLflow for GenAI definerer en 10-stegs kontinuerlig forbedringssyklus: Production App (traces) | User Feedback (thumbs up/down) | Monitor & Score (LLM judges) | Identify Issues (Trace UI) | Domain Expert Review (Review App) | Build Eval Dataset | Tune Scorers | Evaluate New Versions | Compare Results (MLflow evaluation runs) | Deploy or Iterate.",
|
||||
"disposition": "outdated"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#8",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R8",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/machine-learning/concept-data-collection?view=azureml-api-2",
|
||||
"evidence_quote": "Azure Machine Learning Data collector provides real-time logging of input and output data from models that are deployed to managed online endpoints or Kubernetes online endpoints. Azure Machine Learning stores the logged inference data in Azure blob storage.",
|
||||
"reason": "Model Monitor, retraining via pipelines og blue-green på managed online endpoints holder, men datainnsamlingen skjer via Data collector til Azure Blob Storage — Azure ML har ingen «inference tables» (det er et Databricks-begrep); én bærende del er feil.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/feedback-loops-continuous-improvement.md",
|
||||
"line": 443,
|
||||
"claim": "Feedback loop-komponentene i Azure Machine Learning er: Data collection via inference tables på managed endpoints | Monitoring via Model Monitor | Alerting via Azure Monitor Alerts (e-post/webhook ved threshold breach) | Retraining via Azure ML Pipelines | A/B-testing via staging endpoints | Deployment via Managed Online Endpoints med blue-green deployment.",
|
||||
"disposition": "outdated"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#9",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R8",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/foundry/concepts/observability",
|
||||
"evidence_quote": "Microsoft Foundry provides three core capabilities that work together to deliver comprehensive observability across the AI application lifecycle: Evaluation ... Monitoring ... Tracing",
|
||||
"reason": "Foundry Observability-dashboardet og AI red teaming agent står på siden, men den kanoniske Foundry-enumereringen inneholder verken Review App, «Agent Evaluation» eller MLflow Datasets i Unity Catalog — det er Databricks-MLflow-komponenter, ikke Microsoft Foundry-komponenter.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/feedback-loops-continuous-improvement.md",
|
||||
"line": 471,
|
||||
"claim": "Feedback loop-komponentene for GenAI i Microsoft Foundry er: MLflow Tracing (Databricks, span-nivå telemetri) | Review App (thumbs up/down og tekstlig feedback) | Agent Evaluation (LLM judges) | Microsoft Foundry Observability (dashboard for kvalitetstrender, latens, feil) | MLflow Datasets i Unity Catalog (versjonerte testsett) | AI Red Teaming Agent (adversarial testing for sikkerhet).",
|
||||
"disposition": "outdated"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#11",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R8",
|
||||
"evidence_url": "https://learn.microsoft.com/en-us/ai-builder/feedback-loop",
|
||||
"evidence_quote": "Select Feedback loop and add the documents that could improve your model. Tag these new documents and retrain the model.",
|
||||
"reason": "Power Automate-ruting på confidence score og Dataverse-lagring stemmer, men feedback loop legger IKKE reviewede samples automatisk til treningssettet — dokumentene må velges, tagges og modellen retrenes manuelt; SharePoint nevnes heller ikke som lagring (siden sier eksplisitt Dataverse-tabellen «AI Builder Feedback Loop»).",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/feedback-loops-continuous-improvement.md",
|
||||
"line": 552,
|
||||
"claim": "Feedback loop-komponentene i Power Platform AI er: Power Automate (ruting av low-confidence predictions til human review) | Dataverse / SharePoint (lagring av feedback-data) | AI Builder Feedback Loop (legger reviewede samples automatisk til treningssettet) | AI Builder (manuell/planlagt retraining).",
|
||||
"disposition": "outdated"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#12",
|
||||
"judge_verdict": "source_silent",
|
||||
"rule": "",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/well-architected/sustainability/sustainability-cost-optimization-recommendations",
|
||||
"evidence_quote": "",
|
||||
"reason": "Learn omtaler «lower-carbon regions» som prinsipp (WAF sustainability, Sustainable AI design) men navngir ingen regioner; ingen learn.microsoft.com-side utpeker Sweden Central eller Norway East som low-carbon — karbonintensitetsdata publiseres utenfor Learn (Emissions Impact Dashboard / carbon optimization), så verken bekreftelse eller motbevis finnes.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/feedback-loops-continuous-improvement.md",
|
||||
"line": 607,
|
||||
"claim": "Sweden Central og Norway East angis som low-carbon Azure-regioner for karbonbevisst deployment.",
|
||||
"disposition": "unsourced"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#13",
|
||||
"judge_verdict": "source_silent",
|
||||
"rule": "",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/machine-learning/concept-azure-machine-learning-v2?view=azureml-api-2",
|
||||
"evidence_quote": "",
|
||||
"reason": "STANDARD_DS3_v2 opptrer kun som generisk eksempelstørrelse for compute cluster/instance, og AutoML-CV-siden anbefaler NCsv3-serien generelt; ingen Learn-side foreskriver disse SKU-ene knyttet til daglig tabular- vs. ukentlig CV-retraining — mappingen er udokumentert, ikke motsagt.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/feedback-loops-continuous-improvement.md",
|
||||
"line": 639,
|
||||
"claim": "Compute-SKU-ene som brukes til retraining i Azure Machine Learning er Standard_DS3_v2 (4 vCPU) for daglig retraining av tabular ML og GPU-SKU-en NC6s_v3 for ukentlig retraining av computer vision-modeller.",
|
||||
"disposition": "unsourced"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#14",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R2",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/machine-learning/concept-plan-manage-cost?view=azureml-api-2",
|
||||
"evidence_quote": "When you create resources for an Azure Machine Learning workspace, you also create resources for other Azure services. They are: Azure Container Registry basic account, Azure Blob Storage (general purpose v2), Azure Key Vault, Azure Monitor ... Each VM is billed per hour that it runs.",
|
||||
"reason": "Den kanoniske «full billing model»-siden beskriver hele faktureringsmodellen uten NOEN edition/tier — «Azure ML Enterprise» finnes ikke (Basic/Enterprise-editionene er utgått); per-use compute-delen stemmer, men den navngitte SKU-en gjør ikke det.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/feedback-loops-continuous-improvement.md",
|
||||
"line": 663,
|
||||
"claim": "Azure ML Enterprise er inkludert i Azure-subscription og faktureres som per-use compute pricing.",
|
||||
"disposition": "outdated"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#16",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R4",
|
||||
"evidence_url": "https://learn.microsoft.com/ai-builder/credit-management",
|
||||
"evidence_quote": "| Power Apps per app | 250 | Maximum = 1,000,000 AI Builder credits per tenant. Per app licenses purchased before November 2022, don't include any credits. |",
|
||||
"reason": "Per user = 500 stemmer (Power Apps Premium/per user), men gjeldende rad gir Per app 250 kreditter — claimets «ingen kreditter inkludert» matcher kun den tidsstemplede legacy-regelen for lisenser kjøpt før november 2022; feedback loop-støtte per lisenstype er dessuten ikke dokumentert.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/feedback-loops-continuous-improvement.md",
|
||||
"line": 670,
|
||||
"claim": "Power Platform-lisensene gir følgende AI Builder-kreditter og feedback loop-støtte: Per User = 500 kreditter/mnd med feedback loop-støtte | Per App = ingen kreditter inkludert og ingen feedback loop-støtte (krever Per User) | AI Builder add-on = kreditter kjøpes ekstra, med feedback loop-støtte.",
|
||||
"disposition": "outdated"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/genaiops-llm-specific-practices.md",
|
||||
"batch": "R7.3",
|
||||
"judged_at": "2026-07-25",
|
||||
"per_file_verdict": "flagged",
|
||||
"claim_count": 17,
|
||||
"verified": null,
|
||||
"verified_by": null,
|
||||
"flags": [
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#5",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R8",
|
||||
"evidence_url": "https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-information-retrieval",
|
||||
"evidence_quote": "Semantic ranking works as a secondary reranking step after the initial BM25 or hybrid search.",
|
||||
"reason": "Vector, full-text and hybrid are confirmed as search types, but the cited page maps semantic ranker differently - it is a reranking step applied after retrieval, and it is absent from the page's own Search types enumeration (vector, full-text, hybrid, manual multiple queries); one load-bearing part of the taxonomy is wrong.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/genaiops-llm-specific-practices.md",
|
||||
"line": 96,
|
||||
"claim": "Retrieval-metodene i Azure AI Search er vector | full-text | hybrid | semantic ranker.",
|
||||
"disposition": "outdated"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#6",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/foundry/openai/how-to/fine-tuning-cost-management",
|
||||
"evidence_quote": "After your model is trained, you can deploy it using one of the following hosting options: 1. Standard: pay per-token at the same rate as base model Standard deployments with an additional $1.70/hour hosting fee. Offers regional data residency guarantees.",
|
||||
"reason": "The source lists four production hosting options for fine-tuned models (Standard, Global Standard, Regional Provisioned Throughput, Developer Tier) where Standard and Global Standard carry an SLA, so PTU is one option among several, not a requirement for production use.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/genaiops-llm-specific-practices.md",
|
||||
"line": 117,
|
||||
"claim": "Fine-tuning krever PTU (provisioned throughput) for produksjonsbruk.",
|
||||
"disposition": "outdated"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#13",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R3",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/foundry/concepts/deployments-overview",
|
||||
"evidence_quote": "Foundry provides two deployment options: 1. Standard deployment in Foundry resources - For Foundry Models ... This option is the preferred and most capable deployment path. 2. Managed compute deployment (preview) - Available for all Open Source Software (OSS) models, including models from partner and community, and custom models.",
|
||||
"reason": "The claim's organizing frame is superseded: current Foundry deployment runs through Standard deployment in Foundry resources or managed compute (billing by token usage or PTU), not via Managed Online Endpoints, which the Prompt Flow migration page treats as an Azure Machine Learning construct with no direct Foundry equivalent.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/genaiops-llm-specific-practices.md",
|
||||
"line": 164,
|
||||
"claim": "Deployment i Microsoft Foundry skjer via Managed Online Endpoints med alternativene serverless | PTU | PAYG.",
|
||||
"disposition": "outdated"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#16",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R7",
|
||||
"evidence_url": "https://learn.microsoft.com/en-us/azure/api-management/azure-openai-token-limit-policy",
|
||||
"evidence_quote": "Limit large language model API token usage ... The llm-token-limit policy prevents large language model (LLM) API usage spikes on a per key basis by limiting consumption of language model tokens to either a specified rate (number per minute), a quota over a specified period, or both.",
|
||||
"reason": "The zone-redundant/multi-region part holds, but the load-bearing policy name does not: the URL for the Azure OpenAI-named policy now serves the renamed policy Limit large language model API token usage (llm-token-limit), and the API Management policy reference AI gateway table lists only Limit large language model API token usage and Emit large language model API token metrics - no policy named Limit Azure OpenAI API token usage.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/genaiops-llm-specific-practices.md",
|
||||
"line": 211,
|
||||
"claim": "Azure API Management har innebygde Azure OpenAI-policyer «Limit Azure OpenAI API token usage» | «Emit metrics for consumption», og er sone-redundant og multi-region.",
|
||||
"disposition": "outdated"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/governance-audit-ml-operations.md",
|
||||
"batch": "R7.3",
|
||||
"judged_at": "2026-07-25",
|
||||
"per_file_verdict": "flagged",
|
||||
"claim_count": 23,
|
||||
"verified": null,
|
||||
"verified_by": null,
|
||||
"flags": [
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#19",
|
||||
"judge_verdict": "source_silent",
|
||||
"rule": "",
|
||||
"evidence_url": "https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/",
|
||||
"evidence_quote": "",
|
||||
"reason": "Ingen learn.microsoft.com-side sier at Unity Catalog er inkludert i DBU-forbruket uten ekstra kostnad; den siterte siden omtaler kun gratis systemtabeller, og Databricks-prising ligger paa den JS-rendrede azure.microsoft.com/pricing-siden.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/governance-audit-ml-operations.md",
|
||||
"line": 411,
|
||||
"claim": "Databricks Unity Catalog er inkludert i DBU-forbruket uten ekstra kostnad.",
|
||||
"disposition": "unsourced"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#22",
|
||||
"judge_verdict": "source_silent",
|
||||
"rule": "",
|
||||
"evidence_url": "https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-llm-logs",
|
||||
"evidence_quote": "",
|
||||
"reason": "Verken den siterte siden eller genai-gateway-capabilities uttaler seg om lisens-/kostnadsinkludering av LLM-logging; APIM-prising ligger paa den JS-rendrede azure.microsoft.com/pricing-siden (genai-siden sier kun at AI gateway ikke er et separat tilbud og at kapabilitet varierer per tier - ikke en prisingspaastand).",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/governance-audit-ml-operations.md",
|
||||
"line": 417,
|
||||
"claim": "LLM-logging er inkludert i Azure API Management-lisensen.",
|
||||
"disposition": "unsourced"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#23",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R7",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/databricks/ldp/concepts/where-is-dlt",
|
||||
"evidence_quote": "The product formerly known as Delta Live Tables (DLT) has been updated to Lakeflow pipelines.",
|
||||
"reason": "Den kanoniske rename-siden sier live at DLT er oppdatert til Lakeflow pipelines, ikke Lakeflow Spark Declarative Pipelines; python-ref bekrefter: In Databricks documentation, the Databricks product is called Lakeflow pipelines, and the open-source framework it extends is Apache Spark Declarative Pipelines (SDP) - selve navnet er paastanden (R7 load-bearing carve-out), saa eksakt-verdi-regelen gjelder fullt ut.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/governance-audit-ml-operations.md",
|
||||
"line": 506,
|
||||
"claim": "Delta Live Tables er omdøpt til Lakeflow Spark Declarative Pipelines i Databricks-dokumentasjonen.",
|
||||
"disposition": "outdated"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/inferencing-optimization-caching.md",
|
||||
"batch": "R7.3",
|
||||
"judged_at": "2026-07-25",
|
||||
"per_file_verdict": "flagged",
|
||||
"claim_count": 31,
|
||||
"verified": null,
|
||||
"verified_by": null,
|
||||
"flags": [
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#19",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R2",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/foundry/openai/how-to/batch",
|
||||
"evidence_quote": "endpoint=\"/chat/completions\", # While passing this parameter is required, the system will read your input file to determine if the chat completions or responses API is needed.",
|
||||
"reason": "The canonical enumerating page supports only chat completions and the Responses API; its Global Batch and Data Zone Batch model-availability tables list solely chat/reasoning models (gpt-5.x, gpt-4.1, gpt-4o, o3/o4-mini) with no embedding models, and the FAQ entry 'Can I use the batch API for embeddings models?' answers with a non-support statement - so the claimed embeddings and legacy completions support is absent.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/inferencing-optimization-caching.md",
|
||||
"line": 384,
|
||||
"claim": "Azure OpenAI Batch API støtter chat completions | embeddings | completions.",
|
||||
"disposition": "outdated"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#20",
|
||||
"judge_verdict": "source_silent",
|
||||
"rule": "",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/machine-learning/reference-managed-online-endpoints-vm-sku-list?view=azureml-api-2",
|
||||
"evidence_quote": "",
|
||||
"reason": "No learn.microsoft.com page states this workload-to-SKU recommendation; Standard_DS3_v2, Standard_NC6s_v3 and Standard_NC24s_v3 appear in the supported managed-online-endpoint SKU list and a deployment example, but nothing recommends them for small tabular / vision / large language model workloads, and PTU is an Azure OpenAI concept rather than an AML compute SKU.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/inferencing-optimization-caching.md",
|
||||
"line": 560,
|
||||
"claim": "Anbefalte compute-SKU-er for inference er Standard_DS3_v2 (små tabulære modeller, CPU) | Standard_NC6s_v3 (deep learning vision, GPU) | Standard_NC24s_v3 eller PTU (store språkmodeller).",
|
||||
"disposition": "unsourced"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#21",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R3",
|
||||
"evidence_url": "https://learn.microsoft.com/en-us/azure/machine-learning/concept-endpoints?view=azureml-api-2",
|
||||
"evidence_quote": "Azure Machine Learning supports standard deployments, online endpoints, and batch endpoints.",
|
||||
"reason": "The claim's three-way frame is superseded: the current three top-level options are standard deployments, online endpoints and batch endpoints; Kubernetes is documented as a compute type ('Compute type | None (serverless) | Managed compute and Kubernetes | Managed compute and Kubernetes'), not a third endpoint category, and standard deployments are missing from the claim's set.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/inferencing-optimization-caching.md",
|
||||
"line": 623,
|
||||
"claim": "Azure Machine Learning tilbyr tre deployment-alternativer: Managed Online Endpoints | Batch Endpoints | Kubernetes Endpoints.",
|
||||
"disposition": "outdated"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#25",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R4",
|
||||
"evidence_url": "https://learn.microsoft.com/en-us/azure/azure-sql-edge/onnx-overview",
|
||||
"evidence_quote": "Azure SQL Edge is retired as of September 30, 2025.",
|
||||
"reason": "The page does document 'native scoring with the PREDICT T-SQL function', but the current effective state of the product is retirement, so presenting Azure SQL Edge in the present tense as an ONNX inferencing option matches a superseded state rather than today's.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/inferencing-optimization-caching.md",
|
||||
"line": 697,
|
||||
"claim": "Azure SQL Edge bruker T-SQL-funksjonen PREDICT for native ONNX-scoring uten eksterne API-kall.",
|
||||
"disposition": "outdated"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#27",
|
||||
"judge_verdict": "source_silent",
|
||||
"rule": "",
|
||||
"evidence_url": "https://learn.microsoft.com/en-us/azure/machine-learning/concept-onnx?view=azureml-api-2",
|
||||
"evidence_quote": "",
|
||||
"reason": "The cited ONNX page states no license, and a docs search surfaced no learn.microsoft.com page stating ONNX Runtime's MIT license or commercial-use terms; that licensing information lives in the GitHub repository, not on Learn, so the claim can be neither confirmed nor refuted here.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/inferencing-optimization-caching.md",
|
||||
"line": 827,
|
||||
"claim": "ONNX Runtime er lisensiert under MIT License og er gratis til kommersiell bruk.",
|
||||
"disposition": "unsourced"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#28",
|
||||
"judge_verdict": "source_silent",
|
||||
"rule": "",
|
||||
"evidence_url": "https://learn.microsoft.com/windows/ai/new-windows-ml/overview",
|
||||
"evidence_quote": "",
|
||||
"reason": "Learn documents Windows ML as built into Windows and serviced through Windows Update, but no learn.microsoft.com page states any licensing or cost term for it, so the claim's load-bearing cost assertion ('uten ekstra lisenskostnad') cannot be confirmed or refuted.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/inferencing-optimization-caching.md",
|
||||
"line": 832,
|
||||
"claim": "Windows ML er inkludert i Windows uten ekstra lisenskostnad.",
|
||||
"disposition": "unsourced"
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#29",
|
||||
"judge_verdict": "not_grounded",
|
||||
"rule": "R8",
|
||||
"evidence_url": "https://learn.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models?view=azureml-api-2",
|
||||
"evidence_quote": "Image classification (multi-class and multi-label)",
|
||||
"reason": "The canonical AutoML task-type enumeration is image classification, image classification multi-label, image object detection and image instance segmentation; 'binær' is absent as a task variety and the claim's parenthetical displaces multi-label, which the cited ONNX article itself supports with its own Multi-label image classification tab, so a stated load-bearing part is wrong.",
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/inferencing-optimization-caching.md",
|
||||
"line": 1037,
|
||||
"claim": "AutoML computer vision-oppgaver med ONNX-støtte er: image classification (binær og multi-class) | object detection | instance segmentation.",
|
||||
"disposition": "outdated"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"file": "skills/ms-ai-engineering/references/mlops-genaiops/llm-evaluation-production.md",
|
||||
"batch": "R7.1",
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue