refactor(examples): replace sector-specific example material with generic, fictitious examples

Reference files, test fixtures, the playground demo project and one design
document now use generic, fictitious examples (buildings, energy, water,
grants, municipal services). The playground demo (17 fixtures plus the
embedded demo state) tells one consistent story: a municipal customer
chatbot that pre-screens housing-benefit applications, classified under
Annex III point 5(a). The embedded demo copies were edited in place rather
than regenerated, because they already carry newer AI Act dates than the
fixture files.

Legal text is unchanged. Test semantics are unchanged. Four dark-theme
onboarding screenshots with outdated placeholder text are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-09-23 14:03:53 +02:00

17 KiB

Raw Blame History

Feature Stores and Feature Engineering

Last updated: 2026-06-24 Status: GA Category: Data Engineering for AI Type: reference Source: https://learn.microsoft.com/azure/machine-learning/concept-what-is-managed-feature-store

Innhold

Introduksjon
Feature Definition and Storage in Silver Layer
Point-in-Time Lookups for Training
Feature Freshness and Refresh Cadences
Data Wrangler for Exploratory Feature Engineering
Feature Monitoring and Drift Detection
Referanser
For arkitekten

Introduksjon

Feature stores er et sentralt mønster i moderne MLOps som løser problemet med feature-gjenbruk, konsistens mellom trening og inferens, og operasjonalisering av feature-pipelines. Azure Machine Learning Managed Feature Store og Microsoft Fabric Data Science gir en komplett plattform for å definere, materialisere, dele og overvåke features på tvers av ML-prosjekter.

For norsk offentlig sektor innebærer feature store-tilnærmingen at data science-team kan dele beregninger på tvers av prosjekter -- for eksempel kan energidata-features brukes både for forbruksprognoser og feildeteksjonsmodeller uten redundant feature engineering. Dette reduserer kostnader, forbedrer konsistens og forkorter tid fra eksperimentering til produksjon.

Denne referansen dekker feature-definisjon og lagring, point-in-time lookups for trening, feature-oppdateringsstrategier, Data Wrangler for utforskende feature engineering, og overvåking av feature-kvalitet og drift.

Feature Definition and Storage in Silver Layer

Feature Store Arkitektur

┌──────────────────────────────────────────────────────────────┐
│                    Azure ML Managed Feature Store              │
│  ┌──────────────┐  ┌──────────────────┐  ┌───────────────┐  │
│  │ Feature Set  │  │ Materialization   │  │ Feature       │  │
│  │ Specification│  │ Store (ADLS Gen2) │  │ Retrieval     │  │
│  │              │  │ Offline + Online  │  │ Component     │  │
│  └──────┬───────┘  └────────┬─────────┘  └───────┬───────┘  │
│         │                   │                     │          │
│         └───────────────────┴─────────────────────┘          │
└──────────────────────────────┬───────────────────────────────┘
                               │
┌──────────────────────────────▼───────────────────────────────┐
│                    Microsoft Fabric                            │
│  ┌──────────┐  ┌──────────────┐  ┌────────────────────────┐ │
│  │ Bronze   │  │ Silver Layer │  │ Gold Layer              │ │
│  │ (raw)    │──│ (features)   │──│ (training datasets)    │ │
│  └──────────┘  └──────────────┘  └────────────────────────┘ │
└──────────────────────────────────────────────────────────────┘

Feature Set Specification

En feature set-spesifikasjon definerer features og valgfri transformasjonslogikk:

# Feature set specification (YAML)
# feature_set_spec/transactions/spec.yaml
"""
name: transactions
version: "1"
description: "Customer transaction features"
entities:
  - name: customer
    version: "1"
    join_keys:
      - customer_id
source:
  type: parquet
  path: "abfss://silver@onelake.dfs.fabric.microsoft.com/transactions"
  timestamp_column: transaction_date
features:
  - name: transaction_7day_count
    type: integer
    description: "Number of transactions in last 7 days"
  - name: transaction_7day_sum
    type: double
    description: "Total transaction amount in last 7 days"
  - name: transaction_30day_avg
    type: double
    description: "Average transaction amount in last 30 days"
"""

Feature Transformations med PySpark

from pyspark.sql import functions as F
from pyspark.sql.window import Window

# Kildedataer fra Silver layer
transactions = spark.read.format("delta").table("silver.transactions")

# Definer vindus-spesifikasjoner
window_7d = (
    Window
    .partitionBy("customer_id")
    .orderBy(F.col("transaction_date").cast("long"))
    .rangeBetween(-7 * 86400, 0)  # 7 dager i sekunder
)

window_30d = (
    Window
    .partitionBy("customer_id")
    .orderBy(F.col("transaction_date").cast("long"))
    .rangeBetween(-30 * 86400, 0)
)

# Beregn features
customer_features = (
    transactions
    .withColumn("txn_7d_count", F.count("*").over(window_7d))
    .withColumn("txn_7d_sum", F.sum("amount").over(window_7d))
    .withColumn("txn_30d_avg", F.avg("amount").over(window_30d))
    .withColumn("txn_30d_max", F.max("amount").over(window_30d))
    .withColumn("days_since_last_txn",
        F.datediff(F.current_date(), F.max("transaction_date").over(
            Window.partitionBy("customer_id"))))
)

# Lagre features i Silver layer
customer_features.write.format("delta") \
    .mode("overwrite") \
    .saveAsTable("silver.customer_transaction_features")

Feature-lagring i Medallion Architecture

Lag	Innhold	Oppdateringsfrekvens
Bronze	Råtransaksjoner fra Dataverse/kildesystemer	Sanntid / daglig
Silver	Feature-beregninger (aggregater, vindus-funksjoner)	Daglig / per time
Gold	Ferdige treningsdatasett (features + labels)	Ved behov
Feature Store	Registrerte, versjonerte features	Materialisert etter plan

Point-in-Time Lookups for Training

Temporal Joins (tidsreise-joiner)

Point-in-time lookups er kritisk for å unngå datalekasje i ML-trening:

# FEIL: Standard join inkluderer fremtidige data (data leakage!)
# features_at_prediction_time = features.join(labels, "customer_id")

# RIKTIG: Point-in-time join
from pyspark.sql.functions import col

# Observations: tidspunkter der vi vil ha features
observations = spark.createDataFrame([
    ("C001", "2026-01-15"),
    ("C002", "2026-01-20"),
    ("C003", "2026-02-01")
], ["customer_id", "observation_date"])

# Features: tidsseriedata
features = spark.read.format("delta").table("silver.customer_transaction_features")

# Point-in-time join: hent features som var gjeldende PÅ observation_date
pit_features = (
    observations.alias("obs")
    .join(
        features.alias("feat"),
        (col("obs.customer_id") == col("feat.customer_id")) &
        (col("feat.feature_date") <= col("obs.observation_date")),
        "left"
    )
    .withColumn("rank", F.row_number().over(
        Window
        .partitionBy("obs.customer_id", "obs.observation_date")
        .orderBy(F.desc("feat.feature_date"))
    ))
    .filter(col("rank") == 1)  # Siste feature-verdi FØR observation_date
    .drop("rank")
)

Azure ML Feature Retrieval Component

# Deklarativ feature retrieval i Azure ML pipeline
from azure.ai.ml import MLClient
from azure.ai.ml.entities import FeatureRetrievalSpec

# Definer feature retrieval spec
feature_retrieval_spec = FeatureRetrievalSpec(
    feature_store_name="my-feature-store",
    features=[
        {
            "feature_set": "transactions:1",
            "features": ["txn_7d_count", "txn_7d_sum", "txn_30d_avg"]
        },
        {
            "feature_set": "demographics:1",
            "features": ["age_group", "region", "income_band"]
        }
    ]
)

# Feature retrieval støtter automatisk point-in-time joins
# basert på timestamp-kolonne i feature set specification

Feature Freshness and Refresh Cadences

Materialiseringsstrategi

Feature-type	Oppdateringsfrekvens	Materialiseringsmetode
Statiske (demografi)	Ukentlig / månedlig	Batch materialisering
Langsom endring (score)	Daglig	Scheduled materialisering
Rask endring (transaksjoner)	Per time / sanntid	Streaming + backfill
Sanntid (lokasjon)	Kontinuerlig	Online store (Redis)

Materialiserings-oppsett

from azure.ai.ml.entities import (
    MaterializationSettings,
    MaterializationComputeResource,
    RecurrenceTrigger
)

# Konfigurer materialisering for en feature set
materialization = MaterializationSettings(
    schedule=RecurrenceTrigger(
        interval=1,
        frequency="Day",
        time_of_day="02:00"  # Kjør kl 02:00 UTC
    ),
    resource=MaterializationComputeResource(
        instance_type="standard_e4s_v3"
    ),
    spark_configuration={
        "spark.driver.cores": 4,
        "spark.driver.memory": "36g",
        "spark.executor.cores": 4,
        "spark.executor.memory": "36g"
    }
)

# Backfill for historisk data
from azure.ai.ml import MLClient

fs_client = MLClient(credential, subscription_id, resource_group, feature_store_name)
poller = fs_client.feature_sets.begin_backfill(
    name="transactions",
    version="1",
    feature_window_start_time="2025-01-01T00:00:00Z",
    feature_window_end_time="2026-02-11T00:00:00Z",
    data_status=["None", "Incomplete"]
)

# Stream jobb-logger
fs_client.jobs.stream(poller.result().job_ids[0])

Online vs. Offline Materialization

Aspekt	Offline Store (ADLS Gen2)	Online Store (Redis)
Bruksområde	Trening, batch-inferens	Real-time inferens
Latens	Sekunder-minutter	Millisekunder
Volum	Ubegrenset	Begrenset av Redis-minne
Kostnad	Lav (lagring)	Høyere (compute)
Format	Delta/Parquet	Key-value

Data Wrangler for Exploratory Feature Engineering

Data Wrangler i Fabric

Data Wrangler er et notebook-basert verktøy for visuell datautforsking og feature engineering:

# Steg 1: Last data i Notebook
import pandas as pd

df = spark.read.format("delta").table("silver.customer_data").toPandas()

# Steg 2: Start Data Wrangler
# Klikk "Data" > "Launch Data Wrangler" i Notebook-menyen
# Velg DataFrame "df"

# Steg 3: Data Wrangler UI tilbyr:
# - Grid-visning med statistikk per kolonne
# - Innebygde visualiseringer (histogrammer, scatter plots)
# - Over 300 transformasjoner
# - AI-drevne forslag (PROSE)
# - Copilot for naturlig språk → kode

# Steg 4: Eksporter kode tilbake til Notebook

Vanlige feature engineering-operasjoner i Data Wrangler

Operasjon	Eksempel	Autogenerert kode
One-hot encoding	Kategoriske variabler	`pd.get_dummies(df, columns=[...])`
Binning	Aldersgrupper	`pd.cut(df['age'], bins=[...])`
Missing values	Imputering	`df['col'].fillna(df['col'].median())`
Standardisering	Z-score	`(df['col'] - mean) / std`
Feature crossing	Kombinasjoner	`df['new'] = df['a'] * df['b']`
Dato-features	Dag, uke, måned	`df['month'] = df['date'].dt.month`

PySpark Feature Engineering Templates

from pyspark.sql import functions as F
from pyspark.ml.feature import VectorAssembler, StandardScaler, StringIndexer

# Kategorisk encoding
indexer = StringIndexer(inputCol="region", outputCol="region_index")

# Numerisk standardisering
assembler = VectorAssembler(
    inputCols=["age", "income", "txn_count"],
    outputCol="features_raw"
)
scaler = StandardScaler(
    inputCol="features_raw",
    outputCol="features_scaled",
    withStd=True,
    withMean=True
)

# Dato-baserte features
df_features = (
    df
    .withColumn("day_of_week", F.dayofweek("event_date"))
    .withColumn("month", F.month("event_date"))
    .withColumn("is_weekend", F.when(
        F.dayofweek("event_date").isin([1, 7]), 1).otherwise(0))
    .withColumn("hour_of_day", F.hour("event_timestamp"))
    .withColumn("days_since_registration",
        F.datediff(F.current_date(), "registration_date"))
)

Feature Monitoring and Drift Detection

Feature Drift-typer

Drift-type	Beskrivelse	Deteksjonsmetode
Data drift	Endring i feature-distribusjon	KS-test, PSI
Concept drift	Endring i forholdet mellom features og target	Modell-ytelse over tid
Schema drift	Endring i datastruktur	Schema-validering
Freshness drift	Data er ikke oppdatert	Timestamp-sjekk

Monitoring i Azure ML Feature Store

from azure.ai.ml.entities import (
    FeatureSetMonitoringSpec,
    MonitorSignal
)

# Konfigurer feature-monitoring
monitoring = FeatureSetMonitoringSpec(
    signal=MonitorSignal(
        feature_data_type_override={
            "txn_7d_count": "numerical",
            "region": "categorical"
        },
        metric_thresholds={
            "numerical": {
                "jensen_shannon_distance": 0.1,
                "population_stability_index": 0.2
            },
            "categorical": {
                "jensen_shannon_distance": 0.1
            }
        }
    ),
    notification_emails=["team@example.no"]
)

Manuell drift-deteksjon i Fabric Notebook

from scipy.stats import ks_2samp
import numpy as np

def detect_feature_drift(reference_df, current_df, features, threshold=0.05):
    """Detekter feature drift mellom referanse- og nåværende data."""
    drift_report = {}

    for feature in features:
        ref_values = reference_df[feature].dropna().values
        curr_values = current_df[feature].dropna().values

        # Kolmogorov-Smirnov test
        stat, p_value = ks_2samp(ref_values, curr_values)

        # Population Stability Index (PSI)
        psi = calculate_psi(ref_values, curr_values, buckets=10)

        drift_report[feature] = {
            "ks_statistic": round(stat, 4),
            "ks_p_value": round(p_value, 4),
            "psi": round(psi, 4),
            "drifted": p_value < threshold or psi > 0.2
        }

    return drift_report

def calculate_psi(reference, current, buckets=10):
    """Beregn Population Stability Index."""
    breakpoints = np.linspace(
        min(reference.min(), current.min()),
        max(reference.max(), current.max()),
        buckets + 1
    )
    ref_counts = np.histogram(reference, breakpoints)[0] / len(reference)
    curr_counts = np.histogram(current, breakpoints)[0] / len(current)

    # Unngå log(0)
    ref_counts = np.clip(ref_counts, 0.001, None)
    curr_counts = np.clip(curr_counts, 0.001, None)

    psi = np.sum((curr_counts - ref_counts) * np.log(curr_counts / ref_counts))
    return psi

Referanser

What is managed feature store? -- Konseptoversikt
What is a Feature Store? (AI Playbook) -- Arkitektur og implementasjon
Tutorial 1: Develop and register a feature set -- Hands-on tutorial
Tutorial 4: Enable online materialization -- Online feature serving
Manage access control for managed feature store -- RBAC og sikkerhet
Accelerate data prep with Data Wrangler -- Data Wrangler guide
Automated ML in Fabric -- AutoML med feature engineering

For arkitekten

Bruk denne referansen når brukeren planlegger ML-infrastruktur, trenger feature-gjenbruk på tvers av prosjekter, eller ønsker å operasjonalisere feature engineering.
Anbefal Azure ML Managed Feature Store for organisasjoner med flere ML-team som trenger å dele features. For enkeltprosjekter er Delta-tabeller i Silver layer ofte tilstrekkelig.
Point-in-time lookups er ikke-forhandlingsbart for tidsserie-features -- uten dette vil modeller lekke fremtidig informasjon og vise urealistisk god ytelse i testing.
For norsk offentlig sektor: Feature stores muliggjør sentral styring av beregninger som brukes på tvers av etater -- Direktoratet for digital tjenesteutvikling kan dele energifeatures med andre etater via feature store-deling.
Start med Data Wrangler for utforskende feature engineering, deretter formaliser i feature set-spesifikasjoner når features er validert og skal til produksjon.

17 KiB Raw Blame History