ms-ai-architect/skills/ms-ai-engineering/references/data-engineering/feature-stores-engineering.md

# Feature Stores and Feature Engineering

**Last updated:** 2026-06-24
**Status:** GA
**Category:** Data Engineering for AI
**Type:** reference
**Source:** https://learn.microsoft.com/azure/machine-learning/concept-what-is-managed-feature-store

---

## Innhold

- [Introduksjon](#introduksjon)
- [Feature Definition and Storage in Silver Layer](#feature-definition-and-storage-in-silver-layer)
- [Point-in-Time Lookups for Training](#point-in-time-lookups-for-training)
- [Feature Freshness and Refresh Cadences](#feature-freshness-and-refresh-cadences)
- [Data Wrangler for Exploratory Feature Engineering](#data-wrangler-for-exploratory-feature-engineering)
- [Feature Monitoring and Drift Detection](#feature-monitoring-and-drift-detection)
- [Referanser](#referanser)
- [For Cosmo](#for-cosmo)

## Introduksjon

Feature stores er et sentralt mønster i moderne MLOps som løser problemet med feature-gjenbruk, konsistens mellom trening og inferens, og operasjonalisering av feature-pipelines. Azure Machine Learning Managed Feature Store og Microsoft Fabric Data Science gir en komplett plattform for å definere, materialisere, dele og overvåke features på tvers av ML-prosjekter.

For norsk offentlig sektor innebærer feature store-tilnærmingen at data science-team kan dele beregninger på tvers av prosjekter -- for eksempel kan trafikkdata-features brukes både for ulykkesprediksjonsmodeller og køvarslingsmodeller uten redundant feature engineering. Dette reduserer kostnader, forbedrer konsistens og forkorter tid fra eksperimentering til produksjon.

Denne referansen dekker feature-definisjon og lagring, point-in-time lookups for trening, feature-oppdateringsstrategier, Data Wrangler for utforskende feature engineering, og overvåking av feature-kvalitet og drift.

---

## Feature Definition and Storage in Silver Layer

### Feature Store Arkitektur

```
┌──────────────────────────────────────────────────────────────┐
│                    Azure ML Managed Feature Store              │
│  ┌──────────────┐  ┌──────────────────┐  ┌───────────────┐  │
│  │ Feature Set  │  │ Materialization   │  │ Feature       │  │
│  │ Specification│  │ Store (ADLS Gen2) │  │ Retrieval     │  │
│  │              │  │ Offline + Online  │  │ Component     │  │
│  └──────┬───────┘  └────────┬─────────┘  └───────┬───────┘  │
│         │                   │                     │          │
│         └───────────────────┴─────────────────────┘          │
└──────────────────────────────┬───────────────────────────────┘
                               │
┌──────────────────────────────▼───────────────────────────────┐
│                    Microsoft Fabric                            │
│  ┌──────────┐  ┌──────────────┐  ┌────────────────────────┐ │
│  │ Bronze   │  │ Silver Layer │  │ Gold Layer              │ │
│  │ (raw)    │──│ (features)   │──│ (training datasets)    │ │
│  └──────────┘  └──────────────┘  └────────────────────────┘ │
└──────────────────────────────────────────────────────────────┘
```

### Feature Set Specification

En feature set-spesifikasjon definerer features og valgfri transformasjonslogikk:

```python
# Feature set specification (YAML)
# feature_set_spec/transactions/spec.yaml
"""
name: transactions
version: "1"
description: "Customer transaction features"
entities:
  - name: customer
    version: "1"
    join_keys:
      - customer_id
source:
  type: parquet
  path: "abfss://silver@onelake.dfs.fabric.microsoft.com/transactions"
  timestamp_column: transaction_date
features:
  - name: transaction_7day_count
    type: integer
    description: "Number of transactions in last 7 days"
  - name: transaction_7day_sum
    type: double
    description: "Total transaction amount in last 7 days"
  - name: transaction_30day_avg
    type: double
    description: "Average transaction amount in last 30 days"
"""
```

### Feature Transformations med PySpark

```python
from pyspark.sql import functions as F
from pyspark.sql.window import Window

# Kildedataer fra Silver layer
transactions = spark.read.format("delta").table("silver.transactions")

# Definer vindus-spesifikasjoner
window_7d = (
    Window
    .partitionBy("customer_id")
    .orderBy(F.col("transaction_date").cast("long"))
    .rangeBetween(-7 * 86400, 0)  # 7 dager i sekunder
)

window_30d = (
    Window
    .partitionBy("customer_id")
    .orderBy(F.col("transaction_date").cast("long"))
    .rangeBetween(-30 * 86400, 0)
)

# Beregn features
customer_features = (
    transactions
    .withColumn("txn_7d_count", F.count("*").over(window_7d))
    .withColumn("txn_7d_sum", F.sum("amount").over(window_7d))
    .withColumn("txn_30d_avg", F.avg("amount").over(window_30d))
    .withColumn("txn_30d_max", F.max("amount").over(window_30d))
    .withColumn("days_since_last_txn",
        F.datediff(F.current_date(), F.max("transaction_date").over(
            Window.partitionBy("customer_id"))))
)

# Lagre features i Silver layer
customer_features.write.format("delta") \
    .mode("overwrite") \
    .saveAsTable("silver.customer_transaction_features")
```

### Feature-lagring i Medallion Architecture

| Lag | Innhold | Oppdateringsfrekvens |
|---|---|---|
| **Bronze** | Råtransaksjoner fra Dataverse/kildesystemer | Sanntid / daglig |
| **Silver** | Feature-beregninger (aggregater, vindus-funksjoner) | Daglig / per time |
| **Gold** | Ferdige treningsdatasett (features + labels) | Ved behov |
| **Feature Store** | Registrerte, versjonerte features | Materialisert etter plan |

---

## Point-in-Time Lookups for Training

### Temporal Joins (tidsreise-joiner)

Point-in-time lookups er kritisk for å unngå datalekasje i ML-trening:

```python
# FEIL: Standard join inkluderer fremtidige data (data leakage!)
# features_at_prediction_time = features.join(labels, "customer_id")

# RIKTIG: Point-in-time join
from pyspark.sql.functions import col

# Observations: tidspunkter der vi vil ha features
observations = spark.createDataFrame([
    ("C001", "2026-01-15"),
    ("C002", "2026-01-20"),
    ("C003", "2026-02-01")
], ["customer_id", "observation_date"])

# Features: tidsseriedata
features = spark.read.format("delta").table("silver.customer_transaction_features")

# Point-in-time join: hent features som var gjeldende PÅ observation_date
pit_features = (
    observations.alias("obs")
    .join(
        features.alias("feat"),
        (col("obs.customer_id") == col("feat.customer_id")) &
        (col("feat.feature_date") <= col("obs.observation_date")),
        "left"
    )
    .withColumn("rank", F.row_number().over(
        Window
        .partitionBy("obs.customer_id", "obs.observation_date")
        .orderBy(F.desc("feat.feature_date"))
    ))
    .filter(col("rank") == 1)  # Siste feature-verdi FØR observation_date
    .drop("rank")
)
```

### Azure ML Feature Retrieval Component

```python
# Deklarativ feature retrieval i Azure ML pipeline
from azure.ai.ml import MLClient
from azure.ai.ml.entities import FeatureRetrievalSpec

# Definer feature retrieval spec
feature_retrieval_spec = FeatureRetrievalSpec(
    feature_store_name="my-feature-store",
    features=[
        {
            "feature_set": "transactions:1",
            "features": ["txn_7d_count", "txn_7d_sum", "txn_30d_avg"]
        },
        {
            "feature_set": "demographics:1",
            "features": ["age_group", "region", "income_band"]
        }
    ]
)

# Feature retrieval støtter automatisk point-in-time joins
# basert på timestamp-kolonne i feature set specification
```

---

## Feature Freshness and Refresh Cadences

### Materialiseringsstrategi

| Feature-type | Oppdateringsfrekvens | Materialiseringsmetode |
|---|---|---|
| **Statiske** (demografi) | Ukentlig / månedlig | Batch materialisering |
| **Langsom endring** (score) | Daglig | Scheduled materialisering |
| **Rask endring** (transaksjoner) | Per time / sanntid | Streaming + backfill |
| **Sanntid** (lokasjon) | Kontinuerlig | Online store (Redis) |

### Materialiserings-oppsett

```python
from azure.ai.ml.entities import (
    MaterializationSettings,
    MaterializationComputeResource,
    RecurrenceTrigger
)

# Konfigurer materialisering for en feature set
materialization = MaterializationSettings(
    schedule=RecurrenceTrigger(
        interval=1,
        frequency="Day",
        time_of_day="02:00"  # Kjør kl 02:00 UTC
    ),
    resource=MaterializationComputeResource(
        instance_type="standard_e4s_v3"
    ),
    spark_configuration={
        "spark.driver.cores": 4,
        "spark.driver.memory": "36g",
        "spark.executor.cores": 4,
        "spark.executor.memory": "36g"
    }
)

# Backfill for historisk data
from azure.ai.ml import MLClient

fs_client = MLClient(credential, subscription_id, resource_group, feature_store_name)
poller = fs_client.feature_sets.begin_backfill(
    name="transactions",
    version="1",
    feature_window_start_time="2025-01-01T00:00:00Z",
    feature_window_end_time="2026-02-11T00:00:00Z",
    data_status=["None", "Incomplete"]
)

# Stream jobb-logger
fs_client.jobs.stream(poller.result().job_ids[0])
```

### Online vs. Offline Materialization

| Aspekt | Offline Store (ADLS Gen2) | Online Store (Redis) |
|---|---|---|
| **Bruksområde** | Trening, batch-inferens | Real-time inferens |
| **Latens** | Sekunder-minutter | Millisekunder |
| **Volum** | Ubegrenset | Begrenset av Redis-minne |
| **Kostnad** | Lav (lagring) | Høyere (compute) |
| **Format** | Delta/Parquet | Key-value |

---

## Data Wrangler for Exploratory Feature Engineering

### Data Wrangler i Fabric

Data Wrangler er et notebook-basert verktøy for visuell datautforsking og feature engineering:

```python
# Steg 1: Last data i Notebook
import pandas as pd

df = spark.read.format("delta").table("silver.customer_data").toPandas()

# Steg 2: Start Data Wrangler
# Klikk "Data" > "Launch Data Wrangler" i Notebook-menyen
# Velg DataFrame "df"

# Steg 3: Data Wrangler UI tilbyr:
# - Grid-visning med statistikk per kolonne
# - Innebygde visualiseringer (histogrammer, scatter plots)
# - Over 300 transformasjoner
# - AI-drevne forslag (PROSE)
# - Copilot for naturlig språk → kode

# Steg 4: Eksporter kode tilbake til Notebook
```

### Vanlige feature engineering-operasjoner i Data Wrangler

| Operasjon | Eksempel | Autogenerert kode |
|---|---|---|
| **One-hot encoding** | Kategoriske variabler | `pd.get_dummies(df, columns=[...])` |
| **Binning** | Aldersgrupper | `pd.cut(df['age'], bins=[...])` |
| **Missing values** | Imputering | `df['col'].fillna(df['col'].median())` |
| **Standardisering** | Z-score | `(df['col'] - mean) / std` |
| **Feature crossing** | Kombinasjoner | `df['new'] = df['a'] * df['b']` |
| **Dato-features** | Dag, uke, måned | `df['month'] = df['date'].dt.month` |

### PySpark Feature Engineering Templates

```python
from pyspark.sql import functions as F
from pyspark.ml.feature import VectorAssembler, StandardScaler, StringIndexer

# Kategorisk encoding
indexer = StringIndexer(inputCol="region", outputCol="region_index")

# Numerisk standardisering
assembler = VectorAssembler(
    inputCols=["age", "income", "txn_count"],
    outputCol="features_raw"
)
scaler = StandardScaler(
    inputCol="features_raw",
    outputCol="features_scaled",
    withStd=True,
    withMean=True
)

# Dato-baserte features
df_features = (
    df
    .withColumn("day_of_week", F.dayofweek("event_date"))
    .withColumn("month", F.month("event_date"))
    .withColumn("is_weekend", F.when(
        F.dayofweek("event_date").isin([1, 7]), 1).otherwise(0))
    .withColumn("hour_of_day", F.hour("event_timestamp"))
    .withColumn("days_since_registration",
        F.datediff(F.current_date(), "registration_date"))
)
```

---

## Feature Monitoring and Drift Detection

### Feature Drift-typer

| Drift-type | Beskrivelse | Deteksjonsmetode |
|---|---|---|
| **Data drift** | Endring i feature-distribusjon | KS-test, PSI |
| **Concept drift** | Endring i forholdet mellom features og target | Modell-ytelse over tid |
| **Schema drift** | Endring i datastruktur | Schema-validering |
| **Freshness drift** | Data er ikke oppdatert | Timestamp-sjekk |

### Monitoring i Azure ML Feature Store

```python
from azure.ai.ml.entities import (
    FeatureSetMonitoringSpec,
    MonitorSignal
)

# Konfigurer feature-monitoring
monitoring = FeatureSetMonitoringSpec(
    signal=MonitorSignal(
        feature_data_type_override={
            "txn_7d_count": "numerical",
            "region": "categorical"
        },
        metric_thresholds={
            "numerical": {
                "jensen_shannon_distance": 0.1,
                "population_stability_index": 0.2
            },
            "categorical": {
                "jensen_shannon_distance": 0.1
            }
        }
    ),
    notification_emails=["team@example.no"]
)
```

### Manuell drift-deteksjon i Fabric Notebook

```python
from scipy.stats import ks_2samp
import numpy as np

def detect_feature_drift(reference_df, current_df, features, threshold=0.05):
    """Detekter feature drift mellom referanse- og nåværende data."""
    drift_report = {}

    for feature in features:
        ref_values = reference_df[feature].dropna().values
        curr_values = current_df[feature].dropna().values

        # Kolmogorov-Smirnov test
        stat, p_value = ks_2samp(ref_values, curr_values)

        # Population Stability Index (PSI)
        psi = calculate_psi(ref_values, curr_values, buckets=10)

        drift_report[feature] = {
            "ks_statistic": round(stat, 4),
            "ks_p_value": round(p_value, 4),
            "psi": round(psi, 4),
            "drifted": p_value < threshold or psi > 0.2
        }

    return drift_report

def calculate_psi(reference, current, buckets=10):
    """Beregn Population Stability Index."""
    breakpoints = np.linspace(
        min(reference.min(), current.min()),
        max(reference.max(), current.max()),
        buckets + 1
    )
    ref_counts = np.histogram(reference, breakpoints)[0] / len(reference)
    curr_counts = np.histogram(current, breakpoints)[0] / len(current)

    # Unngå log(0)
    ref_counts = np.clip(ref_counts, 0.001, None)
    curr_counts = np.clip(curr_counts, 0.001, None)

    psi = np.sum((curr_counts - ref_counts) * np.log(curr_counts / ref_counts))
    return psi
```

---

## Referanser

- [What is managed feature store?](https://learn.microsoft.com/en-us/azure/machine-learning/concept-what-is-managed-feature-store) -- Konseptoversikt
- [What is a Feature Store? (AI Playbook)](https://learn.microsoft.com/en-us/ai/playbook/capabilities/model-development/feature-management/) -- Arkitektur og implementasjon
- [Tutorial 1: Develop and register a feature set](https://learn.microsoft.com/en-us/azure/machine-learning/tutorial-get-started-with-feature-store) -- Hands-on tutorial
- [Tutorial 4: Enable online materialization](https://learn.microsoft.com/en-us/azure/machine-learning/tutorial-online-materialization-inference) -- Online feature serving
- [Manage access control for managed feature store](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-setup-access-control-feature-store) -- RBAC og sikkerhet
- [Accelerate data prep with Data Wrangler](https://learn.microsoft.com/en-us/fabric/data-science/data-wrangler) -- Data Wrangler guide
- [Automated ML in Fabric](https://learn.microsoft.com/en-us/fabric/data-science/automated-ml-fabric) -- AutoML med feature engineering

---

## For Cosmo

- **Bruk denne referansen** når brukeren planlegger ML-infrastruktur, trenger feature-gjenbruk på tvers av prosjekter, eller ønsker å operasjonalisere feature engineering.
- Anbefal **Azure ML Managed Feature Store** for organisasjoner med flere ML-team som trenger å dele features. For enkeltprosjekter er **Delta-tabeller i Silver layer** ofte tilstrekkelig.
- **Point-in-time lookups er ikke-forhandlingsbart** for tidsserie-features -- uten dette vil modeller lekke fremtidig informasjon og vise urealistisk god ytelse i testing.
- For norsk offentlig sektor: Feature stores muliggjør **sentral styring** av beregninger som brukes på tvers av etater -- Direktoratet for digital tjenesteutvikling kan dele trafikkfeatures med andre transportetater via feature store-deling.
- Start med **Data Wrangler** for utforskende feature engineering, deretter formaliser i feature set-spesifikasjoner når features er validert og skal til produksjon.