open/ms-ai-architect
1
0
Fork 0
Code Issues Releases 4 Activity
ms-ai-architect/skills/ms-ai-engineering/references/data-engineering/data-sampling-labeling.md
Kjell Tore Guttormsen 544934dc57
 refactor(examples): replace sector-specific example material with generic, fictitious examples 
Reference files, test fixtures, the playground demo project and one design
document now use generic, fictitious examples (buildings, energy, water,
grants, municipal services). The playground demo (17 fixtures plus the
embedded demo state) tells one consistent story: a municipal customer
chatbot that pre-screens housing-benefit applications, classified under
Annex III point 5(a). The embedded demo copies were edited in place rather
than regenerated, because they already carry newer AI Act dates than the
fixture files.

Legal text is unchanged. Test semantics are unchanged. Four dark-theme
onboarding screenshots with outdated placeholder text are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 14:03:53 +02:00

526 lines
18 KiB
Markdown
Raw Blame History

# Data Sampling and Labeling Strategies
**Last updated:** 2026-06-24
**Status:** GA
**Category:** Data Engineering for AI
**Type:** reference
**Source:** https://learn.microsoft.com/azure/machine-learning/how-to-label-data
---
## Innhold
- [Introduksjon](#introduksjon)
- [Stratified Sampling for Class Balance](#stratified-sampling-for-class-balance)
- [Active Learning and Uncertainty Sampling](#active-learning-and-uncertainty-sampling)
- [Crowdsourcing and Labeling Platforms](#crowdsourcing-and-labeling-platforms)
- [Quality Control and Inter-Rater Agreement](#quality-control-and-inter-rater-agreement)
- [Feedback Loops for Continuous Labeling](#feedback-loops-for-continuous-labeling)
- [Referanser](#referanser)
- [For arkitekten](#for-arkitekten)
## Introduksjon
Kvaliteten pa treningsdata er den viktigste faktoren for ytelsen til ML-modeller. Effektiv datasampling sikrer at treningsdatasettet er representativt og balansert, mens systematisk datamerking (labeling) gir modellene de korrekte signalene a laere fra. Azure Machine Learning tilbyr en komplett plattform for datamerking med stotte for bade bilde- og tekstdata, inkludert ML-assistert merking som akselererer prosessen vesentlig.
For norsk offentlig sektor, der data ofte er ubalansert (f.eks. svaert fa svindeltilfeller vs. legitime transaksjoner, eller sjaeldne hendelser i driftsdata), er stratifisert sampling og aktiv laering spesielt viktig. Riktig sampling reduserer merkebehovet med 50-80%, noe som sparer bade tid og kostnader i prosjekter med stramme budsjetter.
Denne referansen dekker hele livssyklusen fra datautvalg gjennom merkeprosesser til kvalitetskontroll, med fokus pa teknikker som er relevante for Microsoft AI-stakken og Azure Machine Learning.
---
## Stratified Sampling for Class Balance
### Problemet med ubalanserte datasett
| Scenario | Positiv klasse | Negativ klasse | Ubalanse-ratio |
|----------|---------------|----------------|----------------|
| Svindeldeteksjon | 0.1% svindel | 99.9% legitim | 1:1000 |
| Reinnleggelsesprediksjon | 2% reinnlagt | 98% ikke reinnlagt | 1:50 |
| Dokumentklassifisering | 5% sensitiv | 95% ikke-sensitiv | 1:19 |
| Feildeteksjon (IoT) | 0.5% feil | 99.5% normal | 1:200 |
### Stratifisert sampling i PySpark
```python
from pyspark.sql import functions as F
def stratified_sample(df, label_column, sample_fractions, seed=42):
"""
Utfor stratifisert sampling for a balansere klasser.
Args:
df: Input DataFrame
label_column: Kolonnen som inneholder klassen
sample_fractions: Dict med {klasse: samplingandel}
seed: Random seed for reproduserbarhet
"""
sampled = df.sampleBy(label_column, fractions=sample_fractions, seed=seed)
return sampled
# Eksempel: Balanser svindeldatasett
# Original: 99.9% legitim, 0.1% svindel
sample_fractions = {
"legitimate": 0.01, # Sample 1% av legitime (reduser fra 99.9k til ~1k)
"fraud": 1.0 # Behold alle svindeltilfeller (~100)
}
balanced_df = stratified_sample(
df_transactions,
label_column="transaction_type",
sample_fractions=sample_fractions
)
print(f"Original: {df_transactions.count()} rader")
print(f"Balansert: {balanced_df.count()} rader")
print("Klassefordeling:")
balanced_df.groupBy("transaction_type").count().show()
```
### Oversampling og undersampling-teknikker
```python
def oversample_minority_class(df, label_column, minority_class, target_ratio=0.5):
"""
Oversample minoritetsklassen ved a duplisere rader.
Args:
target_ratio: Onsket andel av minoritetsklassen
"""
class_counts = df.groupBy(label_column).count().collect()
counts = {row[label_column]: row["count"] for row in class_counts}
minority_count = counts[minority_class]
majority_count = sum(v for k, v in counts.items() if k != minority_class)
# Beregn hvor mange ganger minoritetsklassen ma dupliseres
desired_minority = int(majority_count * target_ratio / (1 - target_ratio))
oversample_factor = desired_minority / minority_count
# Oversample
minority_df = df.filter(F.col(label_column) == minority_class)
majority_df = df.filter(F.col(label_column) != minority_class)
oversampled = minority_df.sample(
withReplacement=True,
fraction=oversample_factor,
seed=42
)
return majority_df.unionByName(oversampled)
# Bruk
balanced = oversample_minority_class(
df_training,
label_column="readmission_status",
minority_class="readmitted",
target_ratio=0.3 # 30% reinnleggelser i treningsdatasettet
)
```
### SMOTE-lignende syntetisk oversampling
```python
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.clustering import KMeans
import numpy as np
def synthetic_oversampling(df, feature_columns, label_column, minority_class, n_synthetic):
"""
Generer syntetiske minoritetseksempler basert pa naeromrade-interpolering.
"""
minority_df = df.filter(F.col(label_column) == minority_class)
# Vektoriser features
assembler = VectorAssembler(inputCols=feature_columns, outputCol="features")
vectorized = assembler.transform(minority_df)
# For hver minoritetsrad: finn naermeste nabo og interpolder
# Forenklet implementering med KMeans for cluster-sentre
kmeans = KMeans(k=min(n_synthetic, minority_df.count()), seed=42)
model = kmeans.fit(vectorized)
# Bruk cluster-sentrene som syntetiske punkter
centers = model.clusterCenters()
synthetic_rows = []
for center in centers:
row = {col: float(center[i]) for i, col in enumerate(feature_columns)}
row[label_column] = minority_class
row["_synthetic"] = True
synthetic_rows.append(row)
synthetic_df = spark.createDataFrame(synthetic_rows)
return df.withColumn("_synthetic", F.lit(False)).unionByName(synthetic_df, allowMissingColumns=True)
```
---
## Active Learning and Uncertainty Sampling
### Prinsippet bak aktiv laering
Aktiv laering velger de mest informative eksemplene for merking, i stedet for a merke tilfeldig:
```
+-- Umerkede data (pool) --+
| |
| [Hogt sikker] -> Hopp over, allerede laert
| [Moderat sikker] -> Hopp over
| [Usikker] -> MERK DENNE! <-- Mest laererik
| [Veldig usikker] -> MERK DENNE! <-- Hoyest prioritet
| |
+---------------------------+
```
### Usikkerhetssamplings-strategier
| Strategi | Formel | Best for |
|----------|--------|----------|
| **Least Confidence** | 1 - max(P(y\|x)) | Generell klassifisering |
| **Margin Sampling** | P(y1\|x) - P(y2\|x) | Naere beslutningsgrenser |
| **Entropy Sampling** | -sum(P(y\|x) * log P(y\|x)) | Multi-class problemer |
| **Query-by-Committee** | Uenighet mellom modeller | Ensemble-basert |
### Implementering av aktiv laering
```python
import numpy as np
from sklearn.ensemble import RandomForestClassifier
class ActiveLearner:
"""
Aktiv laering med usikkerhetssampling for iterativ datamerking.
"""
def __init__(self, model=None, strategy="entropy"):
self.model = model or RandomForestClassifier(n_estimators=100)
self.strategy = strategy
self.labeled_indices = []
self.labels = []
def initial_sample(self, X, n_initial=100):
"""Velg et tilfeldig initialt treningssett."""
indices = np.random.choice(len(X), n_initial, replace=False)
self.labeled_indices = list(indices)
return indices
def query(self, X, n_samples=50):
"""Velg de mest informative eksemplene for merking."""
# Prediksjonssannsynligheter for umerkede data
unlabeled_mask = np.ones(len(X), dtype=bool)
unlabeled_mask[self.labeled_indices] = False
unlabeled_indices = np.where(unlabeled_mask)[0]
if len(unlabeled_indices) == 0:
return np.array([])
X_unlabeled = X[unlabeled_indices]
probas = self.model.predict_proba(X_unlabeled)
# Beregn usikkerhetsscorer
if self.strategy == "entropy":
scores = -np.sum(probas * np.log(probas + 1e-10), axis=1)
elif self.strategy == "least_confidence":
scores = 1 - np.max(probas, axis=1)
elif self.strategy == "margin":
sorted_probas = np.sort(probas, axis=1)
scores = 1 - (sorted_probas[:, -1] - sorted_probas[:, -2])
# Velg top-n mest usikre
top_indices = np.argsort(scores)[-n_samples:]
return unlabeled_indices[top_indices]
def teach(self, X, y, indices, labels):
"""Oppdater modellen med nylig merkede data."""
self.labeled_indices.extend(indices)
self.labels.extend(labels)
X_labeled = X[self.labeled_indices]
y_labeled = np.array(self.labels)
self.model.fit(X_labeled, y_labeled)
# Brukseksempel
learner = ActiveLearner(strategy="entropy")
# Runde 1: Tilfeldig initialt sett
initial_idx = learner.initial_sample(X_pool, n_initial=100)
initial_labels = get_labels_from_labelers(X_pool[initial_idx])
learner.teach(X_pool, None, initial_idx, initial_labels)
# Runde 2-N: Aktiv laering
for round_num in range(10):
query_idx = learner.query(X_pool, n_samples=50)
new_labels = get_labels_from_labelers(X_pool[query_idx])
learner.teach(X_pool, None, query_idx, new_labels)
print(f"Runde {round_num + 1}: Totalt merket = {len(learner.labeled_indices)}")
```
---
## Crowdsourcing and Labeling Platforms
### Azure Machine Learning Data Labeling
Azure ML tilbyr en komplett merkeplattform med stotte for:
| Funksjon | Beskrivelse |
|----------|-------------|
| **Bildeklassifisering** | Multi-class og multi-label |
| **Objektdeteksjon** | Bounding boxes |
| **Instanssegmentering** | Polygoner |
| **Semantisk segmentering** | Piksel-niva (preview) |
| **Tekstklassifisering** | Single og multi-label |
| **Named Entity Recognition** | Tekst-span-merking |
### Opprette et merkeprosjekt
```python
# Azure ML SDK v2 - Opprett bildeklassifiseringsprosjekt
from azure.ai.ml import MLClient
from azure.ai.ml.entities import DataLabelingJob
# Opprett klient
ml_client = MLClient(credential, subscription_id, resource_group, workspace_name)
# Definer merkeprosjekt
labeling_job = DataLabelingJob(
display_name="waste-type-classification",
description="Klassifiser avfallstyper fra bilder ved gjenvinningsstasjon",
labeling_job_type="ImageClassificationMulticlass",
data={"uri": "azureml://datastores/images/paths/waste_images/"},
labels={
"classes": [
{"name": "paper", "display_name": "Papir"},
{"name": "plastic", "display_name": "Plast"},
{"name": "glass", "display_name": "Glass"},
{"name": "metal", "display_name": "Metall"},
{"name": "food", "display_name": "Matavfall"},
{"name": "hazardous", "display_name": "Farlig avfall"},
{"name": "other", "display_name": "Annet"}
]
},
properties={
"ml_assist_enabled": True, # Aktiver ML-assistert merking
"consensus_labeling_enabled": True, # Krev konsensus
"min_label_count": 2 # Minimum 2 merkere per bilde
}
)
# Opprett prosjekt
created_job = ml_client.labeling_jobs.begin_create_or_update(labeling_job)
```
### ML-Assisted Labeling
ML-assistert merking i Azure ML akselererer prosessen gjennom to faser:
```
Fase 1: CLUSTERING
+-- Merkere merker ~300 bilder manuelt
+-- ML-modell grupperer lignende bilder
+-- Merkere ser klynger av like bilder (raskere merking)
Fase 2: PRE-LABELING
+-- Modell trenes pa merkede data
+-- Modell foreslaar etiketter for umerkede bilder
+-- Merkere bekrefter/korrigerer forhondsetiketter
+-- Prosessen gjentas iterativt
```
**Viktige hensyn:**
- Transfer learning akselererer opplaering: Noen ganger trengs kun 300 merkede eksempler
- Konsensus-etiketter brukes for trening naar aktivert
- Bilder shuffles tilfeldig for a redusere bias
- Tekstinnholdet begrenses til ~128 ord for treningseffektivitet
---
## Quality Control and Inter-Rater Agreement
### Konsensus-merking
```python
# Implementer konsensusbasert kvalitetskontroll
from collections import Counter
def calculate_inter_rater_agreement(labels_per_item: dict) -> dict:
"""
Beregn inter-rater agreement (IRA) for merkede data.
Args:
labels_per_item: {item_id: [label_from_rater1, label_from_rater2, ...]}
Returns:
Statistikk over enighet
"""
agreements = []
disagreements = []
for item_id, labels in labels_per_item.items():
counter = Counter(labels)
most_common_label, most_common_count = counter.most_common(1)[0]
total_raters = len(labels)
agreement_ratio = most_common_count / total_raters
if agreement_ratio >= 0.8:
agreements.append({
"item_id": item_id,
"consensus_label": most_common_label,
"agreement": agreement_ratio
})
else:
disagreements.append({
"item_id": item_id,
"labels": dict(counter),
"agreement": agreement_ratio
})
total = len(labels_per_item)
return {
"total_items": total,
"agreed": len(agreements),
"disagreed": len(disagreements),
"agreement_rate": round(len(agreements) / max(total, 1) * 100, 1),
"disagreed_items": disagreements[:10] # Vis topp 10 uenigheter
}
```
### Cohens Kappa for kvalitetsmalinger
```python
from sklearn.metrics import cohen_kappa_score
def evaluate_labeler_quality(rater1_labels, rater2_labels):
"""
Beregn Cohens Kappa mellom to merkere.
Tolkning:
- < 0.20: Darlig enighet
- 0.21-0.40: Moderat enighet
- 0.41-0.60: Moderat enighet
- 0.61-0.80: Substansiell enighet
- 0.81-1.00: Naer perfekt enighet
"""
kappa = cohen_kappa_score(rater1_labels, rater2_labels)
if kappa < 0.40:
quality = "LAV - Gjennomga retningslinjer og gi oppfolging"
elif kappa < 0.60:
quality = "MODERAT - Akseptabel for screening, ikke for endelig trening"
elif kappa < 0.80:
quality = "GOD - Akseptabel for de fleste ML-oppgaver"
else:
quality = "UTMERKET - Hoy kvalitet for treningsdata"
return {"kappa": round(kappa, 3), "quality": quality}
# Eksempel
result = evaluate_labeler_quality(
rater1_labels=["positive", "negative", "positive", "neutral", "positive"],
rater2_labels=["positive", "negative", "neutral", "neutral", "positive"]
)
```
### Kvalitetskontrollpipeline
```
1. Initial merking (2-3 merkere per element)
|
2. Beregn inter-rater agreement (IRA)
|
3. IRA >= 80%? --> Bruk konsensus-etikett
|
4. IRA < 80%? --> Send til ekspert-merker (adjudicator)
|
5. Ekspert avgjer endelig etikett
|
6. Oppdater merkeretningslinjer basert pa uenigheter
|
7. Re-tren ML-assist-modell med nye etiketter
```
---
## Feedback Loops for Continuous Labeling
### Produksjonsdata tilbake til merking
```python
def identify_candidates_for_relabeling(model, new_data_df, confidence_threshold=0.6):
"""
Identifiser prediksjoner med lav konfidens for manuell gjennomgang.
"""
predictions = model.predict_proba(new_data_df)
low_confidence = new_data_df.filter(
F.col("prediction_confidence") < confidence_threshold
)
# Prioriter etter usikkerhet
candidates = low_confidence.orderBy(F.col("prediction_confidence").asc())
# Legg til i merke-ko
candidates.select(
"record_id", "features", "prediction", "prediction_confidence"
).write.format("delta").mode("append") \
.saveAsTable("lakehouse.default.labeling_queue")
return candidates.count()
# Kjor daglig
n_candidates = identify_candidates_for_relabeling(
model=production_model,
new_data_df=todays_predictions,
confidence_threshold=0.6
)
print(f"{n_candidates} nye elementer lagt til merke-koen")
```
### Drift-deteksjon og re-merking
```python
def detect_label_drift(historical_labels_df, recent_labels_df, columns):
"""
Oppdager endringer i etikettdistribusjon over tid.
"""
from scipy.stats import chi2_contingency
for col in columns:
hist_dist = historical_labels_df.groupBy(col).count().toPandas()
recent_dist = recent_labels_df.groupBy(col).count().toPandas()
# Chi-kvadrat-test
contingency = hist_dist.merge(recent_dist, on=col, suffixes=["_hist", "_recent"])
chi2, p_value, dof, expected = chi2_contingency(
contingency[["count_hist", "count_recent"]].values.T
)
if p_value < 0.05:
print(f"DRIFT OPPDAGET i '{col}': p={p_value:.4f}")
print(f" Historisk: {dict(zip(hist_dist[col], hist_dist['count']))}")
print(f" Nylig: {dict(zip(recent_dist[col], recent_dist['count']))}")
```
---
## Referanser
- [Set up an image labeling project](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-create-image-labeling-projects) -- Bildedatamerking i Azure ML
- [Set up a text labeling project](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-create-text-labeling-projects) -- Tekstdatamerking i Azure ML
- [Labeling images and text documents](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-label-data) -- Merkerverktoy og ML-assistert merking
- [Prepare data for computer vision tasks](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-prepare-datasets-for-automl-images) -- Data for AutoML-bildemodeller
- [Label text data for training](https://learn.microsoft.com/en-us/azure/ai-services/language-service/custom-text-classification/how-to/tag-data) -- Merking for Custom Language Models
- [Create and explore datasets with labels](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-labeled-dataset) -- Bruk av merkede datasett i Azure ML
---
## For arkitekten
- **Bruk denne referansen** naar kunder planlegger datamerkings-prosjekter for ML-modeller, eller naar de trenger strategier for a hondtere ubalanserte datasett.
- **Azure ML Data Labeling er forstevalget** for merkingsprosjekter i Microsoft-stakken. ML-assistert merking kan redusere manuelt arbeid med 50-80% etter initiell opplaering.
- **Aktiv laering bor alltid vurderes** for store umerkede datasett -- det reduserer merkekostnader dramatisk ved a prioritere de mest informative eksemplene.
- **Kvalitetskontroll er ikke valgfritt**: Krev konsensus mellom merkere (minimum 2), mal inter-rater agreement, og ha en ekspert-adjudicator for uenigheter.
- **For norsk offentlig sektor**: Vurder personvernaspekter ved merking av data som kan inneholde personopplysninger. Bruk PII-deteksjon for a fjerne sensitiv informasjon for merkerne ser dataene.
Reference in a new issue View git blame Copy permalink