open/ms-ai-architect
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity Actions

feat(ultraplan-local): v1.6.0 — /ultraresearch-local deep research command

Browse source 
Add /ultraresearch-local for structured research combining local codebase
analysis with external knowledge via parallel agent swarms. Produces research
briefs with triangulation, confidence ratings, and source quality assessment.

New command: /ultraresearch-local with modes --quick, --local, --external, --fg.
New agents: research-orchestrator (opus), docs-researcher, community-researcher,
security-researcher, contrarian-researcher, gemini-bridge (all sonnet).
New template: research-brief-template.md.

Integration: --research flag in /ultraplan-local accepts pre-built research
briefs (up to 3), enriches the interview and exploration phases. Planning
orchestrator cross-references brief findings during synthesis.

Design principle: Context Engineering — right information to right agent at
right time. Research briefs are structured artifacts in the pipeline:
ultraresearch → brief → ultraplan --research → plan → ultraexecute.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-04-08 08:58:35 +02:00
commit baa2d0220b
488 changed files with 213221 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

551
skills/ms-ai-advisor/references/prompt-engineering/multimodal-prompt-design.md Normal file
View file

@ -0,0 +1,551 @@
# Multimodal Prompt Design with Images and Text
**Last updated:** 2026-02
**Status:** GA
**Category:** Prompt Engineering & LLM Optimization
---
## Introduksjon
Multimodal prompt design handler om å utforme effektive instruksjoner som kombinerer tekst og bilder for å maksimere responskvaliteten fra Large Multimodal Models (LMM). Vision-enabled modeller som GPT-4o, GPT-4o mini, GPT-4 Turbo with Vision, GPT-5-serien og o-serien kan analysere bilder og generere tekstlige responser basert på både visuelt og tekstlig innhold.
**Nøkkelkonsepter:**
- Vision-enabled modeller kombinerer Natural Language Processing (NLP) med visuell forståelse
- Støtter både URL-baserte bilder (HTTP/HTTPS) og Base64-enkodede bilder
- Bildeinput teller som tokens og påvirker kostnad og latency
- Kan håndtere opptil 10 bilder per chat request
- Detail-parameter (`low`, `high`, `auto`) styrer tokenforbruk og responskvalitet
**Tekniske tokens:**
| Modell | Low detail | High detail (1024×1024) |
|--------|-----------|------------------------|
| GPT-4o / GPT-4 Turbo | 85 tokens | 4160 tokens |
| GPT-4o mini | 2833 tokens | Varierer med dimensjon |
## Kjernekomponenter
### 1. Input-formater
**URL-basert bildeinnput:**
```json
{
"type": "image_url",
"image_url": {
"url": "https://example.com/image.jpg",
"detail": "high"
}
}
```
**Base64-enkodet bildeinnput:**
```json
{
"type": "image_url",
"image_url": {
"url": "data:image/jpeg;base64,<base64_string>"
}
}
```
**Python-eksempel for lokal fil:**
```python
import base64
from mimetypes import guess_type
def local_image_to_data_url(image_path):
mime_type, _ = guess_type(image_path)
if mime_type is None:
mime_type = 'application/octet-stream'
with open(image_path, "rb") as image_file:
base64_encoded_data = base64.b64encode(image_file.read()).decode('utf-8')
return f"data:{mime_type};base64,{base64_encoded_data}"
```
### 2. Detail Parameter Settings
| Setting | Oppførsel | Use case | Token-påvirkning |
|---------|----------|----------|------------------|
| `auto` | Modellen velger selv basert på bildestørrelse | Default, balansert | Varierer |
| `low` | 512×512 lavoppløselig analyse | Rask responsgivning, grov kategorisering | Lavt (85 tokens GPT-4o) |
| `high` | Segmentert analyse i 512×512-blokker | Detaljanalyse, OCR, objektdeteksjon | Høyt (4160+ tokens) |
### 3. Message Content Array Structure
Multimodale prompts bruker content-array i stedet for enkel string:
```python
messages=[
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe this picture:"
},
{
"type": "image_url",
"image_url": {
"url": "<image_url>",
"detail": "high"
}
}
]
}
],
max_tokens=2000
```
**Viktig:** Alltid sett `max_tokens` eller output blir trunkert.
## Arkitekturmønstre
### Pattern 1: Single Image Analysis
**Bruksområde:** Bildeanalyse, beskrivelse, kategorisering
**Best practice:** Plasser bildet FØR teksten i prompten
```python
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": image_url}},
{"type": "text", "text": "What objects are visible in this image?"}
]
}
],
max_tokens=500
)
```
### Pattern 2: Multi-Image Comparison
**Bruksområde:** Before/after, A/B testing, damage assessment
**Begrensning:** Maks 10 bilder per request
```python
content = [
{"type": "text", "text": "Compare these two images and identify differences:"},
{"type": "image_url", "image_url": {"url": image1_url, "detail": "high"}},
{"type": "image_url", "image_url": {"url": image2_url, "detail": "high"}}
]
```
### Pattern 3: Few-shot Learning with Images
**Bruksområde:** Konsistent formatering, klassifisering med eksempler
```python
messages = [
{"role": "system", "content": "You classify dog breeds with weight and height."},
{"role": "user", "content": [
{"type": "text", "text": "Q: What breed is this?"},
{"type": "image_url", "image_url": {"url": pomeranian_url}}
]},
{"role": "assistant", "content": "Breed: Pomeranian; weight: 3-7 lbs; height: 8-14 inches"},
{"role": "user", "content": [
{"type": "text", "text": "Q: What breed is this?"},
{"type": "image_url", "image_url": {"url": new_dog_url}}
]}
]
```
### Pattern 4: Step-by-step Visual Analysis
**Bruksområde:** Komplekse scenarioer, recipe extraction, damage assessment
```python
# Steg 1: Beskrivelse
"First, describe everything you see in this image in detail."
# Steg 2: Ekstraksjon
"Based on your description, extract the recipe ingredients and instructions."
# Steg 3: Strukturering
"Format the output as a JSON object with 'ingredients' and 'steps' arrays."
```
### Pattern 5: Multimodal RAG (Retrieval-Augmented Generation)
**Bruksområde:** Enterprise search over dokument med bilder/diagrammer
**To tilnærminger:**
1. **Image verbalization:** LLM beskriver bilder → embeddes som tekst → hybrid search
2. **Direct multimodal embeddings:** Bilder og tekst embeddes direkte i samme vektorrom
| Tilnærming | Fordel | Ulempe | Use case |
|-----------|--------|--------|----------|
| Verbalization | Semantisk dybde, LLM-sitérbare beskrivelser | LLM-kall per bilde, høyere latency | Diagrammer, flowcharts, infografikk |
| Direct embeddings | Rask, ingen LLM-kall ved indexing | Ingen forklaring av relasjoner | Visual similarity, produktsøk |
**Azure AI Search multimodal pipeline:**
1. Document extraction (Document Extraction / Layout / Content Understanding skill)
2. Text chunking (Text Split skill)
3. Image verbalization (GenAI Prompt skill + LLM)
4. Embedding (Azure OpenAI / Foundry / Azure Vision)
5. Knowledge store (for image storage og retrieval)
## Beslutningsveiledning
### Når bruke multimodal prompting?
| Scenario | Anbefalt tilnærming | Detail setting |
|----------|-------------------|----------------|
| Produktkatalog beskrivelser | Single image + kontekstuell system prompt | `auto` eller `high` |
| Skadevurdering (forsikring) | Multi-image + task-oriented prompt | `high` |
| OCR + strukturert ekstraksjon | High detail + step-by-step prompting | `high` |
| Social media content moderation | Low detail for rask screening | `low` |
| Medisinske bilder | **IKKE bruk** (out of scope for modellen) | N/A |
### Prompt Engineering Prinsipper
| Prinsipp | Beskrivelse | Eksempel |
|----------|-------------|----------|
| **Contextual specificity** | Legg til kontekst om bruksområde | "Describe for an outdoor product catalog, enthusiastic tone" |
| **Task-oriented** | Definer spesifikk oppgave | "Analyze car damage for insurance report, detail all visible damage" |
| **Handle refusals** | Be om forklaring, bryt ned request | "What information do you need to plan this meal?" |
| **Add examples** | Few-shot learning med bilde+tekst par | Se Pattern 3 over |
| **Break down requests** | Del komplekse oppgaver i steg | Se Pattern 4 over |
| **Define output format** | Spesifiser JSON, Markdown, HTML, osv. | "Return as JSON with 'ingredients' and 'steps' arrays" |
### Håndtering av refusals
```python
# Initial prompt
"Plan this meal" # → "Sorry, I can't provide that information."
# Follow-up strategy
"What information do you need?"
# → Modellen lister opp: antall personer, allergier, anledning, osv.
# Refined prompt
"Plan a dinner for 4 people, vegetarian, casual setting. Image shows [...]"
# → Modellen gir detaljert plan
```
## Integrasjon med Microsoft-stakken
### Azure OpenAI Service
**Endpoint:** `https://{RESOURCE_NAME}.openai.azure.com/openai/v1/chat/completions`
**Autentisering:**
- API key: `api-key` header
- Managed Identity: `DefaultAzureCredential` + bearer token provider
**Python SDK:**
```python
from openai import OpenAI
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
token_provider = get_bearer_token_provider(
DefaultAzureCredential(),
"https://cognitiveservices.azure.com/.default"
)
client = OpenAI(
base_url="https://YOUR-RESOURCE.openai.azure.com/openai/v1/",
api_key=token_provider
)
```
### Azure AI Foundry (tidligere Azure AI Studio)
**Supported models for multimodal:**
- GPT-5 series (gpt-5, gpt-5-mini, gpt-5-nano)
- GPT-4.1 series
- GPT-4.5
- GPT-4o series (gpt-4o, gpt-4o-mini)
- o-series reasoning models (o1, o3, o4-mini)
**Model deployment types:**
- Standard deployment (region-bound)
- Global-standard deployment (dynamic routing, høyere quota)
### Prompt Flow Integration
**Azure OpenAI GPT-4 Turbo with Vision tool:**
```yaml
# Prompt template
# system:
As an AI assistant, your task involves interpreting images and responding to questions.
Remember to provide accurate answers based on the information present in the image.
# user:
Can you tell me what the image depicts?
![image]({{image_input}})
```
**Tool configuration:**
1. Select Azure OpenAI connection
2. Specify deployment (GPT-4o, GPT-4o-mini, etc.)
3. Set `image_input` parameter (URL eller upload)
4. Validate and parse input
5. Run flow
### Azure AI Search Multimodal Integration
**Import data wizard → Multimodal RAG:**
**Forutsetninger:**
| Provider | Image verbalization | Multimodal embeddings |
|----------|-------------------|----------------------|
| Azure Foundry | phi-4, gpt-4o, gpt-5 (LLM) + text-embedding-3-* | N/A |
| Azure OpenAI | gpt-4o, gpt-5 (LLM) + text-embedding-3-* | N/A |
| Azure Vision | N/A | Multimodal embeddings (built-in) |
**Pipeline-steg (wizard):**
1. Data source: Azure Blob / ADLS Gen2
2. Content extraction: Document Extraction / Layout / Content Understanding skill
3. Text chunking: Text Split skill
4. Image verbalization (optional): GenAI Prompt skill
5. Embedding: Azure OpenAI / Foundry / Azure Vision
6. Knowledge store: Lagrer bilder for retrieval
**Query-tid:**
- Hybrid queries (text + vector) for verbalized content
- Image-to-vector queries KUN med Azure Vision multimodal embeddings vectorizer
### Power Platform Integration
**AI Builder + GPT-4o via Azure OpenAI connector:**
- Custom connector til Azure OpenAI endpoint
- Parse Base64-enkoded input fra Power Apps
- Return response til Power Automate flow
## Offentlig sektor (Norge)
### Compliance og databehandling
| Aspekt | Vurdering |
|--------|-----------|
| **GDPR** | Bilder kan inneholde personopplysninger → databehandleravtale påkrevd |
| **Schrems II** | Azure OpenAI EU-regioner (West Europe, North Europe) anbefales |
| **Sikkerhetsloven** | Klassifisert informasjon: IKKE send til sky-LLM |
| **Offentleglova** | Vurder om bildeinnhold er offentlig eller unntatt |
### Use cases offentlig sektor
| Sektor | Use case | Multimodal pattern |
|--------|----------|-------------------|
| **Vegvesen** | Skaderegistrering vei/bruer fra drone-bilder | Multi-image damage assessment |
| **NAV** | Automatisk dokumentklassifisering (skjema med vedlegg) | OCR + structured extraction |
| **Helsedirektoratet** | Visuell analyse av offentlige helsedata (grafer) | ⚠️ IKKE medisinske bilder |
| **Kulturminnevern** | Katalogisering av bygninger/artefakter | Product catalog pattern |
| **Krisehåndtering** | Situasjonsanalyse fra feltbilder | Step-by-step visual analysis |
**Viktig:** Multimodal embeddings er IKKE designet for medisinsk diagnostikk.
### Kostnadskontroll
**Strategier:**
- Bruk `low` detail for initielt screening, `high` kun for prioriterte bilder
- Pre-filter bilder med Azure AI Vision (klassisk) før LLM-analyse
- Batch-prosessering med Azure Batch + OpenAI
- Monitor token usage via Azure Monitor + Cost Management
## Kostnad og lisensiering
### Token-kostnader (per bilde)
**GPT-4o (2024-11-20 deployment):**
| Detail | Dimensjon | Input tokens | Estimert kostnad (NOK)* |
|--------|-----------|--------------|------------------------|
| `low` | Any | 85 | ~0.11 kr |
| `high` | 1024×1024 | 4160 | ~5.41 kr |
| `high` | 1024×1536 (portrait) | 6240 | ~8.11 kr |
| `high` | 1536×1024 (landscape) | 6208 | ~8.07 kr |
**GPT-4o mini (2024-07-18 deployment):**
| Detail | Dimensjon | Input tokens | Estimat kostnad (NOK)* |
|--------|-----------|--------------|------------------------|
| `low` | Any | 2833 | ~0.47 kr |
| `high` | 1024×1024 | Lavere enn GPT-4o | ~1-2 kr |
*Basert på ca. $0.0025 per 1K input tokens GPT-4o, $0.00015 per 1K GPT-4o mini (jan 2026), vekslingskurs ~10.5 NOK/USD. Verifiser aktuelle priser.
### Lisensiering
**Azure OpenAI:**
- Krever Azure-abonnement
- Pay-as-you-go (consumption-based)
- Ingen lisenskostnad utover API-kall
**M365 Copilot:**
- Multimodal capabilities i Copilot for M365 (chat with images)
- Krever M365 E3/E5 + Copilot lisens (~$30/bruker/måned)
- Begrenset til M365-kontekst (SharePoint, OneDrive, Teams)
**Power Platform:**
- AI Builder credits for custom connectors til Azure OpenAI
- Premium connector: $40/bruker/måned eller $200/kapasitet/måned
- Per-request costing via Azure OpenAI on top
### TCO-optimalisering
| Strategi | Besparelse | Trade-off |
|----------|-----------|-----------|
| Bruk GPT-4o mini i stedet for GPT-4o | ~94% | Noe lavere kvalitet |
| `low` detail i stedet for `high` | ~98% (GPT-4o) | Mister findetaljer |
| Pre-filter med Azure AI Vision | 50-80% | Ekstra kompleksitet |
| Batch-prosessering (asynkront) | 50% rabatt (Azure OpenAI batch API) | Latency 24t |
| Cache responses (semantic cache) | Varierer | Treff-rate avhengig |
## For arkitekten (Cosmo)
### Discovery-spørsmål
Når kunde ønsker multimodal løsning, kartlegg:
1. **Bildetyper:**
- Hva slags bilder? (foto, skjermbilder, diagrammer, dokumenter)
- Typisk oppløsning og størrelse?
- Volum (bilder/dag, bilder/måned)?
2. **Use case:**
- Hva skal skje med bildene? (kategorisering, OCR, beskrivelse, damage assessment)
- Responstidskrav? (sanntid vs. batch)
- Ønsket output-format? (JSON, tekst, strukturert data)
3. **Integrasjon:**
- Hvor kommer bildene fra? (bruker-upload, blob storage, SharePoint)
- Hvor skal responser? (app, database, Power BI)
- Eksisterende systemer?
4. **Compliance:**
- Inneholder bildene personopplysninger?
- Klassifiseringsnivå (offentlig, begrenset, konfidensiell)?
- GDPR-krav?
### Decision Tree
```
Multimodal scenario?
├─ Volum < 100 bilder/dag
│ └─ Azure OpenAI direct API (GPT-4o mini, low detail)
│
├─ Volum 100-10k bilder/dag
│ ├─ Sanntid påkrevd?
│ │ ├─ Ja → Azure OpenAI + caching + auto-scaling
│ │ └─ Nei → Azure OpenAI Batch API (50% rabatt)
│ └─ OCR primært? → Azure AI Document Intelligence i stedet
│
├─ Volum > 10k bilder/dag
│ └─ Azure AI Search multimodal pipeline + Azure Vision embeddings
│
└─ Trengs søk over historiske bilder?
└─ Azure AI Search multimodal RAG (verbalization eller direct embeddings)
```
### Red Flags
⚠️ **Unngå multimodal LLM når:**
- Medisinsk diagnostikk (out of scope)
- Høy sikkerhetsgradert materiale (risiko for datalekkasje)
- Sanntids-video (bruk Azure Video Indexer i stedet)
- Kun OCR behov (Azure AI Document Intelligence er billigere)
- Ekstrem høy volum real-time (cost explosion)
### Proof-of-Concept anbefaling
**2-ukers POC:**
1. **Uke 1:** Bygg baseline med Azure OpenAI Playground
- Test 20-50 representative bilder
- Evaluer `low` vs `high` detail
- Test 3-5 prompt-variasjoner
- Mål accuracy og token usage
2. **Uke 2:** Implementer mini-pipeline
- Python/C# script med OpenAI SDK
- Integrer med blob storage
- Logger tokens og cost
- Demo til stakeholders
**Success criteria:**
- Accuracy > 85% på use case
- Token cost innenfor budsjett
- Latency < 5 sekunder (95th percentile)
### Arkitekturmaler
**Template 1: Simple image analysis API**
```
User → Azure Function (HTTP trigger)
→ OpenAI SDK (GPT-4o mini)
→ Parse response
→ Return JSON
```
**Template 2: Multimodal RAG**
```
Documents (PDF) → Azure AI Search Multimodal wizard
→ GenAI Prompt skill (verbalization)
→ Azure OpenAI embedding
→ Vector index
User query → Hybrid search (text + vector)
→ GPT-4o with grounding
→ Response + image citations
```
**Template 3: Batch processing**
```
Blob upload → Event Grid trigger
→ Azure Function (queue message)
→ OpenAI Batch API submit
→ Poll for completion (24h)
→ Write results to Cosmos DB
```
### Monitoring og observability
**Nøkkel-metrikker:**
- Tokens per request (avg, p50, p95, p99)
- Cost per image analyzed (NOK)
- Latency (end-to-end)
- Error rate (content filter, API errors)
- Accuracy (human-in-the-loop validation)
**Azure Monitor dashboard:**
```kusto
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where OperationName == "ChatCompletions_Create"
| extend tokens_used = toint(properties_s.usage.total_tokens)
| extend has_image = properties_s contains "image_url"
| summarize avg(tokens_used), percentile(tokens_used, 95) by bin(TimeGenerated, 1h), has_image
```
## Kilder og verifisering
**Microsoft Learn dokumentasjon (verifisert 2026-02):**
- [Use vision-enabled chat models](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/gpt-with-vision) — Offisiell how-to guide for GPT-4o/GPT-4 Turbo with Vision
- [Image prompt engineering techniques](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/gpt-4-v-prompt-engineering) — Best practices for multimodal prompting
- [Multimodal search in Azure AI Search](https://learn.microsoft.com/en-us/azure/search/multimodal-search-overview) — RAG-arkitektur med image verbalization og direct embeddings
- [Azure OpenAI models](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/models) — Modelloversikt og token-kostnader
- [Quickstart: Multimodal search in Azure portal](https://learn.microsoft.com/en-us/azure/search/search-get-started-portal-image-search) — Wizard-basert oppsett
- [Get started with multimodal vision chat apps](https://learn.microsoft.com/en-us/azure/developer/ai/get-started-app-chat-vision) — End-to-end sample app med Base64 encoding
**Code samples:**
- Azure-Samples/cognitive-services-sample-data-files (GitHub)
- Azure AI Foundry multimodal RAG sample app (https://aka.ms/azs-multimodal-sample-app-repo)
**Confidence markers:**
- ✅ **High confidence:** Token counts, API structure, detail parameter behavior (direkte fra offisiell docs)
- ✅ **High confidence:** Prompt engineering patterns (bekreftet i Microsoft Learn)
- ⚠️ **Medium confidence:** Kostberegninger i NOK (basert på jan 2026 pricing, kan variere)
- ⚠️ **Medium confidence:** Offentlig sektor use cases (inferert fra generelle patterns, ikke Microsoft-spesifikt)
**Sist verifisert:** 2026-02-04
**Neste review:** 2026-04 (eller ved nye GPT-modeller)