# Multimodal RAG with Gemini Embedding 2 and Claude Code

Search across PDFs, images, and documents using plain English.
No coding required. Claude Code builds everything from prompts.

![Search for "What is the largest planet?" returns both the Jupiter photograph and the PDF fact sheet](docs/demo-screenshot.png)

> **Gemini Embedding 2** converts text, images, and video into the same
> searchable space. **Claude Code** builds the app. **Pinecone** stores the
> vectors. You just copy four prompts.

## Table of Contents

- [Quick Start](#quick-start)
- [What This Does](#what-this-does)
- [Prerequisites](#prerequisites)
- [Step-by-Step Guide](#step-by-step-guide)
- [Example Data](#example-data)
- [Why Image Descriptions Matter](#why-image-descriptions-matter)
- [Costs](#costs)
- [Troubleshooting](#troubleshooting)
- [How It Works](#how-it-works)
- [License](#license)

## Quick Start

```bash
git clone https://git.thedharmalab.com/ktg/multimodal-rag-guide.git
cd multimodal-rag-guide
claude
```

Then paste the prompt from [`prompts/01-setup.md`](prompts/01-setup.md) into Claude Code.

Four prompts, 30 minutes, working multimodal search.

## What This Does

One search box that understands PDFs, images, and text at the same time.

Ask "What is the largest planet in our solar system?" and the system
returns the Jupiter fact sheet from a PDF, the Voyager photograph of
the Great Red Spot from a JPG, and a confidence score for each result.
One question, multiple formats, ranked by meaning.

This is called Retrieval-Augmented Generation (RAG). Google's
Gemini Embedding 2 handles the multimodal part: it converts different
content types into the same numerical format so they become searchable
together. Claude Code handles the building part: it reads your prompts
and writes all the code. You handle neither.

## Prerequisites

| Requirement | Cost | What it does |
|---|---|---|
| [Claude Code](https://claude.ai) | Part of Claude Pro ($20/mo) or Max | Builds the app and answers questions |
| [Google AI Studio](https://aistudio.google.com/) | Free tier | Gemini Embedding 2 API key |
| [Pinecone](https://www.pinecone.io/) | Free tier | Vector database for storing embeddings |

No programming knowledge required.

## Step-by-Step Guide

### Step 0: Get your API keys (10 minutes)

**Google AI Studio** (for Gemini Embedding 2):

1. Go to [aistudio.google.com](https://aistudio.google.com/)
2. Sign in with a Google account
3. Click "Get API key" in the left sidebar
4. Click "Create API key" and copy it

**Pinecone** (for the vector database):

1. Go to [pinecone.io](https://www.pinecone.io/) and create a free account
2. In the dashboard, click "Create Index"
3. Name it `space-search`, set dimensions to `3072`, choose `cosine` metric
4. Select the free "Starter" plan
5. Copy your API key from "API Keys"

### Step 1: Clone and start Claude Code (5 minutes)

```bash
git clone https://git.thedharmalab.com/ktg/multimodal-rag-guide.git
cd multimodal-rag-guide
claude
```

Paste the prompt from [`prompts/01-setup.md`](prompts/01-setup.md).
Claude Code creates the project structure and installs dependencies.

When done, copy `env.template` to `.env` and fill in your API keys.

### Step 2: Ingest your files (10 minutes)

Paste the prompt from [`prompts/02-ingest.md`](prompts/02-ingest.md).

Claude Code reads each file, splits it into chunks, generates
embeddings via Gemini Embedding 2, and stores everything in Pinecone.

### Step 3: Search (5 minutes)

Paste the prompt from [`prompts/03-search.md`](prompts/03-search.md).

Claude Code builds a web interface. Open `http://localhost:3333`
in your browser and try these searches:

| Query | Expected results |
|---|---|
| "What is the largest planet?" | Jupiter fact sheet + Jupiter image |
| "First Moon landing" | Aldrin image + solar system overview |
| "Which moon has volcanoes?" | Moons PDF mentioning Io |
| "How far is Jupiter from Earth?" | Jupiter fact sheet with exact distance |

A single question pulls results from both PDFs and images.

### Step 4: Make it your own

Replace the NASA example files with your own content:

1. Add PDFs, images, or documents to `example-data/`
2. Write descriptions for images (see [`example-data/descriptions.md`](example-data/descriptions.md))
3. Paste [`prompts/04-improve.md`](prompts/04-improve.md) to re-index

Ideas: company documents, research papers, travel photos,
recipe collections, course notes.

## Example Data

The `example-data/` folder contains NASA public domain files
(no copyright restrictions):

| File | Description |
|---|---|
| `solar-system-overview.pdf` | Overview of our solar system |
| `jupiter-fact-sheet.pdf` | Detailed data about Jupiter |
| `solar-system-moons.pdf` | Guide to planetary moons |
| `earthrise.jpg` | Earth from lunar orbit, Apollo 8 (1968) |
| `aldrin-moon.jpg` | Buzz Aldrin on the Moon, Apollo 11 (1969) |
| `jupiter-great-red-spot.jpg` | Jupiter by Voyager 1 (1979) |
| `iss-over-earth.jpg` | The Moon seen from the ISS |
| `descriptions.md` | Image descriptions for search quality |

## Why Image Descriptions Matter

The search system finds images through their text descriptions,
not by "seeing" them. A description like "Photo of a planet" only
matches searches containing those exact concepts. A description
like "Full-disk portrait of Jupiter captured by Voyager 1 in 1979,
showing horizontal cloud bands and the Great Red Spot" matches
searches about Jupiter, Voyager missions, storms, and cloud patterns.

See [`example-data/descriptions.md`](example-data/descriptions.md)
for side-by-side examples.

## Costs

$0 extra if you already have a Claude subscription. Both Gemini
Embedding 2 and Pinecone have free tiers that cover this guide
and well beyond.

See [costs.md](costs.md) for the full breakdown.

## Troubleshooting

See [troubleshooting.md](troubleshooting.md) for the 10 most common
problems. The most effective fix for almost anything: copy the exact
error message and paste it into Claude Code.

## How It Works

```
Your files --> Chunking --> Gemini Embedding 2 --> Pinecone (vector DB)
                                                        |
Your question --> Gemini Embedding 2 --> Search --> Claude answers
```

Gemini Embedding 2 converts all content types (text, images, video,
audio) into numerical vectors in one shared space. Pinecone stores
and searches those vectors. Claude reads the matching content and
generates answers.

For plain-English explanations of embeddings, vector databases, RAG,
and chunking, see [concepts.md](concepts.md).

## Built With

- [Claude Code](https://claude.ai) by Anthropic
- [Gemini Embedding 2](https://ai.google.dev/) by Google
- [Pinecone](https://www.pinecone.io/)

## License

[MIT](LICENSE)

---

Part of [The Dharma Lab](https://thedharmalab.com). Read the
[full article](https://thedharmalab.com/) for the story behind
this project.
