> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/HandsOnLLM/Hands-On-Large-Language-Models/llms.txt
> Use this file to discover all available pages before exploring further.
# Chapter 9: Multimodal Large Language Models
> Adding vision capabilities to large language models through CLIP, BLIP-2, and other multimodal architectures
[](https://colab.research.google.com/github/HandsOnLLM/Hands-On-Large-Language-Models/blob/main/chapter09/Chapter%209%20-%20Multimodal%20Large%20Language%20Models.ipynb)
Chapter 9 explores multimodal large language models that can process and understand both images and text, opening up new possibilities for AI applications.
## Overview
Multimodal models combine vision and language understanding, enabling AI systems to perform tasks like image captioning, visual question answering, and image-text retrieval.
### Key Topics Covered
* CLIP (Contrastive Language-Image Pre-training)
* Image and text embeddings
* Similarity search across modalities
* BLIP-2 for image captioning and VQA
* Multimodal architectures
## CLIP: Bridging Vision and Language
CLIP learns to align images and text in a shared embedding space, enabling zero-shot image classification and cross-modal retrieval.
### Loading an Image
```python theme={null}
from urllib.request import urlopen
from PIL import Image
# Load an AI-generated image
puppy_path = "https://raw.githubusercontent.com/HandsOnLLM/Hands-On-Large-Language-Models/main/chapter09/images/puppy.png"
image = Image.open(urlopen(puppy_path)).convert("RGB")
caption = "a puppy playing in the snow"
```
### Setting Up CLIP
```python theme={null}
from transformers import CLIPTokenizerFast, CLIPProcessor, CLIPModel
model_id = "openai/clip-vit-base-patch32"
# Load tokenizer for text
clip_tokenizer = CLIPTokenizerFast.from_pretrained(model_id)
# Load processor for images
clip_processor = CLIPProcessor.from_pretrained(model_id)
# Main model for embeddings
model = CLIPModel.from_pretrained(model_id)
```
### Creating Text Embeddings
```python theme={null}
# Tokenize text
inputs = clip_tokenizer(caption, return_tensors="pt")
print(inputs)
```
**Output:**
```text theme={null}
{'input_ids': tensor([[49406, 320, 6829, 1629, 530, 518, 2583, 49407]]),
'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1]])}
```
```python theme={null}
# Convert back to tokens
tokens = clip_tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
print(tokens)
```
**Output:**
```text theme={null}
['<|startoftext|>', 'a', 'puppy', 'playing', 'in', 'the', 'snow', '<|endoftext|>']
```
```python theme={null}
# Create text embedding
text_embedding = model.get_text_features(**inputs)
print(text_embedding.shape)
```
**Output:**
```text theme={null}
torch.Size([1, 512])
```
### Creating Image Embeddings
```python theme={null}
# Preprocess image
processed_image = clip_processor(
text=None,
images=image,
return_tensors='pt'
)['pixel_values']
print(processed_image.shape)
```
**Output:**
```text theme={null}
torch.Size([1, 3, 224, 224])
```
```python theme={null}
# Create image embedding
image_embedding = model.get_image_features(processed_image)
print(image_embedding.shape)
```
**Output:**
```text theme={null}
torch.Size([1, 512])
```
CLIP embeddings are 512-dimensional vectors that exist in the same space for both images and text.
### Computing Similarity
```python theme={null}
# Normalize embeddings
text_embedding /= text_embedding.norm(dim=-1, keepdim=True)
image_embedding /= image_embedding.norm(dim=-1, keepdim=True)
# Calculate cosine similarity
text_embedding = text_embedding.detach().cpu().numpy()
image_embedding = image_embedding.detach().cpu().numpy()
score = text_embedding @ image_embedding.T
print(score)
```
**Output:**
```text theme={null}
[[0.33149636]]
```
## Multi-Image Comparison
Compare multiple images with multiple captions to find the best matches.
```python theme={null}
import numpy as np
# Load multiple images
puppy_path = "https://raw.githubusercontent.com/HandsOnLLM/Hands-On-Large-Language-Models/main/chapter09/images/puppy.png"
cat_path = "https://raw.githubusercontent.com/HandsOnLLM/Hands-On-Large-Language-Models/main/chapter09/images/cat.png"
car_path = "https://raw.githubusercontent.com/HandsOnLLM/Hands-On-Large-Language-Models/main/chapter09/images/car.png"
paths = [puppy_path, cat_path, car_path]
images = [Image.open(urlopen(path)).convert("RGBA") for path in paths]
captions = [
"a puppy playing in the snow",
"a pixelated image of a cute cat",
"A supercar on the road with the sunset in the background"
]
# Embed all images
image_embeddings = []
for image in images:
image_processed = clip_processor(images=image, return_tensors='pt')['pixel_values']
image_embedding = model.get_image_features(image_processed).detach().cpu().numpy()[0]
image_embeddings.append(image_embedding)
image_embeddings = np.array(image_embeddings)
# Embed all captions
text_embeddings = []
for caption in captions:
inputs = clip_tokenizer(caption, return_tensors="pt")
text_emb = model.get_text_features(**inputs).detach().cpu().numpy()[0]
text_embeddings.append(text_emb)
text_embeddings = np.array(text_embeddings)
# Calculate similarity matrix
from sklearn.metrics.pairwise import cosine_similarity
sim_matrix = cosine_similarity(image_embeddings, text_embeddings)
```
### Visualizing Similarities
The similarity matrix shows how well each image matches each caption:
| | "puppy in snow" | "pixelated cat" | "supercar sunset" |
| - | - | - | - |
| Puppy Image | **0.33** | 0.19 | 0.11 |
| Cat Image | 0.15 | **0.35** | 0.09 |
| Car Image | 0.08 | 0.13 | **0.31** |
Higher values indicate stronger semantic alignment between image and text.
## Sentence-BERT for CLIP
Simplify CLIP usage with the Sentence Transformers library.
```python theme={null}
from sentence_transformers import SentenceTransformer, util
# Load SBERT-compatible CLIP model
model = SentenceTransformer('clip-ViT-B-32')
# Encode images and text
image_embeddings = model.encode(images)
text_embeddings = model.encode(captions)
# Compute similarities
sim_matrix = util.cos_sim(image_embeddings, text_embeddings)
print(sim_matrix)
```
## BLIP-2: Advanced Multimodal Understanding
BLIP-2 combines a vision encoder, Q-Former, and language model for sophisticated image understanding tasks.
### Setting Up BLIP-2
```python theme={null}
from transformers import AutoProcessor, Blip2ForConditionalGeneration
import torch
# Load processor and model
blip_processor = AutoProcessor.from_pretrained(
"Salesforce/blip2-opt-2.7b",
revision="51572668da0eb669e01a189dc22abe6088589a24"
)
model = Blip2ForConditionalGeneration.from_pretrained(
"Salesforce/blip2-opt-2.7b",
revision="51572668da0eb669e01a189dc22abe6088589a24",
torch_dtype=torch.float16
)
# Send model to GPU
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
```
### Image Preprocessing
```python theme={null}
# Load image
car_path = "https://raw.githubusercontent.com/HandsOnLLM/Hands-On-Large-Language-Models/main/chapter09/images/car.png"
image = Image.open(urlopen(car_path)).convert("RGB")
# Preprocess image
inputs = blip_processor(image, return_tensors="pt").to(device, torch.float16)
print(inputs["pixel_values"].shape)
```
**Output:**
```text theme={null}
torch.Size([1, 3, 224, 224])
```
### Text Preprocessing
```python theme={null}
# Tokenizer information
print(blip_processor.tokenizer)
```
**Output:**
```text theme={null}
GPT2TokenizerFast(name_or_path='Salesforce/blip2-opt-2.7b', vocab_size=50265, ...)
```
```python theme={null}
# Tokenize text
text = "Her vocalization was remarkably melodic"
token_ids = blip_processor(image, text=text, return_tensors="pt")
token_ids = token_ids.to(device, torch.float16)["input_ids"][0]
# Convert to tokens
tokens = blip_processor.tokenizer.convert_ids_to_tokens(token_ids)
print(tokens)
```
**Output:**
```text theme={null}
['', 'Her', '_vocal', 'ization', '_was', '_remarkably', '_mel', 'odic']
```
BLIP-2 uses the GPT-2 tokenizer. The `Ġ` character represents spaces and is replaced with `_` for clarity.
## Use Case 1: Image Captioning
Generate descriptive captions for images automatically.
```python theme={null}
# Load image
image = Image.open(urlopen(car_path)).convert("RGB")
# Preprocess
inputs = blip_processor(image, return_tensors="pt").to(device, torch.float16)
# Generate caption
generated_ids = model.generate(**inputs, max_new_tokens=20)
generated_text = blip_processor.batch_decode(generated_ids, skip_special_tokens=True)
caption = generated_text[0].strip()
print(caption)
```
**Output:**
```text theme={null}
"an orange supercar driving on the road at sunset"
```
### Rorschach Test Example
```python theme={null}
# Load Rorschach inkblot
url = "https://upload.wikimedia.org/wikipedia/commons/7/70/Rorschach_blot_01.jpg"
image = Image.open(urlopen(url)).convert("RGB")
# Generate caption
inputs = blip_processor(image, return_tensors="pt").to(device, torch.float16)
generated_ids = model.generate(**inputs, max_new_tokens=20)
generated_text = blip_processor.batch_decode(generated_ids, skip_special_tokens=True)
print(generated_text[0].strip())
```
**Output:**
```text theme={null}
"a black and white ink drawing of a bat"
```
Image captioning can be used for accessibility (alt text), content moderation, search indexing, and automatic tagging.
## Use Case 2: Visual Question Answering
Ask questions about images and get specific answers.
```python theme={null}
# Load image
image = Image.open(urlopen(car_path)).convert("RGB")
# Create prompt
prompt = "Question: Write down what you see in this picture. Answer:"
# Process image and prompt together
inputs = blip_processor(
image,
text=prompt,
return_tensors="pt"
).to(device, torch.float16)
# Generate answer
generated_ids = model.generate(**inputs, max_new_tokens=30)
generated_text = blip_processor.batch_decode(generated_ids, skip_special_tokens=True)
answer = generated_text[0].strip()
print(answer)
```
**Output:**
```text theme={null}
"A sports car driving on the road at sunset"
```
### More VQA Examples
```python theme={null}
# What color is the car?
prompt = "Question: What color is the car? Answer:"
inputs = blip_processor(image, text=prompt, return_tensors="pt").to(device, torch.float16)
generated_ids = model.generate(**inputs, max_new_tokens=10)
answer = blip_processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
print(answer)
```
**Output:**
```text theme={null}
"orange"
```
```python theme={null}
# What time of day?
prompt = "Question: What time of day is it? Answer:"
inputs = blip_processor(image, text=prompt, return_tensors="pt").to(device, torch.float16)
generated_ids = model.generate(**inputs, max_new_tokens=10)
answer = blip_processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
print(answer)
```
**Output:**
```text theme={null}
"sunset"
```
## Architecture Overview
Processes images into visual features using a Vision Transformer (ViT).
Querying Transformer that bridges the vision and language modalities, extracting relevant visual information for the language model.
Generates text based on visual features and optional text prompts. BLIP-2 uses OPT or FlanT5.
## Key Takeaways
CLIP aligns images and text in a common space, enabling zero-shot classification and cross-modal retrieval.
BLIP-2 can automatically generate descriptive captions for images without task-specific training.
Ask specific questions about image content and get accurate, grounded answers.
Modern architectures combine specialized vision and language components for sophisticated reasoning.
## Applications
* **Accessibility**: Generate alt text for images automatically
* **Content Moderation**: Detect inappropriate visual content
* **E-commerce**: Search products by image or description
* **Education**: Interactive visual learning with VQA
* **Healthcare**: Medical image analysis with natural language queries
* **Robotics**: Enable robots to understand and describe their visual environment
## Model Comparison
| Model | Image Size | Text-Image | Captioning | VQA | Parameters |
| - | - | - | - | - | - |
| CLIP ViT-B/32 | 224×224 | ✓ | ✗ | ✗ | 151M |
| CLIP ViT-L/14 | 224×224 | ✓ | ✗ | ✗ | 428M |
| BLIP-2 OPT-2.7B | 224×224 | ✓ | ✓ | ✓ | 3.8B |
| BLIP-2 FlanT5-XXL | 224×224 | ✓ | ✓ | ✓ | 15.5B |
## Further Resources
* [CLIP Paper](https://arxiv.org/abs/2103.00020)
* [BLIP-2 Paper](https://arxiv.org/abs/2301.12597)
* [Hugging Face Transformers](https://huggingface.co/docs/transformers)
* [Sentence Transformers](https://www.sbert.net/)
* [OpenAI CLIP](https://github.com/openai/CLIP)
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.