Session 01 of 1 · The last lecture of Part 2
CEG5305 Introduction to Generative AI with Foundation Models
Junyuan Hong · AY2026/27 Semester 1
Before we start
These slides were prepared with the help of AI tools (Claude, by Anthropic), working from the reference slides and papers listed here.
Georgia Tech CS 7643, “Vision-Language Models”, Zsolt Kira, Spring 2026
Georgia Tech CS 8803-VLM, “Introduction & the VLM → agent arc”, Zsolt Kira, Fall 2026
Georgia Tech CS 8803-VLM, “LLaVA and Qwen3-VL”, Y. Wang, E. Anand, I. Narang, Fall 2026
Key papers (full references below): CLIP · Flamingo · LLaVA · Qwen3-VL
Recap · Lecture 3
| Part | Question | Key message |
|---|---|---|
| A | How does learning get rid of labels? | A pretext task makes the label from the data itself. |
| B | How do we learn features by contrast? | Pull matching pairs together and push the rest away. |
| C | What is the state of the art? | One frozen backbone serves many tasks. |
CLIP can rank labels you give it, but it cannot answer a question about the image.
Outline · Lecture 4 = Topic 7 · Part 2 so far
Today's question
An LLM reads a sequence of vectors. An image must become vectors that the LLM can use.
How the methods developed, and which problem each one solved.
LLaVA step by step, and the same steps in code with nanoVLM.
Advanced topics: more pixels, thinking, one model for all, acting in the world.
Part A
Reference slides: Georgia Tech CS 7643, Lecture 19, “Vision-Language Models”, Zsolt Kira, Spring 2026.
A · Multi-modality model before LLMs

Figure: Yang et al. (2023), Fig. 1.
A · Multi-modality model before LLMs
← Vision-only eraVision+Language fusionLarge Multi-Modal Models →
The timeline is redrawn as text, with every word of the deck page.
A · Multi-modality model before LLMs

A · Multi-modality model before LLMs
Language Models
Vision Models
A · Multi-modality model before LLMs

How should we encode this (representations?)?
How will they be learned?
Photos cropped from that page (the deck gives no source).
A · Multi-modality model before LLMs

The page content is cropped from the deck (the deck gives no source).
A · Multi-modality model before LLMs
Potential ways of representing an image?

Figure: Dosovitskiy et al. (2021), Fig. 1 (left part).
A · Multi-modality model before LLMs
→ Trained with contrastive loss; no generation capability by itself
→ More parameter-efficient cross-modal interaction via cross-attention layers
→ Simplest approach; leverages full LLM capability; most common in 2023–2024
Key design choice: freeze vs. fine-tune the vision encoder
→ Frozen ViT: faster alignment, less forgetting; trained ViT: more flexible
A · Multi-modality model before LLMs
Early Fusion
Late Fusion
A · Multi-modality model before LLMs Recap: Lecture 3

Method of alignment: Contrastive Learning
Data: 400M image-text pairs
Single image Encoder (ResNet, ViT)
Downside? Coarse-grained. Has to represent (somewhere) notion of objects, relationships, locations, etc.)
Figure: Radford et al. (2021), Fig. 1. CLIP was taught in Lecture 3.
A · Multi-modality model before LLMs

Figure: Minderer et al. (2022), Fig. 1.
A · Multi-modality model before LLMs

Figure: Minderer et al. (2023), Fig. A2 (first row; images from Wikimedia Commons).
Part A · How LLMs learned to see
A · Vision-language model

Cropped from the deck.
A · Vision-language model
→ Spatial structure vs. token sequence — must bridge dimensionality and distribution
The diagram is illustrative and added for this lecture.
A · Vision-language model
1 Connector: what sits between the two models
Linear Projector (MLP)
→ Fast, effective for high-res; maps ViT features → LLM hidden dim
Q-Former (BLIP-2)
→ 32–64 fixed queries collapse variable-length features into fixed-size representation
Perceiver Resampler (Flamingo)
→ Key advantage: variable resolution images → fixed latent budget
2 LLM side: change the LLM or not
LLM-side adapters
→ Reduces catastrophic forgetting of language knowledge during vision alignment
3 Training: which loss
Joint training objective
→ Image captioning loss, contrastive loss, or both during alignment pretraining
Hover over a box to read the original text. Boxes regrouped by role; headings 1 to 3 and the diagrams are added for this lecture.
A · Vision-language model
• Flamingo: Multimodal In-Context-Learning


Figure: Alayrac et al. (2022), Fig. 1 (rows 1 and 4).
A · Vision-language model

A Gated Cross attention mechanism is proposed to fuse images and text.
A · Vision-language model
Mix multi-modality context in embedding space

| Language Model | Pre-trained: 70B Chinchilla |
|---|---|
| Connection Module | Perceiver Resampler Gated Cross-attention + Dense |
| Vision Encoder | Pre-trained: Nonrmalizer-Free ResNet (NFNet) |
Figure: Alayrac et al. (2022), Fig. 3.
A · Vision-language model
![Flamingo Figure 5: the Perceiver Resampler; video frames at t=0, 1, 2 go through the vision encoder, get time embeddings, and are flattened into X_f; attention with Q = learned latent queries X and K = V = [X_f, X]; then a feed-forward layer; repeated num_layers times; beside it, the pseudo-code of perceiver_resampler](./assets/partA/a-alayrac2022-fig5.png)
Maps a variable size grid of visual features from the Vision Encoder to a fixed number of output token (5 in the figure.)
Figure: Alayrac et al. (2022), Fig. 5. The deck labels “fixed #” (the learned latent queries) and “pseudo code” are under the figure.
A · Vision-language model
The bullet list beside Alayrac et al. (2022), Fig. 5.
A · Vision-language model
![Flamingo Figure 9: Image-Text Pairs dataset [N=1, T=1, H, W, C] with a flamingo picture and 'This is an image of a flamingo.'; Video-Text Pairs dataset [N=1, T>1, H, W, C] with 'A kid doing a kickflip.'; Multi-Modal Massive Web (M3W) dataset [N>1, T=1, H, W, C] with 'Welcome to my website!', a dog picture, 'This is a picture of my dog.', a cat picture, 'This is a picture of my cat.'](./assets/partA/a-alayrac2022-fig9.png)
Figure: Alayrac et al. (2022), Fig. 9. The four datasets (right) are the sub-points of “Training on a mixture”.
A · Vision-language model

Figure: Alayrac et al. (2022), arXiv v1, Fig. 9.
A · Vision-language model
Problem 1
Understanding only: pick the best text for the photo.
Or generation only: write a caption.
→
One model trained for both.
Problem 2

Web text Tw is noisy.
→
A filter removes the web texts that do not match the image (red).
A captioner writes new texts Ts, and the same filter checks them.
The model trains only on the pairs that pass (green).
A · Vision-language model
• BLIP2

| Language Model | Pre-trained: FLAN-T5/OPT |
|---|---|
| Connection Module | Q-Former: Lightweight Querying Transformer |
| Vision Encoder | Contrastive pre-trained: EVA/CLIP |
Problem: Flamingo trains a large connection module and is costly. BLIP-2 freezes the image encoder and the language model, and trains only a small Q-Former.
Figure: Li et al. (2023), Fig. 1. The transition line is added from Li et al. (2023).
A · Vision-language model

Table: Li et al. (2023), Table 2.
A · Vision-language model

The diagram is cropped from that page.
Part A · How LLMs learned to see
A · State of art
A number of aspects improved over time:
A · State of art

Figure: Bai et al. (2023), Fig. 3.
A · State of art

Figure: Wang et al. (2024), Fig. 2.
A · State of art
The deck takes the figure from the Qwen3-VL README; the same figure is Bai et al. (2025), Fig. 1, and the crop is from that report.
A · State of art
Quoting https://github.com/QwenLM/Qwen3-VL?tab=readme-ov-file.
A · State of art
Quoting https://github.com/QwenLM/Qwen3-VL?tab=readme-ov-file.
A · State of art
The page has no title; “Summary” is ours.
Part A — history: how LLMs learned to see
Each step solved a problem of the one before it. Part B builds the simplest form of this, LLaVA, step by step in code.
Coming up in Part B: build and train a vision-language model, step by step, with code.
Part B
LLaVA step by step, and the same steps in code with nanoVLM.
Reference slides: Georgia Tech CS 8803-VLM, Lecture 6, “LLaVA and Qwen3-VL”, Y. Wang, E. Anand, I. Narang, Fall 2026. Code: nanoVLM, Hugging Face.
B · Problem & foundations RECAP · PART A
The methods of Part A take an image, and sometimes a short text. They return a label, a box, or a short text.
Input

An image, and sometimes a short text.
Output, by method (example outputs for this image)
| Method | Text input | Output |
|---|---|---|
| Classify (CLIP) | “a taxi”, “a bus”, “a bicycle” | best label: “a taxi” |
| Detect (OWL-ViT) | “person”, “ironing board” | one box for each |
| Caption (BLIP) | none | “a man ironing on the back of a taxi” |
| Short answer (BLIP-2) | “What color is the taxi?” | “yellow” |
Each method names what is in the image. None of them answers “what is unusual about this image?”.
Recap of Part A. The example outputs are illustrative.
B · Problem & foundations 01 / THE RESEARCH PROBLEM

THE SAME IMAGE, A DIFFERENT TARGET
“Describe the scene” and “What is unusual?” require different answers from the same pixels.
Question: “What is unusual about this image?”
Recognize
The scene contains a person, clothing, an ironing board, and a vehicle.
Relate
The activity is ironing; the setup is on a vehicle rather than in an ordinary indoor workspace.
Answer the instruction
Explain that unexpected activity–setting combination. Merely naming the objects misses the request.
Visual instruction following = relevant visual evidence + the requested response behavior—not an object list.
Part B · Make VLM chat
B · LLaVA: A simple instruction tuning method for VLM. 03 / WHY CAPTIONS ARE NOT ENOUGH
Rephrasing “describe this image” does not automatically create supervision for localization, comprehensive description, or reasoning.
Cheap caption expansion
Human: Describe the image.
Assistant: People try to fit their luggage in an SUV.

Three instructions, three different targets
Captions are useful for initial alignment; richer instruction–answer pairs teach the assistant to change its response with the task.
The deck cites Liu et al., §3, pp. 2–4; Table 14, p. 23. Image with boxes: Liu et al. (2023), Tables 1 and 14 (a COCO image).
B · LLaVA: A simple instruction tuning method for VLM. 02 / THE LLAVA IDEA
Large Language and Vision Assistant: reuse pretrained capabilities, then teach the visual interface and the interaction style.
1. Generate instruction following data for training models
→
→
2. A lightweight visual model connected to LLM
→
→
B · LLaVA: A simple instruction tuning method for VLM. 02 / THE LLAVA IDEA
ConversationFocused answers and multi-turn follow-up.
58K
Detailed descriptionBroad coverage of entities and relationships.
23K
Complex reasoningExplanations that connect evidence to implications.
77K
B · LLaVA: A simple instruction tuning method for VLM. 04 / HOW THE TEXT-ONLY TEACHER SEES
LLaVA generates visual instruction data without giving the GPT-4 teacher an image input.
The image being described

An offline supervision pipeline
Caption context. People are loading luggage into an SUV in an underground parking area.
person: [0.681, 0.242, 0.774, 0.694]
Normalized box coordinates encode spatial location.
GPT-4 teacher
Annotations + a few seed examples
Generated supervision
Question + target response
LLaVA training
+
→
→
The deck cites Liu et al., §3; Tables 13–14, pp. 22–23. Image with boxes: Liu et al. (2023), Tables 1 and 14.
B · LLaVA: A simple instruction tuning method for VLM. 08 / THE VISUAL INTERFACE
Trace the two input paths

What each component does
CLIP ViT-L/14 (frozen). Encode the image into Nᵥ visual vectors. These are learned features, not necessarily one object per vector.
Projection W. Apply the same learned map to every visual vector, changing the feature width to Vicuna’s embedding dimension.
LLM/Vicuna decoder (frozen). Attend to visual and text embeddings together, then predict the answer one token at a time.
The linear projection aligns feature dimensions. It does not produce a caption.
The deck cites Liu et al., §4.1, Figure 1, Equation (1), p. 4. Figure: Liu et al. (2023), Fig. 1. CLIP (Radford et al., 2021): Lecture 3.
B · LLaVA: A simple instruction tuning method for VLM. 09 / TRAINING STAGE 1
Learn W using image–caption pairs before asking the language model to adapt to complex multimodal interactions.
Only the projection is trainable
CLIP
Frozen
W
Train
Vicuna
Frozen
c = original caption; q = a brief-description request.
φ₀ stays fixed; gradients still pass through the decoder to W.
The deck cites Liu et al., §4.2; §5; Appendices C and E, Figure 7 (Liu et al., 2023).
B · LLaVA: A simple instruction tuning method for VLM. 10 / TRAINING STAGE 2
“End-to-end” in this paper does not mean every component updates: the CLIP vision encoder remains frozen.
Two task-specific fine-tuning branches
Trainable parameters. Update θ = {W, φ}; retain the frozen CLIP encoder.
Multimodal chat
158K generated examples; 3 epochs.
Learn conversation, detailed description, and reasoning.
ScienceQA
Separate task-specific training.
Generate reasoning, then a multiple-choice answer.
One sequence, selective prediction targets
SCHEMATIC MULTI-TURN SERIALIZATION
Image + user Q₁
Assistant A₁
User Q₂
Assistant A₂
A marks assistant-target positions (and stop behavior).
Past questions and answers remain conditioning context.
Stage 1 teaches the interface; Stage 2 changes the response policy. Questions provide context, while assistant answers supply the loss.
The deck cites Liu et al., §4.2, Table 2; §5; §5.2 (Liu et al., 2023).
B · LLaVA: A simple instruction tuning method for VLM.
What we ask
LLaVA-Bench (COCO). 30 images, 90 questions: conversation, detailed description and complex reasoning.
LLaVA-Bench (In-the-Wild). 24 images, 60 questions: memes, paintings, sketches, indoor and outdoor scenes.
ScienceQA. Multiple-choice questions. The score is accuracy.
How an answer gets a score
→
→
→
→
→
Source: Liu et al. (2023), Sec. 5 and Table 5.
Part B · Make VLM chat
B · A small demo: nanoVLM STEP 1 · THE MAP
| LLaVA part | nanoVLM file | Lines |
|---|---|---|
| Vision encoder | models/vision_transformer.py | 251 |
| Projection W | models/modality_projector.py | 45 |
| Language model | models/language_model.py | 681 |
| All three, joined | models/vision_language_model.py | 310 |
New, compared with nanoGPT
An image encoder: a ViT (Lecture 3).
A projector: a pixel shuffle, then one linear layer.
The language model reads vectors, not token ids.
It uses RoPE, RMSNorm, a gated MLP and GQA: the parts that Lecture 1 named.
nanoVLM (Wiedmann et al., 2025), commit 4e0c096 (27 Oct 2025), Apache-2.0; line counts from wc -l at that commit. nanoGPT: Karpathy (2022). Map of parts: Liu et al. (2023), Fig. 1.
B · A small demo: nanoVLM STEP 1 · THE CHECKPOINT
Parameters, from the checkpoint file
The projector is 3.1% of all parameters (computed).
Why this one: generate.py loads it by default, and the current code reads it.
Checkpoint lusxvr/nanoVLM-230M-8k (MIT licence): config.json and the tensor shapes in model.safetensors, revision a76fa2d, read 10 Oct 2026. Code: nanoVLM (Wiedmann et al., 2025), commit 4e0c096. Tschannen et al. (2025); Allal et al. (2025).
B · A small demo: nanoVLM STEP 2 · MATH
H = image side, p = patch side.
LLaVA (2023), 224 px: \((224/14)^2 = 16^2 = 256\)
LLaVA-1.5: \((336/14)^2 = 24^2 = 576\)
nanoVLM, one tile: \((512/16)^2 = 32^2 = 1{,}024\)
Pixel shuffle, factor 4: \(1{,}024 / 4^2 = 64\) tokens per tile
Square photo, 16 tiles + 1 global view: \(17 \times 64 = 1{,}088\)
ALL RESULTS ON THIS SLIDE ARE COMPUTED
LLaVA (2023) at 224 px and LLaVA-1.5 with CLIP-ViT-L-336px, patch 14 (Liu et al., 2024, Table 2). nanoVLM (Wiedmann et al., 2025), commit 4e0c096: data/custom_transforms.py resizes the long side to max_img_size = 2,048 and cuts 512 × 512 tiles; data/processors.py adds the global view. The 17 tile-marker tokens are not counted here.
B · A small demo: nanoVLM STEP 3 · THE PROJECTOR
The encoder gives a 32 × 32 grid of vectors.
Pixel shuffle cuts the grid into 4 × 4 blocks. It puts the 16 vectors of a block end to end.
1,024 vectors of width 768
→ 64 vectors of width 12,288
Neighbor patches stay together. The LLM gets 16 times fewer image tokens.
Numbers for lusxvr/nanoVLM-230M-8k (mp_pixel_shuffle_factor 4). nanoVLM (Wiedmann et al., 2025), models/modality_projector.py, commit 4e0c096. Pixel shuffle in small VLMs: Marafioti et al. (2025), SmolVLM, Fig. 4.
B · A small demo: nanoVLM STEP 3 · THE PROJECTOR
self.input_dim = cfg.vit_hidden_dim * (cfg.mp_pixel_shuffle_factor**2) self.output_dim = cfg.lm_hidden_dim⋮ self.proj = nn.Linear(self.input_dim, self.output_dim, bias=False)⋮def forward(self, x): x = self.pixel_shuffle(x) x = self.proj(x)
input_dim \(= 768 \times 4^2 = 12{,}288\)
output_dim \(= 576\), no bias
weights \(= 12{,}288 \times 576 = 7{,}077{,}888\)
COMPUTED
nanoVLM (Wiedmann et al., 2025), models/modality_projector.py (45 lines), commit 4e0c096, Apache-2.0 (common indentation removed). lusxvr/nanoVLM-230M-8k stores MP.proj.weight with shape [576, 12288].
B · A small demo: nanoVLM STEP 4 · IMAGE INTO THE SEQUENCE
if hasattr(tokenizer, "global_image_token"): image_string += tokenizer.global_image_token image_string += tokenizer.image_token * mp_image_token_length if n_h == 1 and n_w == 1: # If there is only one patch, treat it as the global image continuefor i in range(n_h): for j in range(n_w): image_string += getattr(tokenizer, f'r{i+1}c{j+1}') image_string += tokenizer.image_token * mp_image_token_length
<|im_start|>user⏎ <|global_image|> <|image|>×64 <row_1_col_1> <|image|>×64 … <row_4_col_4> <|image|>×64 What is this?<|im_end|>⏎ <|im_start|>assistant⏎
Each placeholder is one token. The next step puts the image vectors at these places.
nanoVLM (Wiedmann et al., 2025), data/processors.py, lines 33–41, commit 4e0c096, Apache-2.0 (common indentation removed). Text line: our checkpoint’s chat template, a square photo; ⏎ = line break.
B · A small demo: nanoVLM STEP 4 · IMAGE INTO THE SEQUENCE
token_embd = self.decoder.token_embedding(input_ids) # [B, T_sequence, D_lm]if images_tensor is not None: image_embd = self.vision_encoder(images_tensor) image_embd = self.MP(image_embd) # [num_images, mp_image_token_length, D_lm] token_embd = self._replace_img_tokens_with_embd(input_ids, token_embd, image_embd)⋮mask = (input_ids == self.tokenizer.image_token_id)updated_token_embd[mask] = image_embd.view(-1, image_embd.size(-1)).to(updated_token_embd.dtype) # torch flattens before assigning
The LLM reads vectors, not token ids (Lecture 1). Every token gets its embedding. Then the 64 vectors per tile go to the <|image|> places, in tile order.
nanoVLM (Wiedmann et al., 2025), models/vision_language_model.py, lines 64–69 (forward) and 46–47 (_replace_img_tokens_with_embd), commit 4e0c096, Apache-2.0 (common indentation removed).
B · A small demo: nanoVLM STEP 5 · THE LOSS
The image tokens are in \(x_{\le t}\): the model reads them, but it does not predict them. The answer includes <|im_end|>, so the model learns when to stop.
Our notation for the \(\mathcal{A}\) in LLaVA’s stage-2 loss (Train it, stage 2) (Wang, Anand & Narang, 2026); see Lecture 1, cross entropy and Lecture 2, SFT loss · mask. Tokens are schematic. Mask and labels: nanoVLM (Wiedmann et al., 2025), data/datasets.py, commit 4e0c096.
B · A small demo: nanoVLM STEP 5 · THE LOSS
data/datasets.py if msg["role"] == "assistant": start = cursor + self.prefix_len end = cursor + seg_len mask[start:end] = [1] * (end - start) # attend to these tokens⋮labels = input_ids.clone().masked_fill(~mask, -100)labels = labels.roll(-1) # Shift labels for causal LMlabels[-1] = -100 # Last token has no targetmodels/vision_language_model.pyloss = F.cross_entropy(logits.reshape(-1, logits.size(-1)), targets.reshape(-1), ignore_index=-100)
The mask is 1 on each answer. The labels shift by one place, so each place predicts the next token. ignore_index=-100 skips the other places, and the mean is over the answer tokens only.
nanoVLM (Wiedmann et al., 2025), data/datasets.py, lines 97–100 and 146–148; models/vision_language_model.py, line 78; commit 4e0c096, Apache-2.0 (common indentation removed).
B · A small demo: nanoVLM NANOVLM
models/config.pylr_mp: float = 0.00512lr_vision_backbone: float = 5e-5 #0.0005 #lr_language_backbone: float = 5e-5 #0train.pyif train_cfg.lr_mp > 0: param_groups.append({'params': list(model.MP.parameters()), 'lr': train_cfg.lr_mp})
The new projector starts from random weights. Its learning rate is 102.4 times larger (computed). A learning rate of 0 freezes a part (lines 312–314).
nanoVLM (Wiedmann et al., 2025), models/config.py lines 59–61, train.py lines 310–311 (all three parts: lines 309–326), commit 4e0c096, Apache-2.0 (common indentation removed). LLaVA: Liu et al. (2023), §4.2.
B · A small demo: nanoVLM GENERATE
for _ in range(max_new_tokens):⋮ next_token_id = torch.multinomial(probs, num_samples=1)⋮ next_token_embed = self.decoder.token_embedding(next_token_id) # [B, 1, D_lm]⋮ decode_step_output, kv_cache_list = self.decoder( next_token_embed, attention_mask=attention_mask, kv_cache=kv_cache_list, start_pos=current_token_start_pos )
Prefill. One pass reads the image tokens and the question. It fills the KV cache (lines 98–103).
Decode. Each new token reads the cache and adds only its own key and value.
nanoVLM (Wiedmann et al., 2025), models/vision_language_model.py, lines 116–143 of generate (sampling: top-k 50, top-p 0.9, temperature 0.5, line 83), commit 4e0c096, Apache-2.0 (common indentation removed).
B · A small demo: nanoVLM DEMO
SmolVLM-256M has the same kind of parts: a SigLIP B/16 encoder (ours: SigLIP 2), a pixel shuffle and SmolLM2-135M.
Demo: Hugging Face Space HuggingFaceTB/SmolVLM-256M-Instruct-WebGPU (Apache-2.0). SmolVLM-256M: Marafioti et al. (2025). SigLIP: Lecture 3. nanoVLM: Wiedmann et al. (2025).
Part B — deep dive: build and train a VLM
LLaVA showed the recipe in 2023. nanoVLM shows the same recipe in four PyTorch files, about 1,300 lines.
Coming up in Part C: what changed since 2023, and where vision-language models are today.
Part C
Qwen3-VL, then one model for all modalities, agents and robots. Each as why, how and results.
Reference slides: Georgia Tech CS 8803-VLM, Fall 2026: Lecture 6, “LLaVA and Qwen3-VL” (Wang, Anand, Narang), and Lecture 1 (Kira).
C · Qwen3-VL 01 / The research problem
A strong language model is the starting point—not a capability to sacrifice when adding vision.
How do we add rich visual understanding, long context, and useful actions—while preserving language ability?
Dense: 2B / 4B / 8B / 32B
MoE: 30B-A3B / 235B-A22B
Visual inputs
Images, video, documents
Vision encoder
Dynamic-resolution features
Merger + DeepStack
Compress and transfer
Qwen3 decoder
Reason, ground, use tools
Top half. Deck source: Report §§1–2, pp. 2–4 (Bai et al., 2025).
C · Qwen3-VL 01 / The research problem
Inject multiple ViT depths.
Balance position frequencies.
Make elapsed time explicit.
The system goal: strong perception + reliable information transfer + 256K multimodal context + effective training.
Bottom half. Deck source: Report §§1–2, pp. 2–4 (Bai et al., 2025).
C · Qwen3-VL 05 / DeepStack: the mechanism
The important dimension is depth: add information to existing visual positions instead of appending more positions.
ViT feature level 3
Spatial feature grid
LLM layer 3
Add projected features
ViT feature level 2
Spatial feature grid
LLM layer 2
Add projected features
ViT feature level 1
Spatial feature grid
LLM layer 1
Add projected features
Left panel; level diagram rebuilt from that page, words unchanged. Deck source: Report §2.2 and Fig. 1, pp. 3–4 (Bai et al., 2025).
C · Qwen3-VL 05 / DeepStack: the mechanism
\(D_k = M_k\big(F^{(a_k)}\big) \in \mathbb{R}^{N_v \times d_L}, \qquad k = 1,2,3\)
\(H^{(k)} = T_k\big(H^{(k-1)}\big) + S_{\mathcal{V}}(D_k)\)
Meaning. \(M_k\) maps a selected ViT feature set to LLM width; \(S_{\mathcal{V}}\) places it at the corresponding visual positions.
Cost. Extra feature extraction, mergers, and additions—but no additional context positions from stacking.
Keep the distinction: multi-level features ≠ more image scales; unchanged context length ≠ zero extra computation.
Right panel and closing line. Deck source: Report §2.2 and Fig. 1, pp. 3–4 (Bai et al., 2025) • equations are schematic.
C · Qwen3-VL 18 / Read the results by capability
Selected within-family comparisons for Qwen3-VL-235B-A22B, as reported in Table 2.
| Benchmark | Instruct | Thinking | What the row tests |
|---|---|---|---|
| MathVision | 66.5 | 74.6 | Visual mathematical reasoning |
| MUIRBENCH | 73.0 | 80.1 | Multi-image understanding |
| DocVQA test | 97.1 | 96.5 | Document question answering |
| OCRBench | 920 | 875 | Text recognition; own score scale |
| Video-MME, no subtitles | 79.2 | 79.0 | General video understanding |
Table; numbers checked against the report's Table 2, p. 15. Deck source: Report Table 2, p. 15; §§5.2–5.4, pp. 14–17 (Bai et al., 2025).
C · Qwen3-VL 18 / Read the results by capability
More reasoning does not guarantee better visual reading.
Compare within each benchmark; score scales differ.
A 256K context, a larger model, or a Thinking label is a capability setting—not a guarantee of correctness on every input.
The two cards and closing line. Deck source: Report Table 2, p. 15; §§5.2–5.4, pp. 14–17 (Bai et al., 2025).
C · Qwen3-VL 13 / Multimodal reasoning data
A visual-math error can arise from misreading the diagram, faulty reasoning, or both.
Perception supervision. Programmatically render diagrams; generate 1M point-grounding samples, 2M perception QA pairs, and 6M rich diagram captions.
Reasoning supervision. Curate over 60M educational exercises and over 12M image-paired long-reasoning samples; validate answers and filter trajectories.
Perception: “Which lines intersect?”
Reasoning: “What follows from that intersection?”
Left panel. Deck source: Report §3.2.8, pp. 8–9 (Bai et al., 2025).
C · Qwen3-VL 13 / Multimodal reasoning data
Give only the text to Qwen3-30B-nothink
Test visual-mathematics training questions.
↓
Can it solve the problem correctly?
YES → discard
Text alone is enough for this model.
NO → retain for curation
A candidate requiring visual evidence.
This is an empirical filter—not proof of nonzero conditional mutual information.
High visual-benchmark accuracy is less meaningful when a model can ignore the visual input and exploit textual shortcuts.
Right panel and closing line. Deck source: Report §3.2.8, pp. 8–9; §4.2.2, p. 11 (Bai et al., 2025).
C · Qwen3-VL 16 / Reinforcement learning
Qwen3-VL combines reasoning RL with general RL for instruction following, alignment, and behavioral correction.
\(\displaystyle \max_{\theta}\; \mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)}\big[R(x, y)\big]\)
Tasks. Mathematics, code, logical reasoning, visual grounding, and puzzles with rule- or executor-based verification.
Training method. The report uses SAPO, a smooth adaptive policy-gradient method; it does not detail its update equations here.
General RL adds instruction-following, preference, language-consistency, and targeted error correction, using rules and model judges.
Left panel. Deck source: Report §4.4, pp. 11–12 (Bai et al., 2025) • objective and sampling probability are explanatory.
C · Qwen3-VL 16 / Reinforcement learning
Remove initially unproductive queries. Sample 16 responses with a strong checkpoint; discard queries with no correct responses.
Then remove easy queries per model. Sample 16 responses and filter pass rates above 90%. The curated pool is about 30K queries.
\(\Pr(\text{no success in 16 trials}) = (1-p)^{16}\)
Independent-trial illustration: zero observed successes does not mean a task is impossible.
Reward design and data selection determine the learning signal. A verifier or judge is not automatically immune to reward hacking.
Right panel and closing line. Deck source: Report §4.4, pp. 11–12 (Bai et al., 2025) • objective and sampling probability are explanatory.
C · Qwen3-VL 19 / Evidence and limitations
The report offers concrete controlled evidence for DeepStack—and narrower evidence for extreme context length.
Internal 15B-A2B model; 200B pretraining tokens; validation evaluation with no post-training.
| Metric | Baseline | DeepStack |
|---|---|---|
| Reported average | 74.7 | 76.0 |
| InfoVQA | 71.9 | 74.2 |
| DocVQA | 89.5 | 91.1 |
| MMMU | 52.9 | 54.1 |
Left panel; numbers checked against the report's Table 12, p. 24. Deck source: Report Table 12, p. 24 (Bai et al., 2025).
C · Qwen3-VL 19 / Evidence and limitations
100%
Up to 256K tokens / 30 min
99.5%
≈1M tokens with YaRN extension
Scope. Locate an inserted salient frame and answer its question. This does not establish equally strong long-video reasoning.
Comparison caveat. Video frame caps differed: Qwen 2,048; Gemini 512; GPT-5 256; Claude 100. The authors flag imperfect fairness.
1M tokens is an extrapolation experiment—not the native 256K training window. Headline benchmark gains do not identify a single cause.
Right panel and closing line. Deck source: Report Fig. 3 and §5.12.3, p. 25; §5.9, p. 20 (Bai et al., 2025).
Part C · Advanced topics and today's state of the art
C · Part 1: Building a VLM One Model That Reads and DrawsWeek 5

Chameleon: one mixed-modal autoregressive model, early fusion, image and text tokens in a single vocabulary.

BAGEL: separate understanding and generation experts sharing one attention stack — a different bet on the same goal.
Figures: Chameleon Team (2024), Fig. 1; Deng et al. (2025), Fig. 2.
C · Part 1: Building a VLM Other Modalities

The page has only this figure and its source line (deck: "Yang et al., Binding Touch to Everything: Learning Unified Multimodal Tactile Representations, CVPR 2024"); its page title "Other Modalities" is used as the slide title. Figure: Yang et al. (2024), Fig. 1.
C · Part 1: Building a VLM Result
| Chameleon-34B against | Chameleon wins | Tie | Chameleon loses |
|---|---|---|---|
| Gemini-Pro (text answer) | 53.5% | 31.2% | 15.3% |
| GPT-4V (text answer) | 46.0% | 31.4% | 22.6% |
| Gemini-Pro with images added (Gemini+) | 41.5% | 34.5% | 24.0% |
| GPT-4V with images added (GPT-4V+) | 35.8% | 31.6% | 32.6% |
Test: each prompt asks for an answer with text and images. Three human annotators see two answers in random order. They pick the better answer or "about the same".
Closest case: against GPT-4V+ the two models are almost equal (35.8% against 32.6%). Gemini and GPT-4V write text only, so the "+" versions have images added by the authors.
Source: Chameleon Team (2024), §4.1 (1,048 prompts) and §4.2.2, Figure 9b, pp. 11–13; read 2 Nov 2026.
Part C · Advanced topics and today's state of the art
C · Part 3: Making It Act CS 8803-VLM, part 3
Tools, browsers, and the whole desktop — Weeks 9–11.
| 1 | Week 1 | Pixels & words | Patches and sub-words become one token stream. | Transformer · ViT |
| 2 | Week 2 | Aligned representations | Vision and language share an embedding space. | CLIP → SigLIP 2 · DINOv3 |
| 3 | Weeks 3–5 | Vision-language models | A language model that can see, and generate. | Flamingo · LLaVA → Qwen3-VL |
| 4 | Weeks 6–9 | Reasoning & reliability | Think before answering; know when you are wrong. | MM-CoT · RLVR · POPE |
| 5 | Weeks 9–11 | Agents | Perceive, plan, and take actions in software. | ViperGPT · Mind2Web · OSWorld |
| 6 | Weeks 11–13 | Action & world models | Act in the physical world; imagine what follows. | RT-2 · OpenVLA · DreamZero |
The same roadmap as p. 24, stage 5 highlighted. SigLIP 2: Lecture 3.
C · Part 3: Making It Act Computer Use: The Whole DesktopWeek 10 · seminal

Figure: Xie et al. (2024), Fig. 1.
C · Part 3: Making It Act Result
| Who does the task | Input to the model | Tasks finished |
|---|---|---|
| Humans (not familiar with the software) | – | 72.36% |
| Best model: GPT-4 | Accessibility tree (a text list of the screen elements) | 12.24% |
| GPT-4V | Screenshot only | 5.26% |
| Claude-3-Opus | Screenshot only | 2.42% |
The test: 369 tasks on real computers, checked by a script.
The gap: 60 points. The paper names GUI grounding (linking a plan to the right place on the screen) as a main cause.
Source: Xie et al. (2024), §4.2, Table 5, p. 10 (human row and model rows); abstract (369 tasks); read 2 Nov 2026. 2024 baseline models.
Part C · Advanced topics and today's state of the art
C · Part 4: Acting in the World CS 8803-VLM, part 4
Vision-language-action models, real-time control, and world models — Weeks 11–13.
| 1 | Week 1 | Pixels & words | Patches and sub-words become one token stream. | Transformer · ViT |
| 2 | Week 2 | Aligned representations | Vision and language share an embedding space. | CLIP → SigLIP 2 · DINOv3 |
| 3 | Weeks 3–5 | Vision-language models | A language model that can see, and generate. | Flamingo · LLaVA → Qwen3-VL |
| 4 | Weeks 6–9 | Reasoning & reliability | Think before answering; know when you are wrong. | MM-CoT · RLVR · POPE |
| 5 | Weeks 9–11 | Agents | Perceive, plan, and take actions in software. | ViperGPT · Mind2Web · OSWorld |
| 6 | Weeks 11–13 | Action & world models | Act in the physical world; imagine what follows. | RT-2 · OpenVLA · DreamZero |
The same roadmap as p. 24, stage 6 highlighted. SigLIP 2: Lecture 3.
C · Part 4: Acting in the World Actions as TokensWeek 12 · seminal

Discretize the end-effector delta, put it in the vocabulary, and co-fine-tune on web VQA.
The page has no headline, so its page title is the slide title. Figure: Brohan et al. (2023), Fig. 1.
C · Part 4: Acting in the World Generalist VLAs and the Data ProblemWeek 12 · seminal

An open 7B VLA trained on 970k cross-embodiment episodes — the bottleneck is trajectories, not parameters.
The page has no headline, so its page title is the slide title. Figure: Kim et al. (2024), Fig. 1.
C · Part 4: Acting in the World Fast, Continuous ControlWeek 13

Diffusion Policy denoises a whole action sequence conditioned on recent observations, instead of emitting one step at a time.
First fix (deck: "Chi et al., Diffusion Policy, RSS 2023 · Tang et al., VLASH, 2025/26 — figures from the papers."). Figure: Chi et al. (2023), Fig. 2 (arXiv v5, the extended journal version).
C · Part 4: Acting in the World Fast, Continuous ControlWeek 13

Asynchronous inference acts on predictions made for a future state, so inference delay stops turning into jerky motion.
Second fix. Figure: Tang et al. (2025), Fig. 3.
C · Part 4: Acting in the World World Models: Learning to ImagineWeek 13 · seminal

Figure: Ha & Schmidhuber (2018), Fig. 4.
C · Part 4: Acting in the World World-Action ModelsWeek 13 · frontier

The page has no headline, so its page title is the slide title. Figure: Ye et al. (2026), Fig. 1.
C · Part 4: Acting in the World Result
| Task | Agent | Average score |
|---|---|---|
| VizDoom Take Cover | Best earlier result (OpenAI Gym leaderboard) | 820 ± 58 |
| VizDoom Take Cover | World model, trained only inside its dream, tested in the real game | 1092 ± 556 |
| CarRacing-v0 | Best earlier deep RL method (A3C, discrete) | 652 ± 10 |
| CarRacing-v0 | World model (V, M and C together) | 906 ± 21 |
Score: Take Cover: time steps survived (solved above 750). CarRacing-v0: average reward over 100 trials (solved at 900).
Dream training: model M predicts the next frame, with temperature τ = 1.15. The spread (± 556) is large.
Source: Ha & Schmidhuber (2018), §3.3 and Table 1 (CarRacing-v0, p. 6); §4.4 and Table 2 (Take Cover, τ = 1.15, pp. 8 and 10); read 2 Nov 2026.
Part C · Advanced topics and today's state of the art
C · Today's state of the art, and safety Read on 10 Oct 2026Check again the week of 4 Nov 2026
| Rank (interval) | Model on the leaderboard | Developer | Weights | Score |
|---|---|---|---|---|
| 1 (1–10) | claude-fable-5-high | Anthropic | Closed | 1308 |
| 2 (1–19) | qwen3.8-max | Alibaba | Closed | 1301 |
| 6 (1–26) | gemini-3.7-flash-high | Closed | 1297 | |
| 8 (1–37) | gpt-6.1-sol-max | OpenAI | Closed | 1294 |
| 35 (11–47) | glm-5.3-flash | Z.ai | Open (MIT) | 1273 |
| 44 (34–58) | gemma-4-31b | Open (Apache 2.0) | 1262 | |
| 83 (74–91) | qwen3-vl-235b-a22b-instruct | Alibaba | Open (Apache 2.0) | 1215 |
Score: a rating from human votes on two answers to one image. The intervals overlap: the top rank is not certain.
The top changes fast. In May 2026, CS231n named Gemini the best closed VLM. Qwen3-VL (Nov 2025) is now rank 83.
Vision Arena leaderboard, arena.ai/leaderboard/vision (page date 9 Oct 2026; read 10 Oct 2026): rank, 95% rank interval, model name and licence as listed; one closed row per developer, the top two open rows, and Qwen3-VL. Open weight vs fully open source: CS231n pp. 88–89. "Gemini widely considered the best proprietary Vision-Language model": Fei-Fei et al. (2026b), CS231n lecture 16, p. 86.
C · Today's state of the art, and safety LO5

Often it does not refuse. On six open VLMs, 82.50% of the requests got a harmful answer (average).
Why: the LLM learned to refuse harmful text. The image path did not get that safety training.

Change each pixel by a very small amount (ε = 16/255). The change is hard to see. One such image made MiniGPT-4 obey harmful requests that it refused before.
The same problem as prompt injection through a tool result (Lecture 2 session 02): data and instructions share one input.
Gong et al. (2025), Fig. 1 and abstract (82.50%, six open-source LVLMs). Qi et al. (2024), Fig. 1 (top part only; the model outputs in the figure are not shown) and its caption (ε = 16/255, MiniGPT-4).
Part C — advanced topics and today's state of the art
Part C: six methods, each as why, how and results: the Qwen3-VL recipe, thinking, one model for all modalities, computer-use agents, robots and world models; then today's leaderboard and one safety risk.
Close of Part 2
What each lecture gave us, and what is still open.
Close of Part 2 · Topic 7
| Syllabus phrase | Where today |
|---|---|
| Extending LLMs to other data | Parts A and B |
| … and prompting | Part B, last segment |
| Especially visual data (LLVMs) | All three parts |
| (CLIP) | Lecture 3; Part A |
| Applications; other multi-modal models | Part C |
Close of Part 2 · the plan from the first lecture
| Lecture | What it gave a multi-modal model |
|---|---|
| 1 · 01 Architecture | The Transformer and next-token prediction |
| 1 · 02 Training | Pre-training at scale, and cheap adaptation (LoRA) |
| 2 · 01 Post-training | Instruction tuning and RL: answers, not continuations |
| 2 · 02 Prompting | Prompts, reasoning and agents with a fixed model |
| 3 Self-supervised | Image encoders trained without labels, such as CLIP |
| 4 Multi-modal | Connect the encoder to the LLM, train it, prompt it, let it act |
Close of Part 2 · Still open
Close of Part 2 · Where to go next
Close of Part 2 · Revision
The format and the logistics of the exam are on Canvas.
Where we got to
An encoder turns the image into vectors, a small projector maps them to the LLM's width, and the LLM learns from image–text data and instructions. The same recipe now reasons and acts.
Thank you for taking Part 2. Questions: on Canvas.