Part 2 - Lecture 4

Extending LLMs to Large
Multi-Modal Models

Session 01 of 1 · The last lecture of Part 2

CEG5305 Introduction to Generative AI with Foundation Models
Junyuan Hong · AY2026/27 Semester 1

2026.11.04v2

Before we start

About these slides

These slides were prepared with the help of AI tools (Claude, by Anthropic), working from the reference slides and papers listed here.

Key papers (full references below): CLIP · Flamingo · LLaVA · Qwen3-VL

Recap · Lecture 3

Last week: train foundation models without labels

PartQuestionKey message
AHow does learning get rid of labels?A pretext task makes the label from the data itself.
BHow do we learn features by contrast?Pull matching pairs together and push the rest away.
CWhat is the state of the art?One frozen backbone serves many tasks.

CLIP can rank labels you give it, but it cannot answer a question about the image.

Outline · Lecture 4 = Topic 7 · Part 2 so far

Today: from seeing, to building, to acting

Today · 4 Nov · 3 h · the last lecture of Part 2
AHow LLMs learned to see18:07 · 38 min
☕Break18:45 · 10 min
BMake VLM chat18:55 · 50 min
☕Break19:45 · 10 min
·Close of Part 220:34 · 12 min

Today's question

How did LLMs learn to see, and how do you build one?

An LLM reads a sequence of vectors. An image must become vectors that the LLM can use.

Part A · history

How the methods developed, and which problem each one solved.

Part B · deep dive

LLaVA step by step, and the same steps in code with nanoVLM.

Part C · today

Advanced topics: more pixels, thinking, one model for all, acting in the world.

Part A

How LLMs learned to see

Reference slides: Georgia Tech CS 7643, Lecture 19, “Vision-Language Models”, Zsolt Kira, Spring 2026.

A · Multi-modality model before LLMs

Motivation: Vision-Language Models

MM-REACT Figure 1: eight example tasks with a question and the model's answer: visual math and text reasoning, visual-conditioned joke or meme, spatial and coordinate understanding, visual planning and prediction, multi-image reasoning, multi-hop document understanding, open-world concept understanding, video summarization and event localization
Yang et al., MM-ReAct MM-ReAct: Prompting ChatGPT for Multimodal Reasoning and Action

Figure: Yang et al. (2023), Fig. 1.

A · Multi-modality model before LLMs

History and Evolution of Multi-Modal AI

2012AlexNet
2017Transformer
2020GPT-3
2021CLIPDALL·E
2022FlamingoStableDiff
2023GPT-4VLLaVA
2024+Gemini/ClaudeQwen-VL(1-3)InternVL…

← Vision-only eraVision+Language fusionLarge Multi-Modal Models →

The timeline is redrawn as text, with every word of the deck page.

A · Multi-modality model before LLMs

The Early Days: Neural Image Captioning

Diagram: an image goes through a VGGNet image embedding (4096-dim), then a Linear layer, into the first of a chain of RNN cells; each RNN outputs P(next); the words <start>, Two, people, and, two horses. feed the RNN cells
  • Features:
    • Pre-trained visual encoder (vector)
    • Linear projection layer to map to “captionable” space
    • Vector serves as initial input to RNN

A · Multi-modality model before LLMs

Unimodal Models Recap

Language Models

  • Transformer-based auto-regressive LLMs
  • BERT (masked language modeling)
  • GPT series (causal language models)
  • Input: tokenized text sequences
  • Strong: reasoning, generation, QA
  • Weak: cannot perceive visual input

Vision Models

  • CNNs: local feature extraction (ResNet)
  • ViT: image split into 16×16 patches
  • DINO, MAE: self-supervised vision
  • Input: pixel grids → patch embeddings
  • Strong: object recognition, detection
  • Weak: lack semantic language grounding

A · Multi-modality model before LLMs

VLMs: Considerations

Three photos (people with a boat on a beach, a softball player running past a catcher and an umpire, two dogs on a gravel shore), each with an arrow to a question mark

How should we encode this (representations?)?

How will they be learned?

Photos cropped from that page (the deck gives no source).

A · Multi-modality model before LLMs

VLMs: Considerations

Diagram: three photos with arrows to a Visual Feature Space box and a question mark; a Language Model box that holds a Features Space (Encoder) and a Decoder, with captions in English and German around it; the questions How/what should we train? Using what data? What tasks can we do? What should the interface be?

The page content is cropped from the deck (the deck gives no source).

A · Multi-modality model before LLMs

Image Representations

Potential ways of representing an image?

  • Image encoder
    • Any architecture: ResNet, Vision transformer (ViT)
    • Randomly initialized, SL/SSL pre-trained
      • DINOv1-3, MAE, etc.
ViT Figure 1, left part: the image is split into patches; a linear projection of the flattened patches plus position embeddings and an extra class embedding go into a Transformer Encoder; an MLP head outputs the class (Bird, Ball, Car)

Figure: Dosovitskiy et al. (2021), Fig. 1 (left part).

A · Multi-modality model before LLMs

Architecture Overview: Taxonomy

  • Dual-encoder: separate encoders, shared embedding space (CLIP)

    → Trained with contrastive loss; no generation capability by itself

  • Encoder-decoder: encoder processes image, decoder generates text (Flamingo, CogVLM)

    → More parameter-efficient cross-modal interaction via cross-attention layers

  • Decoder-only: image tokens prepended to text tokens in single LLM (LLaVA, GPT-4V)

    → Simplest approach; leverages full LLM capability; most common in 2023–2024

Key design choice: freeze vs. fine-tune the vision encoder

→ Frozen ViT: faster alignment, less forgetting; trained ViT: more flexible

A · Multi-modality model before LLMs

Early Fusion vs. Late Fusion

Early Fusion

  • Merge inputs at the token level
  • Joint attention from layer 1
  • Higher cross-modal information flow
  • Expensive: O((Ntext + Nimg)²)
  • Best for fine-grained grounding
  • Example: Unified-IO, PaLI

Late Fusion

  • Encode each modality separately
  • Merge at final representation layer
  • Lower computational cost
  • Modular — swap encoders easily
  • Best for retrieval, classification
  • Example: CLIP, AudioCLIP

A · Multi-modality model before LLMs Recap: Lecture 3

CLIP: Learning More Aligned Represenations

CLIP Figure 1: (1) contrastive pre-training: a text encoder and an image encoder; the N by N matrix of products I_i·T_j, with the matching pairs on the diagonal; (2) create a dataset classifier from label text 'A photo of a {object}.'; (3) use for zero-shot prediction: the image of a dog matches 'A photo of a dog.'
Radford et al., Learning Transferable Visual Models From Natural Language Supervision

Method of alignment: Contrastive Learning

Data: 400M image-text pairs

Single image Encoder (ResNet, ViT)

Downside? Coarse-grained. Has to represent (somewhere) notion of objects, relationships, locations, etc.)

Figure: Radford et al. (2021), Fig. 1. CLIP was taught in Lecture 3.

A · Multi-modality model before LLMs

Open-Vocabulary Detection

OWL-ViT Figure 1: left, image-level contrastive pre-training of a text and a vision transformer encoder; right, transfer to open-vocabulary detection: query embeddings for 'giraffe', 'tree', 'car' are compared with object image embeddings, and an MLP head predicts boxes
Minderer et al., Simple Open-Vocabulary Object Detection with Vision Transformers

Figure: Minderer et al. (2022), Fig. 1.

A · Multi-modality model before LLMs

Open-Vocabulary Detection

OWLv2 Figure A2, first row: web images with pseudo-box annotations from their captions: a Citroën C3 interior, a map of the Athenian empire in 431 BC, a bus in Luxembourg city
Minderer et al., Scaling Open-Vocabulary Object Detection

Figure: Minderer et al. (2023), Fig. A2 (first row; images from Wikimedia Commons).

Part A · How LLMs learned to see

Vision-language model

1Multi-modality model before LLMs
2Vision-language model
3State of art

A · Vision-language model

VLMs: Considerations

The same diagram as before: three photos, a Visual Feature Space, a Language Model with a Features Space (Encoder) and a Decoder; the questions How/what should we train? Using what data? What tasks can we do? What should the interface be?

Cropped from the deck.

A · Vision-language model

Embedding space alignment: The connector must turn image features into tokens the LLM can read

Goal: map visual / audio features into the LLM's token embedding space

→ Spatial structure vs. token sequence — must bridge dimensionality and distribution

Visual Feature Space from the vision model what we discuss next Connector (illustrative) Language Model Token embeddingspace Decoder a group of men arefishing on a beacha brown dog runs after ablack dog on a shoregirl hits a ball andthe catcher looks on

The diagram is illustrative and added for this lecture.

A · Vision-language model

Embedding Space Alignment

1 Connector: what sits between the two models

Linear Projector (MLP)

patchesMLP (1–2 layers)tokens1 tokenper patch
Linear Projector (MLP): simple 1–2 layer MLP; used in LLaVA

→ Fast, effective for high-res; maps ViT features → LLM hidden dim

Q-Former (BLIP-2)

featuresqueriesfixednumber
Q-Former (BLIP-2): learnable query tokens attend to frozen image encoder

→ 32–64 fixed queries collapse variable-length features into fixed-size representation

Perceiver Resampler (Flamingo)

small imagelarge imagecross-attentionN latentssame N
Perceiver Resampler (Flamingo): cross-attention from N latents to image features

→ Key advantage: variable resolution images → fixed latent budget

2 LLM side: change the LLM or not

LLM-side adapters

x LLM weights (frozen) LoRA A LoRA B +output frozen: keepslanguage skillstrained: smalllow-rank path
LLM-side adapters: lightweight LoRA layers in LLM learn to interpret visual tokens

→ Reduces catastrophic forgetting of language knowledge during vision alignment

3 Training: which loss

Joint training objective

image text vt contrastive: pull v and t together captioning: from v, predict t
Joint training objective: align visual features with corresponding text descriptions

→ Image captioning loss, contrastive loss, or both during alignment pretraining

Hover over a box to read the original text. Boxes regrouped by role; headings 1 to 3 and the diagrams are added for this lecture.

A · Vision-language model

Example VLM: Flamingo

• Flamingo: Multimodal In-Context-Learning

EmergingProperty
Flamingo Figure 1, first row: Input Prompt: a chinchilla image, 'This is a chinchilla. They are mainly found in Chile.'; a shiba image, 'This is a shiba. They are very popular in Japan.'; a flamingo image, 'This is'; Completion: 'a flamingo. They are found in the Caribbean and South America.'
Flamingo Figure 1, fourth row: handwritten '2+1' image, '2+1=3'; '5+6' image, '5+6=11'; '3x6' image; Completion: '3x6=18'

Figure: Alayrac et al. (2022), Fig. 1 (rows 1 and 4).

A · Vision-language model

Flamingo VLM

  • Key component - cross attention that enables The interaction with image/video and text
Flamingo Figure 4: a GATED XATTN-DENSE layer sits before each frozen LM layer; inside it, cross attention with Q from the language input Y and K = V from the vision input X, then tanh gating, a feed-forward layer, tanh gating, then the frozen self-attention and FFW; beside it, the pseudo-code of gated_xattn_dense with alpha_xattn and alpha_dense initialised at 0

A Gated Cross attention mechanism is proposed to fuse images and text.

A · Vision-language model

Flamingo

Mix multi-modality context in embedding space

Flamingo Figure 3: two images each go through a frozen Vision Encoder and a Perceiver Resampler trained from scratch; their outputs feed GATED XATTN-DENSE layers placed before each frozen LM block; the processed text is '<image> This is a very cute dog.<image> This is', and the output text is 'a very serious cat.'
Language ModelPre-trained: 70B Chinchilla
Connection ModulePerceiver Resampler
Gated Cross-attention + Dense
Vision EncoderPre-trained: Nonrmalizer-Free ResNet (NFNet)

Figure: Alayrac et al. (2022), Fig. 3.

A · Vision-language model

Flamingo VLM also enables video understanding

Flamingo Figure 5: the Perceiver Resampler; video frames at t=0, 1, 2 go through the vision encoder, get time embeddings, and are flattened into X_f; attention with Q = learned latent queries X and K = V = [X_f, X]; then a feed-forward layer; repeated num_layers times; beside it, the pseudo-code of perceiver_resampler
fixed #pseudo code
  • Model structure - Supporting both images and videos

Maps a variable size grid of visual features from the Vision Encoder to a fixed number of output token (5 in the figure.)

Figure: Alayrac et al. (2022), Fig. 5. The deck labels “fixed #” (the learned latent queries) and “pseudo code” are under the figure.

A · Vision-language model

Flamingo VLM

  • Model structure - Supporting both images and videos
  • Using pre-trained ResNet to get visual features Xf
  • Compress the encode image into R tokens
  • Core of this module : Attention .
    • Query: the learned latent token X
    • Key=Value: the concatenation of Xf, and the learned latent token X
    • Better performance by concatenating keys and values obtained from latent
  • If the input is video
    • Xf will add time embeddings

The bullet list beside Alayrac et al. (2022), Fig. 5.

A · Vision-language model

Flamingo VLM

  • Data - Obtaining multimodal dataset to induce good generalist capabilities
Flamingo Figure 9: Image-Text Pairs dataset [N=1, T=1, H, W, C] with a flamingo picture and 'This is an image of a flamingo.'; Video-Text Pairs dataset [N=1, T>1, H, W, C] with 'A kid doing a kickflip.'; Multi-Modal Massive Web (M3W) dataset [N>1, T=1, H, W, C] with 'Welcome to my website!', a dog picture, 'This is a picture of my dog.', a cat picture, 'This is a picture of my cat.'
  • M3W: Scrapping 43 million webpages from the Internet
  • Training on a mixture of vision and language datasets
  • M3W(185M images+ 182G text)
  • ALIGN(1.8B images with alt-text)
  • LTIP (312M images/text)
  • VTP(27M short video/text)

Figure: Alayrac et al. (2022), Fig. 9. The four datasets (right) are the sub-points of “Training on a mixture”.

A · Vision-language model

Flamingo VLM

  • Result: Overview of the results of the Flamingo models
Flamingo arXiv v1 Figure 9: aggregated performance relative to SOTA against the number of shots (0, 4, 8, 16, 32); Flamingo 80B rises from about 79% to 97%, Flamingo 9B from about 73% to 85%, Flamingo 3B from about 70% to 80%
  • Larger model sizes and more few-shot examples lead to better performance

Figure: Alayrac et al. (2022), arXiv v1, Fig. 9.

A · Vision-language model

BLIP

Problem 1

A photo of birds flying over a lake at sunset
Earlier models

Understanding only: pick the best text for the photo.

Or generation only: write a caption.

→

BLIP

One model trained for both.

Problem 2

BLIP Figure 4: three web images with their web text T_w and a synthetic text T_s; green texts are accepted by the filter and red texts are rejected. Sunset photo: web text 'from bridge near my house' (rejected), synthetic 'a flock of birds flying over a lake at sunset' (accepted). Plant photo: web text 'in front of a house door in Reichenfels, Austria' (rejected), synthetic 'a potted plant sitting on top of a pile of rocks' (accepted). Castle photo: web text 'the current castle was built in 1180, replacing a 9th century wooden castle' (accepted), synthetic 'a large building with a lot of windows on it' (rejected).
Earlier models

Web text Tw is noisy.

→

BLIP

A filter removes the web texts that do not match the image (red).

A captioner writes new texts Ts, and the same filter checks them.

The model trains only on the pairs that pass (green).

A · Vision-language model

BLIP-2

• BLIP2

BLIP-2 Figure 1: a frozen image encoder feeds a Q-Former (querying transformer) with learned queries; stage 1, vision-and-language representation learning, bootstraps pre-trained image models; stage 2, vision-to-language generative learning, feeds a frozen large language model, which writes 'Love is like a sunset, it's hard to see it coming but when it does it's so beautiful.' for a sunset photo and the request 'Write a romantic message that goes along this photo.'
Language ModelPre-trained: FLAN-T5/OPT
Connection ModuleQ-Former: Lightweight Querying Transformer
Vision EncoderContrastive pre-trained: EVA/CLIP

Problem: Flamingo trains a large connection module and is costly. BLIP-2 freezes the image encoder and the language model, and trains only a small Q-Former.

Figure: Li et al. (2023), Fig. 1. The transition line is added from Li et al. (2023).

A · Vision-language model

Visual Question Answering Examples

BLIP-2 Table 2, zero-shot visual question answering: trainable and total parameters and scores on VQAv2 (val, test-dev), OK-VQA test and GQA test-dev; for example Frozen 40M trainable, 7.1B total, 29.6 on VQAv2 val; Flamingo80B 10.2B trainable, 80B total, 56.3 on VQAv2 test-dev, 50.6 on OK-VQA; BLIP-2 ViT-g FlanT5XXL 108M trainable, 12.1B total, 65.2 and 65.0 on VQAv2, 45.9 on OK-VQA, 44.7 on GQA
Li et al., BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

Table: Li et al. (2023), Table 2.

A · Vision-language model

GPT4-V Gap

Diagram 'Recap on Language Modeling: Large Language Models (LLM)': OpenAI GPT-2 to GPT-3 (in-context-learning, chain-of-thoughts) to ChatGPT/InstructGPT (adds instruction-following) to GPT-4 (adds multimodal input with image); in the multimodal space, Flamingo, BLIP2, GIT … lead to an orange box 'Gap? Instruction-Following → Alignment Research', then to Multimodal GPT-4

The diagram is cropped from that page.

Part A · How LLMs learned to see

State of art

1Multi-modality model before LLMs
2Vision-language model
3State of art

A · State of art

State of Art

A number of aspects improved over time:

  • Scale (of course)
  • Multi-state training
    • pre-training, multi-modal alignment, SFT/Instruction Tuning, RLHF, Post-Training/Reasoning
  • Diversity of data
  • Some architectural (hard to tell what matters, no ablations):
    • Multi-resolution
    • Long context
    • Temporal position embeddings
    • Synthetic data
  • Huge diversity of evaluation benchmarks across many tasks

A · State of art

Qwen-VL Multi-stage Training

Qwen-VL Figure 3, three training stages: Stage1 Pretraining on image-text pairs at low resolution, QwenLM frozen, ViT and CrossAttn with learnable query embeddings trained; Stage2 Multi-task Pretraining on multi-task and interleaved VL data at high resolution, all trained; Stage3 Supervised Finetuning on chat interleaved VL data, ViT frozen, CrossAttn and QwenLM trained
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

Figure: Bai et al. (2023), Fig. 3.

A · State of art

Qwen-VL: More modalities, more resolutions

Qwen2-VL Figure 2: native resolution input; a vision encoder turns each image or video into a different number of tokens (Picture 1, 1092 by 8204 pixels: 11427 tokens; Picture 2, 224 by 28: 8 tokens; Picture 3, 1260 by 700: 1125 tokens; Video 1, 644 by 336, 16 s: 2208 tokens), which go into the QwenLM decoder with the text tokens
Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

Figure: Wang et al. (2024), Fig. 2.

A · State of art

State of Art: Qwen3-VL joins a vision encoder to a Qwen3 LLM

The deck takes the figure from the Qwen3-VL README; the same figure is Bai et al. (2025), Fig. 1, and the crop is from that report.

A · State of art

Key Enhancements

  • Visual Agent: Operates PC/mobile GUIs—recognizes elements, understands functions, invokes tools, completes tasks.
  • Visual Coding Boost: Generates Draw.io/HTML/CSS/JS from images/videos.
  • Advanced Spatial Perception: Judges object positions, viewpoints, and occlusions; provides stronger 2D grounding and enables 3D grounding for spatial reasoning and embodied AI.
  • Long Context & Video Understanding: Native 256K context, expandable to 1M with inference-time extension (YaRN); handles books and hours-long video with full recall and second-level indexing.

Quoting https://github.com/QwenLM/Qwen3-VL?tab=readme-ov-file.

A · State of art

Key Enhancements

  • Enhanced Multimodal Reasoning: Excels in STEM/Math—causal analysis and logical, evidence-based answers.
  • Upgraded Visual Recognition: Broader, higher-quality pretraining is able to “recognize everything”—celebrities, anime, products, landmarks, flora/fauna, etc.
  • Expanded OCR: Supports 39 languages (up from 10), with over 70% accuracy on 32 of them; robust in low light, blur, and tilt; better with rare/ancient characters and jargon; improved long-document structure parsing.
  • Text Understanding on par with pure LLMs: Seamless text–vision fusion for lossless, unified comprehension.

Quoting https://github.com/QwenLM/Qwen3-VL?tab=readme-ov-file.

A · State of art

Summary

  • Simplify design, scale data and training
  • From captioning → reasoning + action
  • From static images → long-context multimodal streams
  • From human data → synthetic, model-generated
  • From SFT-only → full RL-style optimization pipelines
  • Some unified auto-regressive + diffusion but early

The page has no title; “Summary” is ours.

Part A — history: how LLMs learned to see

LLMs learned to see when the image became vectors that the LLM reads like words.

Each step solved a problem of the one before it. Part B builds the simplest form of this, LLaVA, step by step in code.

BREAK
10 minutesBack at 18:55

Coming up in Part B: build and train a vision-language model, step by step, with code.

Part B

Make VLM chat

LLaVA step by step, and the same steps in code with nanoVLM.

Reference slides: Georgia Tech CS 8803-VLM, Lecture 6, “LLaVA and Qwen3-VL”, Y. Wang, E. Anand, I. Narang, Fall 2026. Code: nanoVLM, Hugging Face.

B · Problem & foundations RECAP · PART A

Recognizing objects

The methods of Part A take an image, and sometimes a short text. They return a label, a box, or a short text.

Input

A man in a yellow shirt irons clothes on an ironing board fixed to the back of a yellow taxi SUV in a city street

An image, and sometimes a short text.

Output, by method (example outputs for this image)

MethodText inputOutput
Classify (CLIP)“a taxi”, “a bus”, “a bicycle”best label: “a taxi”
Detect (OWL-ViT)“person”, “ironing board”one box for each
Caption (BLIP)none“a man ironing on the back of a taxi”
Short answer (BLIP-2)“What color is the taxi?”“yellow”

Each method names what is in the image. None of them answers “what is unusual about this image?”.

Recap of Part A. The example outputs are illustrative.

B · Problem & foundations 01 / THE RESEARCH PROBLEM

Recognizing objects is not enough for chat (instruction following)

A man in a yellow shirt irons clothes on an ironing board fixed to the back of a yellow taxi SUV in a city street

THE SAME IMAGE, A DIFFERENT TARGET

“Describe the scene” and “What is unusual?” require different answers from the same pixels.

Question: “What is unusual about this image?”

  1. 1

    Recognize

    The scene contains a person, clothing, an ironing board, and a vehicle.

  2. 2

    Relate

    The activity is ironing; the setup is on a vehicle rather than in an ordinary indoor workspace.

  3. 3

    Answer the instruction

    Explain that unexpected activity–setting combination. Merely naming the objects misses the request.

  • LLaVA: teach this behavior with generated instructions.
  • Qwen3-VL: extend the scope to detail, long context, and actions.

Visual instruction following = relevant visual evidence + the requested response behavior—not an object list.

Part B · Make VLM chat

LLaVA: A simple instruction tuning method for VLM.

1Problem & foundations
2LLaVA: A simple instruction tuning method for VLM.
3A small demo: nanoVLM

B · LLaVA: A simple instruction tuning method for VLM. 03 / WHY CAPTIONS ARE NOT ENOUGH

Can we repurpose caption generation for question answering?

Rephrasing “describe this image” does not automatically create supervision for localization, comprehensive description, or reasoning.

Cheap caption expansion

Human: Describe the image.

Assistant: People try to fit their luggage in an SUV.

COCO photo with object boxes: a black SUV with its back door open in an underground parking area; people and pieces of luggage around it

Three instructions, three different targets

“Where is the vehicle parked?”
Select a specific fact: the underground parking area.
“Describe the objects and their positions.”
Cover people, luggage, the SUV, and their spatial relations.
“What challenges do the people face?”
Explain packing constraints using the scene and practical knowledge.

Captions are useful for initial alignment; richer instruction–answer pairs teach the assistant to change its response with the task.

The deck cites Liu et al., §3, pp. 2–4; Table 14, p. 23. Image with boxes: Liu et al. (2023), Tables 1 and 14 (a COCO image).

B · LLaVA: A simple instruction tuning method for VLM. 02 / THE LLAVA IDEA

LLaVA: a simple model for instruction following

Large Language and Vision Assistant: reuse pretrained capabilities, then teach the visual interface and the interaction style.

1. Generate instruction following data for training models

Captions + boxes“People are loading luggage into an SUV.”

→

GPT-4text onlyteacher

→

Instructions“Where is the vehicle parked?”

2. A lightweight visual model connected to LLM

CLIP ViT modelsees the image

→

Linear layerone layerstudent

→

LLM (Vicuna)writes the answer
  • The connector makes visual evidence accessible.
  • The instruction dataset teaches what a useful response should do.

B · LLaVA: A simple instruction tuning method for VLM. 02 / THE LLAVA IDEA

158K instruction samples, approximately 80K unique images

ConversationFocused answers and multi-turn follow-up.

58K

Detailed descriptionBroad coverage of entities and relationships.

23K

Complex reasoningExplanations that connect evidence to implications.

77K

B · LLaVA: A simple instruction tuning method for VLM. 04 / HOW THE TEXT-ONLY TEACHER SEES

Teacher model (GPT-4) generate instructions

LLaVA generates visual instruction data without giving the GPT-4 teacher an image input.

The image being described

The COCO photo with colored object boxes around the SUV, the people and the luggage

An offline supervision pipeline

Caption context. People are loading luggage into an SUV in an underground parking area.

person: [0.681, 0.242, 0.774, 0.694]

Normalized box coordinates encode spatial location.

GPT-4 teacher

Annotations + a few seed examples

Generated supervision

Question + target response

LLaVA training

Original image

+

Generated question

→

LLaVApredict

→

Generated answer

The deck cites Liu et al., §3; Tables 13–14, pp. 22–23. Image with boxes: Liu et al. (2023), Tables 1 and 14.

B · LLaVA: A simple instruction tuning method for VLM. 08 / THE VISUAL INTERFACE

LLaVA architecture

Trace the two input paths

LLaVA Figure 1: the image X_v goes through the vision encoder to Z_v and the projection W to H_v; the language instruction X_q becomes H_q; the language model f_phi reads both and produces the language response X_a
\[ Z_v = g(X_v), \qquad H_v = W Z_v \] \[ Z_v \in \mathbb{R}^{d_v \times N_v}, \quad W \in \mathbb{R}^{d_L \times d_v}, \quad H_v \in \mathbb{R}^{d_L \times N_v} \]

What each component does

CLIP ViT-L/14 (frozen). Encode the image into Nᵥ visual vectors. These are learned features, not necessarily one object per vector.

Projection W. Apply the same learned map to every visual vector, changing the feature width to Vicuna’s embedding dimension.

LLM/Vicuna decoder (frozen). Attend to visual and text embeddings together, then predict the answer one token at a time.

The linear projection aligns feature dimensions. It does not produce a caption.

The deck cites Liu et al., §4.1, Figure 1, Equation (1), p. 4. Figure: Liu et al. (2023), Fig. 1. CLIP (Radford et al., 2021): Lecture 3.

B · LLaVA: A simple instruction tuning method for VLM. 09 / TRAINING STAGE 1

Align the visual interface while both backbones stay frozen

Learn W using image–caption pairs before asking the language model to adapt to complex multimodal interactions.

Only the projection is trainable

CLIP

Frozen

W

Train

Vicuna

Frozen

\[ \min_{W}\; -\sum_{t} \log p_{W,\phi_0}\!\left(c_t \mid X_v, q, c_{\lt t}\right) \]

c = original caption; q = a brief-description request.
φ₀ stays fixed; gradients still pass through the decoder to W.

The deck cites Liu et al., §4.2; §5; Appendices C and E, Figure 7 (Liu et al., 2023).

B · LLaVA: A simple instruction tuning method for VLM. 10 / TRAINING STAGE 2

Adapt W and Vicuna—and supervise the assistant’s answers

“End-to-end” in this paper does not mean every component updates: the CLIP vision encoder remains frozen.

Two task-specific fine-tuning branches

Trainable parameters. Update θ = {W, φ}; retain the frozen CLIP encoder.

Multimodal chat

158K generated examples; 3 epochs.
Learn conversation, detailed description, and reasoning.

ScienceQA

Separate task-specific training.
Generate reasoning, then a multiple-choice answer.

One sequence, selective prediction targets

SCHEMATIC MULTI-TURN SERIALIZATION

Image + user Q₁

Assistant A₁

User Q₂

Assistant A₂

\[ \mathcal{L}(\theta) = -\sum_{i \in \mathcal{A}} \log p_\theta\!\left(x_i \mid X_v, x_{\lt i}\right) \]

A marks assistant-target positions (and stop behavior).
Past questions and answers remain conditioning context.

Stage 1 teaches the interface; Stage 2 changes the response policy. Questions provide context, while assistant answers supply the loss.

The deck cites Liu et al., §4.2, Table 2; §5; §5.2 (Liu et al., 2023).

B · LLaVA: A simple instruction tuning method for VLM.

How to evaluate LLaVA?

What we ask

LLaVA-Bench (COCO). 30 images, 90 questions: conversation, detailed description and complex reasoning.

LLaVA-Bench (In-the-Wild). 24 images, 60 questions: memes, paintings, sketches, indoor and outdoor scenes.

ScienceQA. Multiple-choice questions. The score is accuracy.

How an answer gets a score

Image + question

→

LLaVA

→

Answer
Text description of the image + question

→

Text-only GPT-4

→

Reference answer
GPT-4 judgerates both answers, 1 to 10

→

Score = 100 × LLaVA ÷ reference
  • The score compares the quality of the answer with the reference. It is not the share of correct answers.
  • In-the-Wild score: LLaVA 67.3, BLIP-2 38.1, OpenFlamingo 19.1.

Source: Liu et al. (2023), Sec. 5 and Table 5.

Part B · Make VLM chat

A small demo: nanoVLM

1Problem & foundations
2LLaVA: A simple instruction tuning method for VLM.
3A small demo: nanoVLM

B · A small demo: nanoVLM STEP 1 · THE MAP

nanoVLM puts LLaVA’s three parts in four files

LLaVA partnanoVLM fileLines
Vision encodermodels/vision_transformer.py251
Projection Wmodels/modality_projector.py45
Language modelmodels/language_model.py681
All three, joinedmodels/vision_language_model.py310

New, compared with nanoGPT

An image encoder: a ViT (Lecture 3).

A projector: a pixel shuffle, then one linear layer.

The language model reads vectors, not token ids.

It uses RoPE, RMSNorm, a gated MLP and GQA: the parts that Lecture 1 named.

nanoVLM (Wiedmann et al., 2025), commit 4e0c096 (27 Oct 2025), Apache-2.0; line counts from wc -l at that commit. nanoGPT: Karpathy (2022). Map of parts: Liu et al. (2023), Fig. 1.

B · A small demo: nanoVLM STEP 1 · THE CHECKPOINT

We use one checkpoint: nanoVLM-230M-8k

Image encoder: SigLIP 2 Lecture 3
B/16, 512 px, 12 blocks, width 768
Projector
Pixel shuffle, factor 4: 64 tokens per tile
Language model: SmolLM2-135M
30 blocks, width 576, 9 heads, 3 key/value heads

Parameters, from the checkpoint file

Image encoder86,433,024Projector7,077,888Language model134,553,024Total 228,063,936

The projector is 3.1% of all parameters (computed).

Why this one: generate.py loads it by default, and the current code reads it.

Checkpoint lusxvr/nanoVLM-230M-8k (MIT licence): config.json and the tensor shapes in model.safetensors, revision a76fa2d, read 10 Oct 2026. Code: nanoVLM (Wiedmann et al., 2025), commit 4e0c096. Tschannen et al. (2025); Allal et al. (2025).

B · A small demo: nanoVLM STEP 2 · MATH

One photo gives nanoVLM 1,088 image tokens

\[ N = \left(\frac{H}{p}\right)^{2} \]

H = image side, p = patch side.

LLaVA (2023), 224 px: \((224/14)^2 = 16^2 = 256\)

LLaVA-1.5: \((336/14)^2 = 24^2 = 576\)

nanoVLM, one tile: \((512/16)^2 = 32^2 = 1{,}024\)

Pixel shuffle, factor 4: \(1{,}024 / 4^2 = 64\) tokens per tile

Square photo, 16 tiles + 1 global view: \(17 \times 64 = 1{,}088\)

ALL RESULTS ON THIS SLIDE ARE COMPUTED

r1c1r1c2r1c3r1c4r2c1r2c2r2c3r2c4r3c1r3c2r3c3r3c4r4c1r4c2r4c3r4c42,048 × 2,048 pixels16 tiles of 512 × 512global view(whole photo,512 × 512)+17 × 64= 1,088image tokens

LLaVA (2023) at 224 px and LLaVA-1.5 with CLIP-ViT-L-336px, patch 14 (Liu et al., 2024, Table 2). nanoVLM (Wiedmann et al., 2025), commit 4e0c096: data/custom_transforms.py resizes the long side to max_img_size = 2,048 and cuts 512 × 512 tiles; data/processors.py adds the global view. The 17 tile-marker tokens are not counted here.

B · A small demo: nanoVLM STEP 3 · THE PROJECTOR

Pixel shuffle joins each 4 × 4 block of patches into one vector

32 × 32 vectors, width 768one block: 4 × 4 = 16 vectors8 × 8 vectors, width 12,28816 × 768 = 12,288 numbers each

The encoder gives a 32 × 32 grid of vectors.

Pixel shuffle cuts the grid into 4 × 4 blocks. It puts the 16 vectors of a block end to end.

1,024 vectors of width 768
→ 64 vectors of width 12,288

Neighbor patches stay together. The LLM gets 16 times fewer image tokens.

Numbers for lusxvr/nanoVLM-230M-8k (mp_pixel_shuffle_factor 4). nanoVLM (Wiedmann et al., 2025), models/modality_projector.py, commit 4e0c096. Pixel shuffle in small VLMs: Marafioti et al. (2025), SmolVLM, Fig. 4.

B · A small demo: nanoVLM STEP 3 · THE PROJECTOR

The projector is one linear layer with 7,077,888 weights

    self.input_dim = cfg.vit_hidden_dim * (cfg.mp_pixel_shuffle_factor**2)    self.output_dim = cfg.lm_hidden_dim⋮    self.proj = nn.Linear(self.input_dim, self.output_dim, bias=False)⋮def forward(self, x):    x = self.pixel_shuffle(x)    x = self.proj(x)

input_dim \(= 768 \times 4^2 = 12{,}288\)

output_dim \(= 576\), no bias

weights \(= 12{,}288 \times 576 = 7{,}077{,}888\)

COMPUTED

nanoVLM (Wiedmann et al., 2025), models/modality_projector.py (45 lines), commit 4e0c096, Apache-2.0 (common indentation removed). lusxvr/nanoVLM-230M-8k stores MP.proj.weight with shape [576, 12288].

B · A small demo: nanoVLM STEP 4 · IMAGE INTO THE SEQUENCE

The prompt holds 64 placeholder tokens for each tile

if hasattr(tokenizer, "global_image_token"):    image_string += tokenizer.global_image_token    image_string += tokenizer.image_token * mp_image_token_length    if n_h == 1 and n_w == 1:  # If there is only one patch, treat it as the global image        continuefor i in range(n_h):    for j in range(n_w):        image_string += getattr(tokenizer, f'r{i+1}c{j+1}')        image_string += tokenizer.image_token * mp_image_token_length

<|im_start|>user⏎ <|global_image|> <|image|>×64 <row_1_col_1> <|image|>×64 … <row_4_col_4> <|image|>×64 What is this?<|im_end|>⏎ <|im_start|>assistant⏎

Each placeholder is one token. The next step puts the image vectors at these places.

nanoVLM (Wiedmann et al., 2025), data/processors.py, lines 33–41, commit 4e0c096, Apache-2.0 (common indentation removed). Text line: our checkpoint’s chat template, a square photo; ⏎ = line break.

B · A small demo: nanoVLM STEP 4 · IMAGE INTO THE SEQUENCE

nanoVLM writes the image vectors over the placeholders

token_embd = self.decoder.token_embedding(input_ids) # [B, T_sequence, D_lm]if images_tensor is not None:    image_embd = self.vision_encoder(images_tensor)    image_embd = self.MP(image_embd)  # [num_images, mp_image_token_length, D_lm]    token_embd = self._replace_img_tokens_with_embd(input_ids, token_embd, image_embd)⋮mask = (input_ids == self.tokenizer.image_token_id)updated_token_embd[mask] = image_embd.view(-1, image_embd.size(-1)).to(updated_token_embd.dtype) # torch flattens before assigning

The LLM reads vectors, not token ids (Lecture 1). Every token gets its embedding. Then the 64 vectors per tile go to the <|image|> places, in tile order.

nanoVLM (Wiedmann et al., 2025), models/vision_language_model.py, lines 64–69 (forward) and 46–47 (_replace_img_tokens_with_embd), commit 4e0c096, Apache-2.0 (common indentation removed).

B · A small demo: nanoVLM STEP 5 · THE LOSS

The loss counts only the tokens of the answer

\[ \mathcal{L}(\theta) = -\,\frac{\sum_t m_t \,\log p_\theta\!\left(x_{t+1} \mid x_{\le t}\right)}{\sum_t m_t}, \qquad m_t = \begin{cases} 1 & x_{t+1} \text{ is in an answer} \\ 0 & \text{else: label } {-100} \end{cases} \]
inputlabelcountedimage + question−100m = 0<|im_start|>−100m = 0assistant−100m = 0⏎Am = 1Acatm = 1cat<|im_end|>m = 1<|im_end|>⏎m = 1⏎−100m = 0

The image tokens are in \(x_{\le t}\): the model reads them, but it does not predict them. The answer includes <|im_end|>, so the model learns when to stop.

Our notation for the \(\mathcal{A}\) in LLaVA’s stage-2 loss (Train it, stage 2) (Wang, Anand & Narang, 2026); see Lecture 1, cross entropy and Lecture 2, SFT loss · mask. Tokens are schematic. Mask and labels: nanoVLM (Wiedmann et al., 2025), data/datasets.py, commit 4e0c096.

B · A small demo: nanoVLM STEP 5 · THE LOSS

Label −100 removes a token from the loss

data/datasets.py    if msg["role"] == "assistant":        start = cursor + self.prefix_len        end   = cursor + seg_len        mask[start:end] = [1] * (end - start)  # attend to these tokens⋮labels = input_ids.clone().masked_fill(~mask, -100)labels = labels.roll(-1) # Shift labels for causal LMlabels[-1] = -100 # Last token has no targetmodels/vision_language_model.pyloss = F.cross_entropy(logits.reshape(-1, logits.size(-1)), targets.reshape(-1), ignore_index=-100)

The mask is 1 on each answer. The labels shift by one place, so each place predicts the next token. ignore_index=-100 skips the other places, and the mean is over the answer tokens only.

nanoVLM (Wiedmann et al., 2025), data/datasets.py, lines 97–100 and 146–148; models/vision_language_model.py, line 78; commit 4e0c096, Apache-2.0 (common indentation removed).

B · A small demo: nanoVLM NANOVLM

nanoVLM trains all three parts, at three learning rates

models/config.pylr_mp: float = 0.00512lr_vision_backbone: float = 5e-5 #0.0005 #lr_language_backbone: float = 5e-5 #0train.pyif train_cfg.lr_mp > 0:    param_groups.append({'params': list(model.MP.parameters()), 'lr': train_cfg.lr_mp})

The new projector starts from random weights. Its learning rate is 102.4 times larger (computed). A learning rate of 0 freezes a part (lines 312–314).

nanoVLM (Wiedmann et al., 2025), models/config.py lines 59–61, train.py lines 310–311 (all three parts: lines 309–326), commit 4e0c096, Apache-2.0 (common indentation removed). LLaVA: Liu et al. (2023), §4.2.

B · A small demo: nanoVLM GENERATE

The LLM reads the image once; the KV cache keeps it

for _ in range(max_new_tokens):⋮        next_token_id = torch.multinomial(probs, num_samples=1)⋮    next_token_embed = self.decoder.token_embedding(next_token_id) # [B, 1, D_lm]⋮    decode_step_output, kv_cache_list = self.decoder(        next_token_embed,        attention_mask=attention_mask,        kv_cache=kv_cache_list,        start_pos=current_token_start_pos    )

Prefill. One pass reads the image tokens and the question. It fills the KV cache (lines 98–103).

Decode. Each new token reads the cache and adds only its own key and value.

nanoVLM (Wiedmann et al., 2025), models/vision_language_model.py, lines 116–143 of generate (sampling: top-k 50, top-p 0.9, temperature 0.5, line 83), commit 4e0c096, Apache-2.0 (common indentation removed).

B · A small demo: nanoVLM DEMO

SmolVLM-256M runs in your browser

SmolVLM-256M has the same kind of parts: a SigLIP B/16 encoder (ours: SigLIP 2), a pixel shuffle and SmolLM2-135M.

Try: “Describe the image.”, then “What is unusual about this image?” Then ask about an object that is not in the photo.

Demo: Hugging Face Space HuggingFaceTB/SmolVLM-256M-Instruct-WebGPU (Apache-2.0). SmolVLM-256M: Marafioti et al. (2025). SigLIP: Lecture 3. nanoVLM: Wiedmann et al. (2025).

Part B — deep dive: build and train a VLM

A VLM is an image encoder, a small projector and an LLM, trained on image–text data to answer questions.

LLaVA showed the recipe in 2023. nanoVLM shows the same recipe in four PyTorch files, about 1,300 lines.

BREAK
10 minutesBack at 19:55

Coming up in Part C: what changed since 2023, and where vision-language models are today.

Part C

Advanced topics and today's state of the art

Qwen3-VL, then one model for all modalities, agents and robots. Each as why, how and results.

Reference slides: Georgia Tech CS 8803-VLM, Fall 2026: Lecture 6, “LLaVA and Qwen3-VL” (Wang, Anand, Narang), and Lecture 1 (Kira).

C · Qwen3-VL 01 / The research problem

Qwen3-VL: from pixels to reasoning and action

A strong language model is the starting point—not a capability to sacrifice when adding vision.

How do we add rich visual understanding, long context, and useful actions—while preserving language ability?

Model family

Dense: 2B / 4B / 8B / 32B
MoE: 30B-A3B / 235B-A22B

Visual inputs

Images, video, documents

→

Vision encoder

Dynamic-resolution features

→

Merger + DeepStack

Compress and transfer

→

Qwen3 decoder

Reason, ground, use tools

Top half. Deck source: Report §§1–2, pp. 2–4 (Bai et al., 2025).

C · Qwen3-VL 01 / The research problem

Qwen3-VL: from pixels to reasoning and action

Three architectural upgrades

DeepStack

Inject multiple ViT depths.

Interleaved MRoPE

Balance position frequencies.

Text timestamps

Make elapsed time explicit.

The system goal: strong perception + reliable information transfer + 256K multimodal context + effective training.

Bottom half. Deck source: Report §§1–2, pp. 2–4 (Bai et al., 2025).

C · Qwen3-VL 05 / DeepStack: the mechanism

Three feature sets enter the first three LLM layers

The important dimension is depth: add information to existing visual positions instead of appending more positions.

Vision encoder depthEarly LLM layers

ViT feature level 3

Spatial feature grid

→ M3 →

LLM layer 3

Add projected features

▲▲

ViT feature level 2

Spatial feature grid

→ M2 →

LLM layer 2

Add projected features

▲▲

ViT feature level 1

Spatial feature grid

→ M1 →

LLM layer 1

Add projected features

Left panel; level diagram rebuilt from that page, words unchanged. Deck source: Report §2.2 and Fig. 1, pp. 3–4 (Bai et al., 2025).

C · Qwen3-VL 05 / DeepStack: the mechanism

Three feature sets enter the first three LLM layers

Aligned residual injection

\(D_k = M_k\big(F^{(a_k)}\big) \in \mathbb{R}^{N_v \times d_L}, \qquad k = 1,2,3\)

\(H^{(k)} = T_k\big(H^{(k-1)}\big) + S_{\mathcal{V}}(D_k)\)

Meaning. \(M_k\) maps a selected ViT feature set to LLM width; \(S_{\mathcal{V}}\) places it at the corresponding visual positions.

Cost. Extra feature extraction, mergers, and additions—but no additional context positions from stacking.

Keep the distinction: multi-level features ≠ more image scales; unchanged context length ≠ zero extra computation.

Right panel and closing line. Deck source: Report §2.2 and Fig. 1, pp. 3–4 (Bai et al., 2025) • equations are schematic.

C · Qwen3-VL 18 / Read the results by capability

Thinking helps some tasks—not every task

Selected within-family comparisons for Qwen3-VL-235B-A22B, as reported in Table 2.

BenchmarkInstructThinkingWhat the row tests
MathVision66.574.6Visual mathematical reasoning
MUIRBENCH73.080.1Multi-image understanding
DocVQA test97.196.5Document question answering
OCRBench920875Text recognition; own score scale
Video-MME, no subtitles79.279.0General video understanding

Table; numbers checked against the report's Table 2, p. 15. Deck source: Report Table 2, p. 15; §§5.2–5.4, pp. 14–17 (Bai et al., 2025).

C · Qwen3-VL 18 / Read the results by capability

Thinking helps some tasks—not every task

Reasoning ≠ extraction

More reasoning does not guarantee better visual reading.

System outcomes, not component ablations

Compare within each benchmark; score scales differ.

A 256K context, a larger model, or a Thinking label is a capability setting—not a guarantee of correctness on every input.

The two cards and closing line. Deck source: Report Table 2, p. 15; §§5.2–5.4, pp. 14–17 (Bai et al., 2025).

C · Qwen3-VL 13 / Multimodal reasoning data

Train the model to see correctly—and force the image to matter

A visual-math error can arise from misreading the diagram, faulty reasoning, or both.

Develop perception and reasoning

Perception supervision. Programmatically render diagrams; generate 1M point-grounding samples, 2M perception QA pairs, and 6M rich diagram captions.

Reasoning supervision. Curate over 60M educational exercises and over 12M image-paired long-reasoning samples; validate answers and filter trajectories.

Perception: “Which lines intersect?”
Reasoning: “What follows from that intersection?”

Left panel. Deck source: Report §3.2.8, pp. 8–9 (Bai et al., 2025).

C · Qwen3-VL 13 / Multimodal reasoning data

Train the model to see correctly—and force the image to matter

Multimodal-necessity filtering

Give only the text to Qwen3-30B-nothink

Test visual-mathematics training questions.

↓

Can it solve the problem correctly?

YES → discard

Text alone is enough for this model.

NO → retain for curation

A candidate requiring visual evidence.

This is an empirical filter—not proof of nonzero conditional mutual information.

High visual-benchmark accuracy is less meaningful when a model can ignore the visual input and exploit textual shortcuts.

Right panel and closing line. Deck source: Report §3.2.8, pp. 8–9; §4.2.2, p. 11 (Bai et al., 2025).

C · Qwen3-VL 16 / Reinforcement learning

Train on verifiable outcomes near the model’s learning frontier

Qwen3-VL combines reasoning RL with general RL for instruction following, alignment, and behavioral correction.

Reasoning RL: objective feedback

\(\displaystyle \max_{\theta}\; \mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)}\big[R(x, y)\big]\)

Tasks. Mathematics, code, logical reasoning, visual grounding, and puzzles with rule- or executor-based verification.

Training method. The report uses SAPO, a smooth adaptive policy-gradient method; it does not detail its update equations here.

General RL adds instruction-following, preference, language-consistency, and targeted error correction, using rules and model judges.

Left panel. Deck source: Report §4.4, pp. 11–12 (Bai et al., 2025) • objective and sampling probability are explanatory.

C · Qwen3-VL 16 / Reinforcement learning

Train on verifiable outcomes near the model’s learning frontier

Curate problems that can teach

Remove initially unproductive queries. Sample 16 responses with a strong checkpoint; discard queries with no correct responses.

Then remove easy queries per model. Sample 16 responses and filter pass rates above 90%. The curated pool is about 30K queries.

\(\Pr(\text{no success in 16 trials}) = (1-p)^{16}\)

Independent-trial illustration: zero observed successes does not mean a task is impossible.

Reward design and data selection determine the learning signal. A verifier or judge is not automatically immune to reward hacking.

Right panel and closing line. Deck source: Report §4.4, pp. 11–12 (Bai et al., 2025) • objective and sampling probability are explanatory.

C · Qwen3-VL 19 / Evidence and limitations

Separate component ablations from whole-system claims

The report offers concrete controlled evidence for DeepStack—and narrower evidence for extreme context length.

DeepStack: matched pretraining comparison

Internal 15B-A2B model; 200B pretraining tokens; validation evaluation with no post-training.

MetricBaselineDeepStack
Reported average74.776.0
InfoVQA71.974.2
DocVQA89.591.1
MMMU52.954.1

Left panel; numbers checked against the report's Table 12, p. 24. Deck source: Report Table 12, p. 24 (Bai et al., 2025).

C · Qwen3-VL 19 / Evidence and limitations

Separate component ablations from whole-system claims

Long context: retrieval ≠ general reasoning

Video needle-in-a-haystack

100%

Up to 256K tokens / 30 min

99.5%

≈1M tokens with YaRN extension

Scope. Locate an inserted salient frame and answer its question. This does not establish equally strong long-video reasoning.

Comparison caveat. Video frame caps differed: Qwen 2,048; Gemini 512; GPT-5 256; Claude 100. The authors flag imperfect fairness.

1M tokens is an extrapolation experiment—not the native 256K training window. Headline benchmark gains do not identify a single cause.

Right panel and closing line. Deck source: Report Fig. 3 and §5.12.3, p. 25; §5.9, p. 20 (Bai et al., 2025).

Part C · Advanced topics and today's state of the art

Part 1: Building a VLM

1Qwen3-VL
2Part 1: Building a VLM
3Part 3: Making It Act
4Part 4: Acting in the World
5Today's state of the art, and safety

C · Part 1: Building a VLM One Model That Reads and DrawsWeek 5

Universal token-in and token-out multi-modal models

Chameleon Figure 1: (a) mixed-modal pre-training: a text prompt and an image prompt (through an image tokenizer) become one token sequence for a mixed-modal auto-regressive LM; (b) mixed-modal generation: the same LM outputs text tokens and image tokens, which an image de-tokenizer turns into an image of banana bread

Chameleon: one mixed-modal autoregressive model, early fusion, image and text tokens in a single vocabulary.

BAGEL Figure 2: an understanding expert (next-token prediction for the language response) and a generation expert (velocity prediction for image, multi-image or video output) each have their own QKV and FFN, and share one multi-modal self-attention; inputs come from a text tokenizer, an understanding encoder and a generation encoder

BAGEL: separate understanding and generation experts sharing one attention stack — a different bet on the same goal.

Figures: Chameleon Team (2024), Fig. 1; Deng et al. (2025), Fig. 2.

C · Part 1: Building a VLM Other Modalities

Other Modalities

UniTouch Figure 1: zero-shot touch understanding (a touch image matched to text prompts such as denim, cotton, nylon by cosine similarity), cross-modal retrieval between touch, image, audio and text, Touch-LLM answering questions about materials and hardness, zero-shot image synthesis with touch, and X-to-touch generation

The page has only this figure and its source line (deck: "Yang et al., Binding Touch to Everything: Learning Unified Multimodal Tactile Representations, CVPR 2024"); its page title "Other Modalities" is used as the slide title. Figure: Yang et al. (2024), Fig. 1.

C · Part 1: Building a VLM Result

Humans prefer Chameleon-34B to Gemini-Pro and GPT-4V more often than the reverse

Chameleon-34B againstChameleon winsTieChameleon loses
Gemini-Pro (text answer)53.5%31.2%15.3%
GPT-4V (text answer)46.0%31.4%22.6%
Gemini-Pro with images added (Gemini+)41.5%34.5%24.0%
GPT-4V with images added (GPT-4V+)35.8%31.6%32.6%

Test: each prompt asks for an answer with text and images. Three human annotators see two answers in random order. They pick the better answer or "about the same".

Closest case: against GPT-4V+ the two models are almost equal (35.8% against 32.6%). Gemini and GPT-4V write text only, so the "+" versions have images added by the authors.

Source: Chameleon Team (2024), §4.1 (1,048 prompts) and §4.2.2, Figure 9b, pp. 11–13; read 2 Nov 2026.

Part C · Advanced topics and today's state of the art

Part 3: Making It Act

1Qwen3-VL
2Part 1: Building a VLM
3Part 3: Making It Act
4Part 4: Acting in the World
5Today's state of the art, and safety

C · Part 3: Making It Act CS 8803-VLM, part 3

Making It Act

Tools, browsers, and the whole desktop — Weeks 9–11.

1Week 1Pixels & wordsPatches and sub-words become one token stream.Transformer · ViT
2Week 2Aligned representationsVision and language share an embedding space.CLIP → SigLIP 2 · DINOv3
3Weeks 3–5Vision-language modelsA language model that can see, and generate.Flamingo · LLaVA → Qwen3-VL
4Weeks 6–9Reasoning & reliabilityThink before answering; know when you are wrong.MM-CoT · RLVR · POPE
5Weeks 9–11AgentsPerceive, plan, and take actions in software.ViperGPT · Mind2Web · OSWorld
6Weeks 11–13Action & world modelsAct in the physical world; imagine what follows.RT-2 · OpenVLA · DreamZero

The same roadmap as p. 24, stage 5 highlighted. SigLIP 2: Lecture 3.

C · Part 3: Making It Act Computer Use: The Whole DesktopWeek 10 · seminal

Computer Use

OSWorld Figure 1: screenshots of two tasks (update a bookkeeping sheet from receipts; fix a snake game's code) and the OSWorld environment: a task initial-state setup config, an agent such as GPT-4V that observes screenshots and the accessibility tree and acts with mouse and keyboard, virtual machines with real operating systems and apps, and execution-based evaluation of the final state
  • A real operating system with real apps: file manager, browser, office suite, terminal
  • Success is verified by running a script against the final machine state
  • Humans clear most tasks; early agents cleared a small fraction.
  • Frontier: better screenshot grounding, agent memory, and recovery after a wrong action.

Figure: Xie et al. (2024), Fig. 1.

C · Part 3: Making It Act Result

In 2024, people finished 72% of OSWorld tasks and the best model 12%

Who does the taskInput to the modelTasks finished
Humans (not familiar with the software)–72.36%
Best model: GPT-4Accessibility tree (a text list of the screen elements)12.24%
GPT-4VScreenshot only5.26%
Claude-3-OpusScreenshot only2.42%

The test: 369 tasks on real computers, checked by a script.

The gap: 60 points. The paper names GUI grounding (linking a plan to the right place on the screen) as a main cause.

Source: Xie et al. (2024), §4.2, Table 5, p. 10 (human row and model rows); abstract (369 tasks); read 2 Nov 2026. 2024 baseline models.

Part C · Advanced topics and today's state of the art

Part 4: Acting in the World

1Qwen3-VL
2Part 1: Building a VLM
3Part 3: Making It Act
4Part 4: Acting in the World
5Today's state of the art, and safety

C · Part 4: Acting in the World CS 8803-VLM, part 4

Acting in the World

Vision-language-action models, real-time control, and world models — Weeks 11–13.

1Week 1Pixels & wordsPatches and sub-words become one token stream.Transformer · ViT
2Week 2Aligned representationsVision and language share an embedding space.CLIP → SigLIP 2 · DINOv3
3Weeks 3–5Vision-language modelsA language model that can see, and generate.Flamingo · LLaVA → Qwen3-VL
4Weeks 6–9Reasoning & reliabilityThink before answering; know when you are wrong.MM-CoT · RLVR · POPE
5Weeks 9–11AgentsPerceive, plan, and take actions in software.ViperGPT · Mind2Web · OSWorld
6Weeks 11–13Action & world modelsAct in the physical world; imagine what follows.RT-2 · OpenVLA · DreamZero

The same roadmap as p. 24, stage 6 highlighted. SigLIP 2: Lecture 3.

C · Part 4: Acting in the World Actions as TokensWeek 12 · seminal

Actions as Tokens

RT-2 Figure 1: internet-scale VQA data and robot action data, where a robot action is written as a string of numbers; RT-2 is a large language model with a ViT that reads the image and the question and outputs action tokens, de-tokenized into a translation and rotation; it is co-fine-tuned and deployed for closed-loop robot control

Discretize the end-effector delta, put it in the vocabulary, and co-fine-tune on web VQA.

The page has no headline, so its page title is the slide title. Figure: Brohan et al. (2023), Fig. 1.

C · Part 4: Acting in the World Generalist VLAs and the Data ProblemWeek 12 · seminal

Generalist VLAs and the Data Problem

OpenVLA Figure 1: large-scale robot training data (970k robot episodes) fine-tune a base VLM (Llama 2 7B with a ViT) into OpenVLA, a vision-language-action model that answers 'Wipe the table.' with action deltas; photos of multi-robot control; fully open-source data, weights and code

An open 7B VLA trained on 970k cross-embodiment episodes — the bottleneck is trajectories, not parameters.

The page has no headline, so its page title is the slide title. Figure: Kim et al. (2024), Fig. 1.

C · Part 4: Acting in the World Fast, Continuous ControlWeek 13

Discrete tokens are a poor fit for smooth, continuous motion. Two fixes: predict chunks, and hide the latency.

Diffusion Policy Figure 2: (a) general formulation: from an image observation sequence, the diffusion policy denoises an action sequence over a prediction horizon and executes part of it; (b) a CNN-based and (c) a Transformer-based noise-prediction network, repeated K times

Diffusion Policy denoises a whole action sequence conditioned on recent observations, instead of emitting one step at a time.

First fix (deck: "Chi et al., Diffusion Policy, RSS 2023 · Tang et al., VLASH, 2025/26 — figures from the papers."). Figure: Chi et al. (2023), Fig. 2 (arXiv v5, the extended journal version).

C · Part 4: Acting in the World Fast, Continuous ControlWeek 13

Discrete tokens are a poor fit for smooth, continuous motion. Two fixes: predict chunks, and hide the latency.

VLASH Figure 3: (a) synchronous inference: the robot stalls while the model infers; (b) naive asynchronous inference: the model predicts from a stale state, so the actions are discontinuous; (c) VLASH rolls the robot state forward and predicts from the future state, giving fast reaction and smooth actions

Asynchronous inference acts on predictions made for a future state, so inference delay stops turning into jerky motion.

Second fix. Figure: Tang et al. (2025), Fig. 3.

C · Part 4: Acting in the World World Models: Learning to ImagineWeek 13 · seminal

Compress the observation, learn the dynamics, and train the controller inside the dream.

  • V compresses the frame, M predicts the next latent, C is a tiny controller sitting on top of both.
  • If your model of the world is good enough, you can plan inside it instead of in the world
  • The 2018 version is a VAE plus an RNN. The 2026 version is a video generator.
World Models Figure 4: at each time step the vision model V encodes the observation into a latent z, the memory RNN M integrates the codes into a hidden state h that can predict future states, and a small controller C uses z and h to choose the action a

Figure: Ha & Schmidhuber (2018), Fig. 4.

C · Part 4: Acting in the World World-Action ModelsWeek 13 · frontier

World-Action Models

DreamZero Figure 1: a world action model takes video, language and proprioception and predicts future videos and continuous actions; it learns from diverse, non-repetitive pretraining data, generalizes to unseen tasks in unseen environments, transfers across embodiments including humans, and adapts to a new robot with 30 minutes of play data
  • Joint video-action modelling borrows supervision from videos alone (cheaper than robot teleop data).
  • Rolling out an imagined future closes the loop: predict, act, re-observe, correct.
  • The claimed payoff is zero-shot transfer to unseen tasks, scenes and embodiments.
  • Predict the future frames and the actions that get you there.
  • Learned from video, not from teleoperation.

The page has no headline, so its page title is the slide title. Figure: Ye et al. (2026), Fig. 1.

C · Part 4: Acting in the World Result

A policy trained only in a learned dream beat the best earlier result in the real game

TaskAgentAverage score
VizDoom Take CoverBest earlier result (OpenAI Gym leaderboard)820 ± 58
VizDoom Take CoverWorld model, trained only inside its dream, tested in the real game1092 ± 556
CarRacing-v0Best earlier deep RL method (A3C, discrete)652 ± 10
CarRacing-v0World model (V, M and C together)906 ± 21

Score: Take Cover: time steps survived (solved above 750). CarRacing-v0: average reward over 100 trials (solved at 900).

Dream training: model M predicts the next frame, with temperature τ = 1.15. The spread (± 556) is large.

Source: Ha & Schmidhuber (2018), §3.3 and Table 1 (CarRacing-v0, p. 6); §4.4 and Table 2 (Take Cover, τ = 1.15, pp. 8 and 10); read 2 Nov 2026.

Part C · Advanced topics and today's state of the art

Today's state of the art, and safety

1Qwen3-VL
2Part 1: Building a VLM
3Part 3: Making It Act
4Part 4: Acting in the World
5Today's state of the art, and safety

C · Today's state of the art, and safety Read on 10 Oct 2026Check again the week of 4 Nov 2026

Today's best vision-language models

Rank (interval)Model on the leaderboardDeveloperWeightsScore
1 (1–10)claude-fable-5-highAnthropicClosed1308
2 (1–19)qwen3.8-maxAlibabaClosed1301
6 (1–26)gemini-3.7-flash-highGoogleClosed1297
8 (1–37)gpt-6.1-sol-maxOpenAIClosed1294
35 (11–47)glm-5.3-flashZ.aiOpen (MIT)1273
44 (34–58)gemma-4-31bGoogleOpen (Apache 2.0)1262
83 (74–91)qwen3-vl-235b-a22b-instructAlibabaOpen (Apache 2.0)1215

Score: a rating from human votes on two answers to one image. The intervals overlap: the top rank is not certain.

The top changes fast. In May 2026, CS231n named Gemini the best closed VLM. Qwen3-VL (Nov 2025) is now rank 83.

Vision Arena leaderboard, arena.ai/leaderboard/vision (page date 9 Oct 2026; read 10 Oct 2026): rank, 95% rank interval, model name and licence as listed; one closed row per developer, the top two open rows, and Qwen3-VL. Open weight vs fully open source: CS231n pp. 88–89. "Gemini widely considered the best proprietary Vision-Language model": Fei-Fei et al. (2026b), CS231n lecture 16, p. 86.

C · Today's state of the art, and safety LO5

An image can carry an attack

Typed text in an image (FigStep)
FigStep Figure 1: a vanilla text query passes the textual module's safety alignment and is rejected; the same query sent as a typographic image through the visual module and the connector reaches the textual module and is accepted

Often it does not refuse. On six open VLMs, 82.50% of the requests got a harmful answer (average).

Why: the LLM learned to refuse harmful text. The image path did not get that safety training.

Adversarial pixels
Qi et al. Figure 1, top: a benign image of a panda and a visual adversarial example that looks the same; the second image is optimized on a small corpus, with a pixel change of at most epsilon

Change each pixel by a very small amount (ε = 16/255). The change is hard to see. One such image made MiniGPT-4 obey harmful requests that it refused before.

The same problem as prompt injection through a tool result (Lecture 2 session 02): data and instructions share one input.

Gong et al. (2025), Fig. 1 and abstract (82.50%, six open-source LVLMs). Qi et al. (2024), Fig. 1 (top part only; the model outputs in the figure are not shown) and its caption (ε = 16/255, MiniGPT-4).

Part C — advanced topics and today's state of the art

Since 2023, VLMs learned to see more, to think, and to act.

Part C: six methods, each as why, how and results: the Qwen3-VL recipe, thinking, one model for all modalities, computer-use agents, robots and world models; then today's leaderboard and one safety risk.

Close of Part 2

Part 2 in one view

What each lecture gave us, and what is still open.

Close of Part 2 · Topic 7

Topic 7: every phrase of the syllabus, and where we did it

Syllabus phraseWhere today
Extending LLMs to other dataParts A and B
… and promptingPart B, last segment
Especially visual data (LLVMs)All three parts
(CLIP)Lecture 3; Part A
Applications; other multi-modal modelsPart C

Close of Part 2 · the plan from the first lecture

One recipe, now with images

LectureWhat it gave a multi-modal model
1 · 01 ArchitectureThe Transformer and next-token prediction
1 · 02 TrainingPre-training at scale, and cheap adaptation (LoRA)
2 · 01 Post-trainingInstruction tuning and RL: answers, not continuations
2 · 02 PromptingPrompts, reasoning and agents with a fixed model
3 Self-supervisedImage encoders trained without labels, such as CLIP
4 Multi-modalConnect the encoder to the LLM, train it, prompt it, let it act

Close of Part 2 · Still open

Questions you could work on

  • Fewer image tokens without losing detail
  • Grounding: counting, positions, small text
  • Long video, and memory over hours
  • One model that understands and generates every modality
  • Agents and robots that stay reliable over long tasks

Close of Part 2 · Where to go next

Where to go next

Code

nanoGPT (Lecture 1) and nanoVLM (today): read them, change one piece, measure.

LLaVA: the paper and its open code.

Courses

Stanford CS231n, Lecture 16 (vision and language)

Georgia Tech CS 8803-VLM (vision-language models and agents)

CMU 11-777 (multimodal machine learning)

Close of Part 2 · Revision

How to revise Part 2

  • Work each derivation again from the start.
  • For each method, say what problem it solves, what it costs, and when it fails.

The format and the logistics of the exam are on Canvas.

Where we got to

How did LLMs learn to see?

Answered today

An encoder turns the image into vectors, a small projector maps them to the LLM's width, and the LLM learns from image–text data and instructions. The same recipe now reasons and acts.

Thank you

Thank you for taking Part 2. Questions: on Canvas.

← → or PageUp/PageDown · F full screen · O overview · B black screen