Part 2 - Lecture 3

Self-Supervised
Foundation Models

Session 01 of 1 · Learning from images without labels

CEG5305 Introduction to Generative AI with Foundation Models
Junyuan Hong · AY2026/27 Semester 1

2026.11.03v1

Before we start

About these slides

These slides were prepared with the help of AI tools (Claude, by Anthropic), working from the reference slides and papers listed here.

Reference slides · full list on the lecture page
Key papers · more on the lecture page

MAE (He et al., 2022) · SimCLR (Chen et al., 2020) · CLIP (Radford et al., 2021) · DINO (Caron et al., 2021)

Recap · Lectures 1 and 2

Recap: Large Language Models

LectureTitleKey message
1 · 01Architecture — what an LLM is made ofArchitecture or scale: why do LLMs work so well?
Transformer · attention · RNN · nanoGPT
1 · 02Training — how the model is actually trainedHow does scale make LLMs work well?
GPU parallelism · scaling laws · PEFT (LoRA)
2 · 01Post-training — from a text predictor to an assistantWhat turns GPT into ChatGPT?
SFT · RLHF · DPO · GRPO
2 · 02Prompting — from one call to a loopHow far can a fixed model go?
ICL · chain-of-thought · agent · harness

Lectures 1 and 2 were about text. Today: images, and no labels.

Outline · Lecture 3 = Topic 6, three hours

One session: history, methods, today's best

Session 01 · today · 3 h

Self-supervised foundation models

☕Break18:35 · 10 min
☕Break19:05 · 10 min
·Summary and next lecture19:34 · 5 min

Recap · Lecture 1

Transformers can do more than language

But how? What is the general principle that we learned from LLM training?

Image generation

DiT · Peebles & Xie (2023)

Video

ViViT · Arnab et al. (2021)

Speech

Whisper · Radford et al. (2023)

Time series

PatchTST · Nie et al. (2023)

Proteins

ESM-2 · Lin et al. (2023)

Examples: eagle, USFWS; Armstrong, Apollo 11, NASA; Muybridge (1878); all public domain. DiT sample, facebookresearch/DiT (CC BY-NC 4.0). Ubiquitin, PDB 1UBQ.

Recap · Lecture 1

LLM pre-training: Learning with manual labels

Input: a text prefix from the web The cat sat on the LLM Prediction mat? The same text, one word further The cat sat on the mat label: the next word (no person wrote it) loss: prediction vs. label

Prediction target = a part of the input. The text gives its own label

Recap · Lecture 1

Pre-training vs. SFT

Supervised learningPre-training (self-supervised)
DataPairs: an instruction and the answer a person wroteRaw text, nothing added
LabelGiven by people, from outside the dataCut from the data: the next word
Cost of more dataEach label needs a person's timeAny text on the web works
What the model learnsOne behaviour: follow instructionsThe structure of language, for many tasks

Supervised: the label comes from people. Self-supervised: the label comes from the data.

Today's question

Self-supervised learning

1 · The idea

What is the general idea of self-supervised learning?

2 · When to use it

When is it the right method?

3 · Beyond language

Does it work for other modalities, such as images?

Part A

How learning gets rid of labels

A · Self-Supervised Learning

Supervised learning for computer vision

Task

Predict the category from a given image.

Dataset

Pairs of an image x and a category y that people wrote. Example: ImageNet, 1,281,167 training images in 1,000 categories.

Learning objective

Give the correct category y a high probability p(y | x): minimise the cross-entropy loss, −log p(y | x).

input image conv layers: features fullyconnected Conv Net 1 · cat0.9 2 · dog0.1 one score per category Prediction: index 1 (cat)

Cat photo: Stanford CS231n (2026), Lecture 12, p. 6, where the image is marked “CC0 public domain”. The scores 0.9 and 0.1 are an illustration.

A · Self-Supervised Learning

Representations: Meaning of Images

Left: nearest neighbours of five test images in pixel space. Right: nearest neighbours in the 4096-dimensional feature space of AlexNet: flowers, elephants, ships, pumpkins and dogs match their own class. AlexNet diagram with the 4096-dim layer boxed in red

Slide from Stanford CS231n (2026), Lecture 12, p. 9. (p. 8 shows the same figure under the title “Last Week: Visualizing and Understanding”.) Figure: Krizhevsky et al. (2012).

A · Self-Supervised Learning

Learned Representations

AlexNet: an input image of a dog, five convolution layers and three fully connected layers; the 4096-dim layer before the class labels is boxed in red

What is the problem with large-scale training?

- We need a lot of labeled data

Is there a way we can train neural networks without the need for huge manually labeled datasets?

Slide from Stanford CS231n (2026), Lecture 12, p. 10. Figure: Krizhevsky et al. (2012); the deck adds “Figures reproduced with permission”.

A · Self-Supervised Learning

Self-Supervised Learning

Diagram: a dataset with no labels trains an encoder on a pretext objective; the trained encoder then goes to a dataset with labels and a downstream objective

1. Pretext Task

  • Define a task based on the data itself
  • No manual annotation
  • Could be considered an unsupervised task;
  • but we learn with supervised learning objectives, e.g., classification or regression.

2. Downstream Task

  • The application you care about
  • You do not have large datasets
  • The dataset is labeled

Slide from Stanford CS231n (2026), Lecture 12, p. 11.

A · Self-Supervised Learning

Self-Supervised Learning – Pretext Task

A dataset with no labels goes into an encoder; the learned representation goes into a decoder, classifier or regressor; the labels or outputs are automatically generated from data

Slide from Stanford CS231n (2026), Lecture 12, p. 12.

A · Self-Supervised Learning

Self-Supervised Learning – Downstream Task (supervised)

A dataset with labels goes into the encoder; the learned representation goes into a small FC layer trained with labels; the decoder of the pretext task is faded out. Label: semi-supervised

Slide from Stanford CS231n (2026), Lecture 12, p. 13.

A · Self-Supervised Learning

Self-supervised pretext tasks

Example: learn to predict image transformations / complete corrupted images

Four pretext tasks on a cat photo: image completion (top half hidden), rotation prediction (rotated by an unknown angle), jigsaw puzzle (four shuffled tiles), colorization (grayscale image)
  1. Solving the pretext tasks allow the model to learn good features.
  2. We can automatically generate labels for the pretext tasks.

Slide from Stanford CS231n (2026), Lecture 12, p. 14.

A · Self-Supervised Learning

How to evaluate a self-supervised learning method?

  • Pretext Task Performance
    • Measure how well the model performs on the task it was trained on without labels.
  • Representation Quality
    • Evaluate the quality of the learned representations
      • Linear Evaluation Protocol: Train a linear classifier on the leaerned representations;
      • Clustering: Measure clustering performance;
      • t-SNE: Visualize the representations to assess their separability.)
  • Robustness and Generalization
    • Test how well the model generalizes to different datasets and is robust to variations.
  • Computational Efficiency
    • Assess the efficiency of the method in terms of training time and resource requirements.
  • Transfer Learning and Downstream Task Performance
    • Assess the utility of the learned representations by transferring them to a downstream supervised task.

Slide from Stanford CS231n (2026), Lecture 12, p. 15. The deck shows this list in one column.

A · Self-Supervised Learning

How to evaluate a self-supervised learning method?

Pipeline: lots of unlabeled data, self-supervised learning (a convnet predicts a 90 degree rotation), feature extractor; then supervised learning with a small amount of labeled data on the target task, and evaluation on the target task (a conv network with a linear classifier predicts bird)

1. Learn good feature extractors from self-supervised pretext tasks, e.g., predicting image rotations

2. Attach a shallow network on the feature extractor; train the shallow network on the target task with small amount of labeled data

Slide from Stanford CS231n (2026), Lecture 12, pp. 16–17. One animation; its final state.

A · Self-Supervised Learning

Broader picture

Today’s lecture

Self-supervised learning in four fields: computer vision (Doersch et al., 2015; boxed as today's lecture), language modeling (GPT-4, OpenAI 2023), speech synthesis (WaveNet, van den Oord et al., 2016), robot and reinforcement learning (Dense Object Net, Florence and Manuelli et al., 2018)

Slide from Stanford CS231n (2026), Lecture 12, p. 18. Works shown: Doersch et al. (2015); OpenAI (2023); van den Oord et al. (2016); Florence et al. (2018).

Part A · How learning gets rid of labels

Pretext tasks from image transformations

1Self-Supervised Learning
2Pretext tasks from image transformations
3Contrastive representation learning

A · Pretext tasks from image transformations

Pretext tasks

Goal

Guide your model to understand visual semantics (representations of meanings)

Define targets by image transformation
  1. Predict rotations
  2. Predict relative patch locations
  3. Solve jigsaw puzzles
  4. Predict missing pixels (inpainting)
  5. Image coloring
  6. Reconstruction-based learning (MAE)

A · Pretext tasks from image transformations

Pretext task: predict rotations

Five photos, each rotated: a bird by 90 degrees, a dog by 270 degrees, a person by 180 degrees, a frog by 0 degrees, an object by 270 degrees

Hypothesis: a model could recognize the correct rotation of an object only if it has the “visual commonsense” of what the object should look like unperturbed.

Slide from Stanford CS231n (2026), Lecture 12, p. 21. Figure: Gidaris et al. (2018).

A · Pretext tasks from image transformations

Pretext task: predict rotations

An image X is rotated by 0, 90, 180 and 270 degrees; each rotated image goes into the same ConvNet model F, which must maximize the probability of the correct rotation label y = 0, 1, 2 or 3

Self-supervised learning by rotating the entire input images.

The model learns to predict which rotation is applied (4-way classification)

Slide from Stanford CS231n (2026), Lecture 12, pp. 22–23. One animation; its final state. Figure: Gidaris et al. (2018).

A · Pretext tasks from image transformations

Evaluation on semi-supervised learning

Test accuracy against the number of training examples (20 to 5000) on CIFAR-10: the semi-supervised model with rotation features is better than the supervised model below about 1000 examples

Self-supervised learning on CIFAR10 (entire training set).

Freeze conv1 + conv2
Learn conv3 + linear layers with subset of labeled CIFAR10 data (classification).

Slide from Stanford CS231n (2026), Lecture 12, p. 24. Figure: Gidaris et al. (2018).

A · Pretext tasks from image transformations

Transfer learned features to supervised learning

Table of PASCAL VOC 2007 classification (mAP), detection (mAP) and segmentation (mIoU) results: ImageNet labels 79.9, 56.8, 48.0; random initialization 53.3, 43.4, 19.8; many self-supervised methods; RotNet (ours) 72.97, 54.4, 39.1. Caption: self-supervised learning with rotation prediction

Full supervised learning

No pretraining

Self-supervised learning on ImageNet (entire training set [1,281,167 images]) with AlexNet.

Finetune on labeled data from Pascal VOC 2007 [trainval: 5,011 images].

Slide from Stanford CS231n (2026), Lecture 12, p. 25. Figure: Gidaris et al. (2018). Image counts in brackets (added): ILSVRC 2012 training set, image-net.org; VOC 2007 trainval, Ultralytics VOC dataset page. Retrieved 3 Nov 2026.

A · Pretext tasks from image transformations

Visualize learned visual attentions

Attention maps at conv1, conv3 and conv5 for three images (people, a dog, a cat): (a) a supervised model and (b) the self-supervised rotation model; both focus on the same object parts

Self-supervised learned attention has better interpretation.

A · Pretext tasks from image transformations

Pretext task: predict relative patch locations

A cat photo with a centre patch and eight numbered neighbours; the task is X = (centre patch, other patch), Y = 3, the position of the second patch. Two example questions with other photos

Slide from Stanford CS231n (2026), Lecture 12, p. 27. Figure: Doersch et al. (2015).

A · Pretext tasks from image transformations

Pretext task: solving “jigsaw puzzles”

A car photo cut into a 3 by 3 grid of tiles; the tiles are shuffled by a permutation from a permutation set; nine networks with shared weights process the tiles and predict the permutation index, one of 64

The model predicts the arrangement of the shuffled tiles (the index of the permutation -- learn the relation of patches in the image), not the object category.

Slide from Stanford CS231n (2026), Lecture 12, p. 28. Figure: Noroozi & Favaro (2016), Fig. 3

A · Pretext tasks from image transformations

Pretext task: predict missing pixels (inpainting)

A street photo with a white square in the centre, and the same photo with the square filled in by the model

Context Encoders: Feature Learning by Inpainting (Pathak et al., 2016)

Slide from Stanford CS231n (2026), Lecture 12, p. 30. Figure: Pathak et al. (2016).

A · Pretext tasks from image transformations

Learning to inpaint by reconstruction

Auto Encoder

A football photo with a missing square goes into an encoder; the encoder features pass through a channel-wise fully connected layer to a decoder; the loss L compares the decoder output with the true missing region

Learning to reconstruct the missing pixels

Slide from Stanford CS231n (2026), Lecture 12, p. 31. Figure: Pathak et al. (2016).

A · Pretext tasks from image transformations

Inpainting evaluation

Input with a missing square, and the reconstruction: the filled square is blurry

Slide from Stanford CS231n (2026), Lecture 12, p. 32. Figure: Pathak et al. (2016).

A · Pretext tasks from image transformations

Learning to inpaint by reconstruction

(We will talk about adversarial learning in the next lecture)

Loss = reconstruction + adversarial learning

\[L(x) = L_{recon}(x) + L_{adv}(x)\] \[L_{recon}(x) = ||M * (x - F_\theta((1 - M) * x))||_2^2\] \[L_{adv} = \max\nolimits_D \mathbb{E}[\log(D(x))] + \log(1 - D(F((1 - M) * x)))]\]

\(*\): Element wise multiplication

\(M = \begin{cases} 0 & \text{not masked} \\ 1 & \text{masked} \end{cases}\)

\(F_\theta\): Encoder

Added: \(x\) is the input image. \(F_\theta\) maps the image with a hole, \((1 - M) * x\), to a full image: image → image. It is the network that we train.

Added: \(D\) is a second network, the discriminator. It takes an image and outputs the probability that the image is real, not inpainted.

Adversarial loss between “real” images and inpainted images

Slide from Stanford CS231n (2026), Lecture 12, p. 33. Source: Pathak et al. (2016). The two blue notes on \(F_\theta\) and \(D\) are added.

A · Pretext tasks from image transformations

Inpainting evaluation

Four images: input with a missing square; reconstruction loss only (blurry); adversarial loss only (sharp but wrong texture); reconstruction plus adversarial (sharp and plausible)

Slide from Stanford CS231n (2026), Lecture 12, p. 34. Figure: Pathak et al. (2016).

A · Pretext tasks from image transformations

Transfer learned features to supervised learning

Table: pretraining method, supervision, pretraining time, and classification, detection and segmentation results. ImageNet labels 78.2, 56.8, 48.0; random Gaussian 53.3, 43.4, 19.8; autoencoder 53.8, 41.9, 25.2; Agrawal, Wang, Doersch; Ours (context) 56.5, 44.5, 30.0

Self-supervised learning on ImageNet training set, transfer to classification (Pascal VOC 2007), detection (Pascal VOC 2007), and semantic segmentation (Pascal VOC 2012)

Slide from Stanford CS231n (2026), Lecture 12, p. 35. Table: Pathak et al. (2016).

A · Pretext tasks from image transformations

Pretext task: image coloring

A grayscale fish photo, the L channel X in R^(H x W x 1), goes through a network F that predicts the colour information, the ab channels Y-hat in R^(H x W x 2)

Slide from Stanford CS231n (2026), Lecture 12, p. 36. The deck credits “Richard Zhang / Phillip Isola”. See Zhang et al. (2016).

A · Pretext tasks from image transformations

Pretext task: image coloring

The grayscale fish, L channel X, and the predicted colour result: concatenate the L and ab channels (X, Y-hat); the fish is yellow and dark blue

Slide from Stanford CS231n (2026), Lecture 12, p. 37. The deck credits “Richard Zhang / Phillip Isola”. See Zhang et al. (2016).

A · Pretext tasks from image transformations

Learning features from colorization: Split-brain Autoencoder

Split-brain autoencoder on a photo of rose hips: network F1 predicts the colour channels from the grayscale channel, network F2 predicts the grayscale channel from the colour channels; together they give the predicted image

F1 sees only the grayscale channel and predicts the colour channels. F2 sees only the colour channels and predicts the grayscale channel. Each network predicts what the other one sees.

Slide from Stanford CS231n (2026), Lecture 12, p. 38. The deck credits “Richard Zhang / Phillip Isola”. Figure: Zhang et al. (2017). The caption under the figure is added.

A · Pretext tasks from image transformations

Learning features from colorization: Split-brain Autoencoder

Idea: cross-channel predictions

Split-Brain Autoencoder: the input X is split into X1 and X2; F1 predicts X2-hat from X1, F2 predicts X1-hat from X2; the two predictions form X-hat

Slide from Stanford CS231n (2026), Lecture 12, p. 39. The deck credits “Richard Zhang / Phillip Isola”. Figure: Zhang et al. (2017).

A · Pretext tasks from image transformations

Transfer learned features to supervised learning

Top-1 accuracy on Places for linear classifiers on each layer, conv1 to pool5: Places labels and ImageNet labels (supervised) are highest; the split-brain autoencoder (this paper) is the best self-supervised method, above Pathak, Zhang, Owens, Donahue, Doersch, Wang and Gupta, Krähenbühl and Gaussian initialization

Self-supervised learning on ImageNet (entire training set).

Use concatenated features from F₁ and F₂

Labeled data is from the Places (Zhou 2016).

Slide from Stanford CS231n (2026), Lecture 12, p. 40. Figure: Zhang et al. (2017). Places: Zhou et al. (2016).

A · Pretext tasks from image transformations

Pretext task: image coloring

A historical black-and-white photo of a thylacine (Tasmanian tiger), coloured automatically

Slide from Stanford CS231n (2026), Lecture 12, p. 41. The deck credits “Richard Zhang / Phillip Isola”. See Zhang et al. (2016).

A · Pretext tasks from image transformations

Pretext task: image coloring

A black-and-white landscape photo of Yosemite (a bridge, trees, a lake and Half Dome), coloured automatically

Slide from Stanford CS231n (2026), Lecture 12, p. 42. The deck credits “Richard Zhang / Phillip Isola”. See Zhang et al. (2016).

A · Pretext tasks from image transformations

Masked Auto Encoders (MAE)

Scaling inpainting idea: Reconstruction with a larger masked portion

MAE on a cat photo. Top row, 50% masking ratio: masked input, reconstruction, reconstruction plus visible patches, original. Bottom row, 75% masking ratio: the same four images; the reconstruction is still close to the original

Slide from Stanford CS231n (2026), Lecture 12, p. 48. Figure: He et al. (2022)

A · Pretext tasks from image transformations

Masked Auto Encoders (MAE)

MAE Figure 1: an input image with most patches masked; the visible patches go into a large encoder; its outputs and grey mask tokens go into a small decoder, which outputs the target image

Slide from Stanford CS231n (2026), Lecture 12, p. 49. Figure: He et al. (2022), Fig. 1.

A · Pretext tasks from image transformations

Masking Methods

  • Similar to the original ViT,
    divide the input into non-overlapping patches.
  • Uniformly sample a very large proportion (75%) of these patches and mask them
  • Masking a high ratio makes predicting the task challenging and meaningful.
  • Also, not using mask tokens and picking a high sampling ratio (masking most of the image, e.g., 75%, and sampling a small visible portion, e.g., 25%, to feed into the encoder) enables the encoder to be very large.
MAE Figure 1, small: masked input, encoder, mask tokens, decoder, target
Figure 6. MAE architecture
from the paper

Slide from Stanford CS231n (2026), Lecture 12, p. 50. Figure: He et al. (2022), Fig. 1 (the page labels it “Figure 6”).

A · Pretext tasks from image transformations

MAE Encoder

  • The encoder only operates
    on unmasked patches (25%)
  • Embeds the patches by linear projection
    and add positional embeddings
  • Uses transformer blocks
MAE Figure 1, small: masked input, encoder, mask tokens, decoder, target
Figure 6. MAE architecture
from the paper
  • Since the input patches are a small part of the input, the encoder is chosen to be very large. (encoder has over 9 times computations per token vs decoder)

Slide from Stanford CS231n (2026), Lecture 12, p. 51. Figure: He et al. (2022), Fig. 1 (the page labels it “Figure 6”).

A · Pretext tasks from image transformations

MAE Decoder

  • Merges the encoder outputs with the shared mask tokens in previously masked places, adding positional encodings to them.
  • Uses transformer blocks, followed by a linear projection for finalizing pixel reconstruction.
MAE Figure 1, small: masked input, encoder, mask tokens, decoder, target
Figure 6. MAE architecture
from the paper
  • Is solely responsible for reconstruction, meaning it is not used post-training. Hence, it is independent of the encoder design, making it flexible (unlike traditional AEs or UNet). This is an asymmetrical autoencoder design.

Slide from Stanford CS231n (2026), Lecture 12, p. 52. Figure: He et al. (2022), Fig. 1 (the page labels it “Figure 6”).

A · Pretext tasks from image transformations

Reconstruction

  • The MSE (mean squared error loss) in the pixel space between the input image and the reconstructed image is adopted.
  • Loss is only computed for masked patches

Slide from Stanford CS231n (2026), Lecture 12, p. 53.

A · Pretext tasks from image transformations

Linear Probing vs Full Fine-tuning

  • In linear probing, the pre-trained model is fixed, and only one linear layer is added at the end, to predict the labels (or produce the output). This method is used to assess the quality of representations from a pre-trained feature extraction model.
  • In fine-tuning, pre-trained model is further trained (not fixed), and one or more layers, possibly with non-linearities are added.
  • linear probing: provides a measure of representation quality of a pre-training in restricted conditions
  • fine-tuning: exploits models near-true potential to adopt for new tasks
Two diagrams. Top: a frozen Encoder (snowflake), its Learned Representation, and a trained Linear layer (flame). Bottom: a trained Encoder (flame), its Learned Representation, and a trained Linear/FC layer (flame)

Slide from Stanford CS231n (2026), Lecture 12, p. 54; diagrams cropped from the same page.

A · Pretext tasks from image transformations

Ablation Studies

So many modeling/hyperparameter choices:

  • Masking ratio
  • Decoder depth
  • Decoder width
  • Mask token (used or not in encoder)
  • Reconstruction target
  • Data augmentation
  • Mask sampling method
  • Training schedule
MAE Figure 5: accuracy against masking ratio from 10 to 90 percent. Fine-tuning: 83.2, 83.4, 83.4, 84.7, 84.9, 85.0, 84.9, 84.9, 84.5, 83.0. Linear probing: 54.6, 58.9, 61.7, 67.0, 69.9, 71.8, 73.2, 73.5, 71.8, 66.1

Slide from Stanford CS231n (2026), Lecture 12, p. 55 (first of two slides). Figure: He et al. (2022), Fig. 5 (ViT-L).

A · Pretext tasks from image transformations

Ablation Studies

MAE Table 1f, mask sampling: random 75 percent, 84.9 fine-tuning, 73.5 linear; block 50 percent, 83.9 and 72.3; block 75 percent, 82.8 and 63.9; grid 75 percent, 84.0 and 66.0
MAE Figure 6: an ambulance and a fox masked by random 75 percent, block 50 percent and grid 75 percent sampling, with their reconstructions

Slide from Stanford CS231n (2026), Lecture 12, p. 55 (second of two slides). Figures: He et al. (2022), Table 1f and Fig. 6 (ViT-L; ft: fine-tuning, lin: linear probing).

A · Pretext tasks from image transformations

Masked Autoencoder – Comparisons

MAE Figure 2, first column: four triplets of masked image, MAE reconstruction and original: a clock, a bus, a dog and a lamp, with 80 percent of the patches masked
MAE Table 3: ImageNet-1K fine-tuning accuracy. Scratch: 82.3 ViT-B, 82.6 ViT-L, 83.1 ViT-H. DINO 82.8 ViT-B. MoCo v3 83.2 and 84.1. BEiT 83.2 and 85.2, with DALLE data. MAE 83.6, 85.9, 86.9, and 87.8 for ViT-H at 448

Slide from Stanford CS231n (2026), Lecture 12, p. 56. Figures: He et al. (2022), Fig. 2 (first column; 80% masked) and Table 3.

A · Pretext tasks from image transformations

Summary: pretext tasks from image transformations

  • Pretext tasks focus on “visual common sense”, e.g., predict rotations, inpainting, rearrangement, and colorization.
  • The models are forced learn good features about natural images, e.g., semantic representation of an object category, in order to solve the pretext tasks.
  • We often do not care about the performance of these pretext tasks, but rather how useful the learned features are for downstream tasks (classification, detection, segmentation).
  • Problems: 1) coming up with individual pretext tasks is tedious, and 2) the learned representations may not be general.

Slide from Stanford CS231n (2026), Lecture 12, pp. 57–58. One animation; its final state.

Part A · How learning gets rid of labels

Contrastive representation learning

1Self-Supervised Learning
2Pretext tasks from image transformations
3Contrastive representation learning

A · Contrastive representation learning

Pretext tasks from image transformations

The four pretext tasks on the cat photo: image completion, rotation prediction, jigsaw puzzle, colorization

Learned representations may be tied to a specific pretext task!

Can we come up with a more general pretext task?

Slide from Stanford CS231n (2026), Lecture 12, p. 59.

A · Contrastive representation learning

A more general pretext task?

The four transformed versions of the cat photo all point with green arrows to the original cat: same object. A red arrow joins the cat and a puppy: different object

Slide from Stanford CS231n (2026), Lecture 12, pp. 60–61. One animation; its final state.

A · Contrastive representation learning

Contrastive Representation Learning

The same picture: the cat and its four transformed versions attract (green arrows); the cat and the puppy repel (red arrow)

Slide from Stanford CS231n (2026), Lecture 12, pp. 62, 64. Page 64 repeats page 62 after the deck’s agenda page.

Part A

Part A: key methods and insights

BREAK
10 minutesBack at 18:45

Coming up in Part B: contrastive representation learning, from the formulation to DINO.

Part B

Contrastive representation learning

B · Intuition and formulation

Contrastive Representation Learning

The same picture with symbols: the cat is the reference x, each transformed version is a positive x+, the puppy is a negative x-

Slide from Stanford CS231n (2026), Lecture 12, p. 65.

B · Intuition and formulation

A formulation of contrastive learning

What we want:

\[ \mathrm{score}(f(x), f(x^+)) >> \mathrm{score}(f(x), f(x^-)) \]

x: reference sample; x+ positive sample; x- negative sample

Given a chosen score function, we aim to learn an encoder function f that yields high score for positive pairs (x, x+) and low scores for negative pairs (x, x-).

Slide from Stanford CS231n (2026), Lecture 12, p. 66.

B · Intuition and formulation

A formulation of contrastive learning

Loss function given 1 positive sample and N - 1 negative samples:

\[ L = -\mathbb{E}_X\left[\log\frac{\textcolor{#3b7d3b}{\exp(s(f(x), f(x^+))}}{\textcolor{#3b7d3b}{\exp(s(f(x), f(x^+))} + \textcolor{#b23b2e}{\sum_{j=1}^{N-1}\exp(s(f(x), f(x_j^-))}}\right] \]
score for the positive pairscore for the N-1 negative pairs

This seems familiar …

Cross entropy loss for a N-way softmax classifier!

I.e., learn to find the positive sample from the N samples

A photo of a cat, x, and the same photo rotated, x plus The cat photo x, and three other photos: a puppy x1 minus, a container ship x2 minus, another cat x3 minus, and dots for more

Slide from Stanford CS231n (2026), Lecture 12, pp. 67–70 (one build, final state). Photos cropped from p. 68.

B · Intuition and formulation

A formulation of contrastive learning

Loss function given 1 positive sample and N - 1 negative samples:

\[ L = -\mathbb{E}_X\left[\log\frac{\exp(s(f(x), f(x^+))}{\exp(s(f(x), f(x^+)) + \sum_{j=1}^{N-1}\exp(s(f(x), f(x_j^-))}\right] \]

Commonly known as the InfoNCE loss (van den Oord et al., 2018)

A lower bound on the mutual information between f(x) and f(x+)

\[ MI[f(x), f(x^+)] - \log(N) \geq -L \]

The larger the negative sample size (N), the tighter the bound

Detailed derivation: Poole et al., 2019

Slide from Stanford CS231n (2026), Lecture 12, p. 71.

Part B · Contrastive representation learning

Instance contrastive learning: SimCLR and MOCO

1Intuition and formulation
2Instance contrastive learning: SimCLR and MOCO
3Sequence contrastive learning: CPC
4Self-Distillation Without Labels, DINO

B · Instance contrastive learning: SimCLR and MOCO

SimCLR: A Simple Framework for Contrastive Learning

Cosine similarity as the score function:

\[ s(u, v) = \frac{u^T v}{||u||\,||v||} \]

Use a projection network g(·) to project features to a space where contrastive learning is applied

Generate positive samples through data augmentation:

  • random cropping, random color distortion, and random blur.
SimCLR Figure 2: an image x is transformed by two augmentations t and t-prime into x-tilde-i and x-tilde-j; each passes through the encoder f to a representation h, then through the projection head g to z; the loss maximizes agreement between z-i and z-j

Slide from Stanford CS231n (2026), Lecture 12, p. 72. Source: Chen et al., 2020. Figure: Chen et al. (2020), Fig. 2.

B · Instance contrastive learning: SimCLR and MOCO

SimCLR: generating positive samples from data augmentation

SimCLR Figure 4: one photo of a dog under ten augmentations: (a) original, (b) crop and resize, (c) crop, resize and flip, (d) colour drop, (e) colour jitter, (f) rotate, (g) cutout, (h) Gaussian noise, (i) Gaussian blur, (j) Sobel filtering

Slide from Stanford CS231n (2026), Lecture 12, p. 73. Source: Chen et al., 2020. Figure: Chen et al. (2020), Fig. 4 (photo: CC BY, Von.grzanka).

B · Instance contrastive learning: SimCLR and MOCO

SimCLR

Generate a positive pair by sampling data augmentation functions

Iterate through and use each of the 2N sample as reference, compute average loss

SimCLR Algorithm 1: for each minibatch of N images, draw two augmentation functions per image, compute the representation h and projection z for both views, compute pairwise cosine similarities s(i,j) for all 2N views, define the loss l(i,j) as minus log of exp(s(i,j)/tau) over the sum of exp(s(i,k)/tau) for k not equal to i, average l over both orders of every positive pair, update f and g; return f and throw away g

InfoNCE loss:
Use all non-positive samples in the batch as x-

Slide from Stanford CS231n (2026), Lecture 12, pp. 74–76 (one build, final state). Source: Chen et al., 2020. Figure: Algorithm 1.

B · Instance contrastive learning: SimCLR and MOCO

Training linear classifier on SimCLR features

SimCLR Figure 1: ImageNet top-1 linear-evaluation accuracy against number of parameters; SimCLR 69.3 with 24M, SimCLR 2x 74.2, SimCLR 4x 76.5 with 375M, equal to the supervised ResNet-50 marker; other methods such as CPCv2-L, MoCo, CMC, PIRL, AMDIM, BigBiGAN, Rotation and InstDisc lie below

Train feature encoder on ImageNet (entire training set) using SimCLR.

Freeze feature encoder, train a linear classifier on top with labeled data.

Slide from Stanford CS231n (2026), Lecture 12, p. 77. Source: Chen et al., 2020. Figure: Chen et al. (2020), Fig. 1.

B · Instance contrastive learning: SimCLR and MOCO

Semi-supervised learning on SimCLR features

SimCLR Table 7: ImageNet top-5 accuracy with 1 percent and 10 percent of the labels; supervised ResNet-50 48.4 and 80.4; SimCLR ResNet-50 75.5 and 87.8, 2x 83.0 and 91.2, 4x 85.8 and 92.6, the best in the table

Train feature encoder on ImageNet (entire training set) using SimCLR.

Finetune the encoder with 1% / 10% of labeled data on ImageNet.

Slide from Stanford CS231n (2026), Lecture 12, p. 78. Source: Chen et al., 2020. Table: Chen et al. (2020), Table 7.

B · Instance contrastive learning: SimCLR and MOCO

SimCLR design choices: projection head

SimCLR Figure 8: top-1 linear accuracy against projection output dimension from 32 to 2048; a non-linear head scores about 64, a linear head about 60 to 61, no head about 50
SimCLR Figure 2 again, with the projection head g and the outputs z-i and z-j marked by a box

Linear / non-linear projection heads improve representation learning.

A possible explanation:

  • contrastive learning objective may discard useful information for downstream tasks
  • representation space z is trained to be invariant to data transformation.
  • by leveraging the projection head g(·), more information can be preserved in the h representation space

Slide from Stanford CS231n (2026), Lecture 12, p. 79. Source: Chen et al., 2020. Figures: Chen et al. (2020), Figs. 8 and 2.

B · Instance contrastive learning: SimCLR and MOCO

SimCLR design choices: large batch size

SimCLR Figure 9: top-1 linear accuracy of ResNet-50 for batch sizes 256 to 8192 and 100 to 1000 training epochs; larger batches score higher, most of all at 100 epochs (57.5 at batch 256 against about 64.7 at 8192); the gap shrinks with longer training. Caption: Linear evaluation models (ResNet-50) trained with different batch size and epochs. Each bar is a single run from scratch.

Large training batch size is crucial for SimCLR!

Large batch size causes large memory footprint during backpropagation: requires distributed training on TPUs (ImageNet experiments)

\[ MI[f(x), f(x^+)] - \log(N) \geq -L \]

Slide from Stanford CS231n (2026), Lecture 12, p. 80. Source: Chen et al., 2020. Figure: Chen et al. (2020), Fig. 9.

B · Instance contrastive learning: SimCLR and MOCO

Momentum Contrastive Learning (MoCo)

MoCo: x query goes through the encoder to q; x0 key, x1 key, x2 key go through the momentum encoder to the queue k0, k1, k2; similarity between q and the keys gives the contrastive loss; no gradient (no_grad) flows into the keys

Key differences to SimCLR:

  • Keep a running queue of keys (negative samples).
  • Compute gradients and update the encoder only through the queries.
  • Decouple min-batch size with the number of keys: can support a large number of negative samples.
  • The key encoder is slowly progressing through the momentum update rules: \[\theta_k \leftarrow m\theta_k + (1 - m)\theta_q\]

Slide from Stanford CS231n (2026), Lecture 12, pp. 81–82. One animation; its final state. Math retyped. Figure: He et al. (2020), Fig. 1.

B · Instance contrastive learning: SimCLR and MOCO

MoCo

Generate a positive pair by sampling data augmentation functions

No gradient through the key

Update the FIFO negative sample queue

# f_q, f_k: encoder networks for query and key# queue: dictionary as a queue of K keys (CxK)# m: momentum# t: temperature f_k.params = f_q.params # initializefor x in loader: # load a minibatch x with N samples    x_q = aug(x) # a randomly augmented version    x_k = aug(x) # another randomly augmented version     q = f_q.forward(x_q) # queries: NxC    k = f_k.forward(x_k) # keys: NxC    k = k.detach() # no gradient to keys     # positive logits: Nx1    l_pos = bmm(q.view(N,1,C), k.view(N,C,1))     # negative logits: NxK    l_neg = mm(q.view(N,C), queue.view(C,K))     # logits: Nx(1+K)    logits = cat([l_pos, l_neg], dim=1)     # contrastive loss, Eqn.(1)    labels = zeros(N) # positives are the 0-th    loss = CrossEntropyLoss(logits/t, labels)     # SGD update: query network    loss.backward()    update(f_q.params)     # momentum update: key network    f_k.params = m*f_k.params+(1-m)*f_q.params     # update dictionary    enqueue(queue, k) # enqueue the current minibatch    dequeue(queue) # dequeue the earliest minibatch

Use the running queue of keys as the negative samples

InfoNCE loss

Update f_k through momentum

Slide from Stanford CS231n (2026), Lecture 12, p. 83. The code is retyped from He et al. (2020), Algorithm 1; each label points to the line that it names.

B · Instance contrastive learning: SimCLR and MOCO

“MoCo V2”

Title: Improved Baselines with Momentum Contrastive Learning, by Xinlei Chen, Haoqi Fan, Ross Girshick and Kaiming He, Facebook AI Research (FAIR)

A hybrid of ideas from SimCLR and MoCo:

  • From SimCLR: non-linear projection head and strong data augmentation.
  • From MoCo: momentum-updated queues that allow training on a large number of negative samples (no TPU required!).

Slide from Stanford CS231n (2026), Lecture 12, p. 84. Paper: X. Chen et al. (2020).

B · Instance contrastive learning: SimCLR and MOCO

MoCo vs. SimCLR vs. MoCo V2

Table 1, ablation of MoCo baselines with ResNet-50: adding an MLP head, extra blur augmentation (aug+) and a cosine schedule raises ImageNet linear accuracy from 60.6 (MoCo v1) to 67.5 at 200 epochs and 71.1 at 800 epochs; the MLP and aug+ rows are boxed

Key takeaways:

  • Non-linear projection head and strong data augmentation are crucial for contrastive learning.

Slide from Stanford CS231n (2026), Lecture 12, p. 85. Table: X. Chen et al. (2020), Table 1.

B · Instance contrastive learning: SimCLR and MOCO

MoCo vs. SimCLR vs. MoCo V2

Table 2, MoCo vs SimCLR, ImageNet linear accuracy: SimCLR 61.9 (batch 256) and 66.6 (batch 8192) at 200 epochs; MoCo v2 67.5 (batch 256); with longer training, SimCLR 69.3 (1000 epochs, batch 4096) and MoCo v2 71.1 (800 epochs, batch 256)

Key takeaways:

  • Non-linear projection head and strong data augmentation are crucial for contrastive learning.
  • Decoupling mini-batch size with negative sample size allows MoCo-V2 to outperform SimCLR with smaller batch size (256 vs. 8192).

Slide from Stanford CS231n (2026), Lecture 12, p. 86. Table: X. Chen et al. (2020), Table 2.

B · Instance contrastive learning: SimCLR and MOCO

MoCo vs. SimCLR vs. MoCo V2

Table 3, memory and time on 8 V100 16G GPUs: MoCo, batch 256, 5.0G per GPU, 53 hours per 200 epochs; end-to-end, batch 256, 7.4G, 65 hours; end-to-end, batch 4096, 93.0G (estimated), n/a

Key takeaways:

  • Non-linear projection head and strong data augmentation are crucial for contrastive learning.
  • Decoupling mini-batch size with negative sample size allows MoCo-V2 to outperform SimCLR with smaller batch size (256 vs. 8192).
  • … all with much smaller memory footprint! (“end-to-end” means SimCLR here)

Slide from Stanford CS231n (2026), Lecture 12, p. 87. Table: X. Chen et al. (2020), Table 3.

Part B · Contrastive representation learning

Sequence contrastive learning: CPC

1Intuition and formulation
2Instance contrastive learning: SimCLR and MOCO
3Sequence contrastive learning: CPC
4Self-Distillation Without Labels, DINO

B · Sequence contrastive learning: CPC

Instance vs. Sequence Contrastive Learning

The cat picture with its transformed versions (attract) and the puppy (repel)

Instance-level contrastive learning:

contrastive learning based on positive & negative instances.
Examples: SimCLR, MoCo

CPC on an audio sequence: encoders g_enc map each time step x to z; an autoregressive model g_ar gives the context c_t, which predicts z at t+1 to t+4

Sequence-level contrastive learning:

contrastive learning based on sequential / temporal orders.
Example: Contrastive Predictive Coding (CPC)

Slide from Stanford CS231n (2026), Lecture 12, p. 88. Right figure: van den Oord et al. (2018), Fig. 1.

B · Sequence contrastive learning: CPC

Contrastive Predictive Coding (CPC)

CPC: a sequence of image patches; the context patches (handwritten digits 2, 3, 4, 5) are encoded and summarized; the model must pick the true future patches (positive: 6, 7, 8, 9) over other patches (negative)

Contrastive: contrast between “right” and “wrong” sequences using contrastive learning.

Predictive: the model has to predict future patterns given the current context.

Coding: the model learns useful feature vectors, or “code”, for downstream tasks, similar to other self-supervised methods.

Slide from Stanford CS231n (2026), Lecture 12, p. 89. Figure: van den Oord et al. (2018); the deck adds a “Figure source” link.

B · Sequence contrastive learning: CPC

Contrastive Predictive Coding (CPC)

The CPC figure with the context code c_t and its predictions of z at t+1 to t+4 boxed
  1. Encode all samples in a sequence into vectors \(z_t = g_{enc}(x_t)\)
  2. Summarize context (e.g., half of a sequence) into a context code \(c_t\) using an auto-regressive model \((g_{ar})\). The original paper uses GRU-RNN here.
  3. Compute InfoNCE loss between the context \(c_t\) and future code \(z_{t+k}\) using the following time-dependent score function: \[s_k(z_{t+k}, c_t) = z_{t+k}^T W_k c_t\] where \(W_k\) is a trainable matrix.

Slide from Stanford CS231n (2026), Lecture 12, pp. 90–92. One animation; its final state (“The original paper uses GRU-RNN here.” is on p. 91 only). Math retyped. Figure: van den Oord et al. (2018).

B · Sequence contrastive learning: CPC

CPC example: modeling audio sequences

CPC on raw audio: the waveform is cut into time steps x; g_enc encodes each step into z; g_ar summarizes the past into c_t, which predicts z at t+1 to t+4

Slide from Stanford CS231n (2026), Lecture 12, p. 93. Figure: van den Oord et al. (2018), Fig. 1.

B · Sequence contrastive learning: CPC

CPC example: modeling audio sequences

Left: t-SNE of CPC audio representations for 10 speakers; each colour is one speaker. Right: linear classification on LibriSpeech; phone classification: random 27.6, MFCC 39.7, CPC 64.6, supervised 74.6; speaker classification: random 1.87, MFCC 17.6, CPC 97.4, supervised 98.5

Slide from Stanford CS231n (2026), Lecture 12, p. 94. Figure and table: van den Oord et al. (2018), Fig. 2 and Table 1.

B · Sequence contrastive learning: CPC

CPC example: modeling visual context

Idea: split image into patches, model rows of patches from top to bottom as a sequence. I.e., use top rows as context to predict bottom rows.

A 256 by 256 pixel input image is cut into 64 by 64 pixel patches with 50% overlap; g_enc encodes each patch; g_ar summarizes the rows above into c_t, which predicts the patch codes z at t+2, t+3 and t+4 in the rows below

Slide from Stanford CS231n (2026), Lecture 12, p. 95. Figure: van den Oord et al. (2018), Fig. 4.

Part B · Contrastive representation learning

Self-Distillation Without Labels, DINO

1Intuition and formulation
2Instance contrastive learning: SimCLR and MOCO
3Sequence contrastive learning: CPC
4Self-Distillation Without Labels, DINO

B · Self-Distillation Without Labels, DINO

DINO: Self-Distillation with No Labels

Title of Emerging Properties in Self-Supervised Vision Transformers (Caron, Touvron, Misra, Jégou, Mairal, Bojanowski, Joulin) and its Figure 1: self-attention maps of the CLS token of a ViT trained with no supervision outline the objects in each photo

Slide from Stanford CS231n (2026), Lecture 12, p. 96. Figure: Caron et al. (2021), Fig. 1.

B · Self-Distillation Without Labels, DINO

DINO

Left: DINO Figure 2: an image x gives two views x1 and x2; the student and the teacher, both g, output p1 and p2 through a softmax; the teacher output is centred and gets no gradient (sg); the loss is minus p2 log p1; the teacher is an exponential moving average (ema) of the student. Right: a diagram of the same steps with centering and sharpening of the teacher output, backprop into the student only, and the update theta teacher = tau times theta teacher + (1 - tau) times theta student

Teacher update (EMA): \(\theta_{teacher} \leftarrow \tau\,\theta_{teacher} + (1-\tau)\,\theta_{student}\). At each step the teacher weights move slowly toward the student weights. \(\tau\) is close to 1 (0.996). Only the student is trained.

Why: the teacher is an average of past students, so its targets are steadier and better than the student's own. Copying the student weights into the teacher fails to converge.

B · Self-Distillation Without Labels, DINO

DINO

DINO Algorithm 1, PyTorch pseudocode without multi-crop: the student gs and the teacher gt get two augmented views; loss = H(t1, s2)/2 + H(t2, s1)/2; the student is updated by SGD; the teacher parameters are a moving average of the student's; the centre C is a moving average of the teacher outputs; H stops the gradient into t, sharpens t with (t - C)/tpt and the softmax, and returns the cross-entropy

Slide from Stanford CS231n (2026), Lecture 12, p. 98. Figure: Caron et al. (2021), Algorithm 1.

B · Self-Distillation Without Labels, DINO

DINO v2

DINOv2 Figure 1: visualization of the first three PCA components of patch features, shown as colours, for birds and planes, elephants, horses, and vehicles; the same parts get the same colours across images

Slide from Stanford CS231n (2026), Lecture 12, p. 99. The deck credits Caron et al. (2021); the figure is Oquab et al. (2024), Fig. 1 (DINOv2).

B · Self-Distillation Without Labels, DINO

Summary: Contrastive Representation Learning

A general formulation for contrastive learning:

\[\text{score}(f(x), f(x^+)) >> \text{score}(f(x), f(x^-))\]

InfoNCE loss: N-way classification among positive and negative samples

\[L = -\mathbb{E}_X \left[\log \frac{\exp(s(f(x), f(x^+))}{\exp(s(f(x), f(x^+)) + \sum_{j=1}^{N-1} \exp(s(f(x), f(x_j^-))}\right]\]

Commonly known as the InfoNCE loss (van den Oord et al., 2018)
A lower bound on the mutual information between f(x) and f(x+)

\[MI[f(x), f(x^+)] - \log(N) \ge -L\]

Slide from Stanford CS231n (2026), Lecture 12, p. 100. Math retyped from the slide.

B · Self-Distillation Without Labels, DINO

Summary: Contrastive Representation Learning

SimCLR: a simple framework for contrastive representation learning

  • Key ideas: non-linear projection head to allow flexible representation learning
  • Simple to implement, effective in learning visual representation
  • Requires large training batch size to be effective; large memory footprint
SimCLR framework: two augmentations t and t' of an image x give x-tilde i and x-tilde j; an encoder f gives the representations h i and h j; a projection head g gives z i and z j; training maximizes the agreement of z i and z j

Slide from Stanford CS231n (2026), Lecture 12, p. 101. Figure: T. Chen et al. (2020), Fig. 2.

B · Self-Distillation Without Labels, DINO

Summary: Contrastive Representation Learning

MoCo (v1, v2): contrastive learning using momentum sample encoder

  • Decouples negative sample size from minibatch size; allows large batch training without TPU
  • MoCo-v2 combines the key ideas from SimCLR, i.e., nonlinear projection head, strong data augmentation, with momentum contrastive learning
The MoCo diagram: encoder for the query, momentum encoder for the queue of keys, similarity and contrastive loss

Slide from Stanford CS231n (2026), Lecture 12, p. 102. Figure: He et al. (2020), Fig. 1.

B · Self-Distillation Without Labels, DINO

Summary: Contrastive Representation Learning

CPC: sequence-level contrastive learning

  • Contrast “right” sequence with “wrong” sequence.
  • InfoNCE loss with a time-dependent score function.
  • Can be applied to a variety of learning problems, but not as effective in learning image representations compared to instance-level methods.
The CPC figure with context, positive and negative patches

Slide from Stanford CS231n (2026), Lecture 12, p. 103. Figure: van den Oord et al. (2018).

Part B

Part B: key methods and insights

BREAK
10 minutesBack at 19:15

Coming up in Part C: today's best vision encoders, and how to choose one.

Part C

Advanced topics and today's state of the art

Today's best encoders combine the three methods, at scale. Which one to use depends on the task.

C

What do we want from a vision foundation model?

  • 1. General purpose: one frozen backbone, many tasks, no per-task finetuning
    • “Foundation model” in vision = features good enough that a linear head is competitive
  • 2. Global features: one vector per image
    • Classification, retrieval, image–text matching. This is what a [CLS] token gives you.
  • 3. Dense features: one vector per patch
    • Segmentation, depth, correspondence, tracking, grounding
  • 4. Scales: gets better with more data and compute, without more annotation
  • 5. A token interface a language model can consume
    • A grid of h×w patch embeddings is already a sequence
  • Theme for the rest of the lecture: global quality and dense quality are surprisingly decoupled
    • A model can get better at ImageNet while getting worse at segmentation.

Slide from Georgia Tech CS 8803-VLM (2026), Lecture 3, p. 41.

C

Three families of self-supervised objectives

FamilyIdeaExamplesStrong atWeak at
Contrastive / joint embeddingTwo augmented views of the same image attract; different images repelSimCLR, MoCoGlobal, linear-probe semanticsNeeds negatives & large batches; dense features mediocre
Masked image modeling (MIM)Hide most patches, reconstruct the missing pixels / tokensBEiT, MAE, iBOTScalable pretraining; strong after finetuningFrozen / linear-probe features are weak
Self-distillation (no negatives)Student matches the output distribution of an EMA teacher across viewsBYOL, DINO, DINOv2/v3Frozen features; emergent object structureCollapse must be prevented explicitly

DINOv2 / DINOv3 deliberately combine the last two: an image-level self-distillation loss for global semantics + a patch-level masked loss for dense features.

Slide from Georgia Tech CS 8803-VLM (2026), Lecture 3, p. 42.

C

The problem: dense features degrade during long training

DINOv3 Figure 6 panels: a photo of sheep on grass with a red cross on one patch (Image, red x = query), then cosine-similarity maps at 200k, 400k, 600k, 800k and 1M iterations; the bright area around the sheep gets noisier and spreads to many unrelated patches as training goes on
  • Each map: cosine similarity between the patch marked with a red cross and every other patch, at increasing training iterations
  • Global metrics (ImageNet linear probe) keep improving the whole time
    • Patch-level maps get noisier and less localized
  • Global and dense performance are decoupled: the global objective progressively dominates, and the patch-level iBOT loss cannot hold the balance
  • This is why “just train longer / bigger” had stopped paying off for dense tasks

Slide from Georgia Tech CS 8803-VLM (2026), Lecture 3, p. 48. Figure: Siméoni et al. (2025), Fig. 6; panels cropped from the deck page.

C

The fix: Gram anchoring

  • Idea: regularize the structure of patch-to-patch similarities
  • Let \(X_s\), \(X_G\) be the \(P{\times}d\) matrices of L2-normalized patch features of the student and of a Gram teacher — an earlier EMA checkpoint whose dense maps were still clean
    \(L_{\text{Gram}} = \lVert X_s X_s^{\top} - X_G X_G^{\top} \rVert_F^2\)
  • Because the loss only constrains the Gram matrix, individual features are free to keep moving
  • Applied as a refinement stage after 1M iterations; the Gram teacher is refreshed every 10k iterations
    • Remarkably, it repairs already-degraded maps rather than merely preventing the damage
  • \(L_{\text{Ref}} = w_D\cdot L_{\text{DINO}} + L_{\text{iBOT}} + w_{DK}\cdot L_{\text{DKoleo}} + w_{\text{Gram}}\cdot L_{\text{Gram}}\)
  • ADE20k dense linear probe: 50.3 → 55.7 mIoU; NYU depth RMSE 0.307 → 0.281, with ImageNet linear unchanged
DINOv3 Figure 10 panels: two photos (a yellow flower; geese on a path) with their patch-similarity maps WITHOUT Gram anchoring (noisy, scattered) and WITH Gram anchoring (smooth, following the object)

Slide from Georgia Tech CS 8803-VLM (2026), Lecture 3, p. 49. Page cites Siméoni et al. (2025), §4.2 and Fig. 10. Panels cropped; formulas retyped.

C

Frozen backbone, no finetuning: where each supervision wins

BackboneSupervisionADE20k
mIoU ↑
NYUv2 depth
RMSE ↓
DAVIS track
J&F ↑
IN1k linear
top-1 ↑
SigLIP 2  g/16image–text (weak)42.70.49462.389.1
PE core  G/14image–text (weak)38.90.59053.189.3
DINOv2  g/14self-supervised49.50.37273.687.3
PE spatial  G/14image–text + distill49.30.36274.5—
DINOv3  7B/16self-supervised55.90.30979.788.4
  • Language-supervised encoders lead on global, nameable semantics (ImageNet linear)
  • Self-supervised encoders lead on everything dense and spatial
    • Segmentation, depth, tracking — often by a very wide margin
  • DINOv3 extends the dense lead while nearly closing the global gap

Slide from Georgia Tech CS 8803-VLM (2026), Lecture 3, p. 51. Page note: Siméoni et al., DINOv3, 2025, Tabs. 3, 5, 7. Linear probes on frozen features; DAVIS at medium resolution.

C

Each method puts its cost in a different place

MethodData processingCompute
MAEImageNet; a random crop is enoughEncoder reads 25% of patches. ViT-L: 31 h, 128 TPU v3 cores
SimCLR, MoCoCrop and colour change are necessaryMany negatives: batch 4,096, or a queue of 65,536
CLIP400M web image–text pairsBatch 32,768. ViT-L/14: 12 days, 256 V100 GPUs
DINO, DINOv2Multi-crop; 1.2B web images curated to 142MTwo networks. DINOv2 g/14: 22,016 A100 GPU-hours

Each paper's own set-up, so not a controlled comparison. MAE: Table 1e, §4.2 (1,600 epochs). SimCLR: Fig. 5, §2.2. MoCo: §4. CLIP: §2.2, §2.5. DINO: §3.2. DINOv2: §3, Table 14.

C

Adaptation: train none, some or all of the weights

Zero-shot

No labels. Image–text models: CLIP, SigLIP 2, or DINOv3 with dino.txt.

Frozen + k-NN or linear head

Few labels, little compute. Needs good features as they are: DINOv3, CLIP.

LoRA

Train small low-rank updates; keep the weights frozen. Lecture 1·02, slides 149–158.

Full fine-tuning

Most labels and compute. MAE is strong here.

The test can change the winner. ViT-L, frozen: MoCo v3 77.6, MAE 73.5. Tune one block: MAE 81.0, MoCo v3 79.9.

⚖️ LO5 No labels does not mean no data problems. Nobody checks billions of web images for consent or bias. DINOv2 removed unsafe images and blurred faces first.

MAE against MoCo v3 (Chen et al., 2021): He et al. (2022), Fig. 9, values read from the plot labels. Zero-shot: Radford et al. (2021); k-NN: Caron et al. (2021); LoRA: Hu et al. (2022). Face blurring: Oquab et al. (2024), §3.

Part C

Part C: key methods and insights

Close · Summary

Three methods: what to predict, and what stops the cheat

MethodWhat the model predictsHow it can cheatWhat stops the cheat
MAEThe hidden image patchesCopy from the neighbouring patchesHide 75% of the patches
SimCLRWhich other view belongs to this imageGive every image the same vectorThe other images in the batch (negatives)
DINOThe teacher's output for another view of the imageThe same output for every imageCentring, sharpening, and a slow teacher

He et al. (2022); Chen et al. (2020); Caron et al. (2021).

Where we got to

Without labels, what should a model predict?

Answered today

Three answers, each with one design choice that stops the cheat: predict the hidden patches (MAE: hide most of them); match another view of the same image (SimCLR: the other images in the batch are negatives); match a slow teacher (DINO: centring and sharpening).

Still open

Captions help with the names of things; images alone give better features for each patch. Today's encoders combine both. Next week: connect an encoder to an LLM.

What comes next · Wed 4 Nov

Next week: Lecture 4, large multi-modal models

The question for next week

CLIP can rank labels you give it, but it cannot answer a question about the image. How do we connect its image encoder to an LLM?

← → or PageUp/PageDown · F full screen · O overview · B black screen