Session 01 of 1 · Learning from images without labels
CEG5305 Introduction to Generative AI with Foundation Models
Junyuan Hong · AY2026/27 Semester 1
Before we start
These slides were prepared with the help of AI tools (Claude, by Anthropic), working from the reference slides and papers listed here.
MAE (He et al., 2022) · SimCLR (Chen et al., 2020) · CLIP (Radford et al., 2021) · DINO (Caron et al., 2021)
Recap · Lectures 1 and 2
| Lecture | Title | Key message |
|---|---|---|
| 1 · 01 | Architecture — what an LLM is made of | Architecture or scale: why do LLMs work so well? Transformer · attention · RNN · nanoGPT |
| 1 · 02 | Training — how the model is actually trained | How does scale make LLMs work well? GPU parallelism · scaling laws · PEFT (LoRA) |
| 2 · 01 | Post-training — from a text predictor to an assistant | What turns GPT into ChatGPT? SFT · RLHF · DPO · GRPO |
| 2 · 02 | Prompting — from one call to a loop | How far can a fixed model go? ICL · chain-of-thought · agent · harness |
Lectures 1 and 2 were about text. Today: images, and no labels.
Outline · Lecture 3 = Topic 6, three hours
Recap · Lecture 1
But how? What is the general principle that we learned from LLM training?
Examples: eagle, USFWS; Armstrong, Apollo 11, NASA; Muybridge (1878); all public domain. DiT sample, facebookresearch/DiT (CC BY-NC 4.0). Ubiquitin, PDB 1UBQ.
Recap · Lecture 1
Prediction target = a part of the input. The text gives its own label
Recap · Lecture 1
| Supervised learning | Pre-training (self-supervised) | |
|---|---|---|
| Data | Pairs: an instruction and the answer a person wrote | Raw text, nothing added |
| Label | Given by people, from outside the data | Cut from the data: the next word |
| Cost of more data | Each label needs a person's time | Any text on the web works |
| What the model learns | One behaviour: follow instructions | The structure of language, for many tasks |
Supervised: the label comes from people. Self-supervised: the label comes from the data.
Today's question
What is the general idea of self-supervised learning?
When is it the right method?
Does it work for other modalities, such as images?
Part A
A · Self-Supervised Learning
Predict the category from a given image.
Pairs of an image x and a category y that people wrote. Example: ImageNet, 1,281,167 training images in 1,000 categories.
Give the correct category y a high probability p(y | x): minimise the cross-entropy loss, −log p(y | x).
Cat photo: Stanford CS231n (2026), Lecture 12, p. 6, where the image is marked “CC0 public domain”. The scores 0.9 and 0.1 are an illustration.
A · Self-Supervised Learning

Slide from Stanford CS231n (2026), Lecture 12, p. 9. (p. 8 shows the same figure under the title “Last Week: Visualizing and Understanding”.) Figure: Krizhevsky et al. (2012).
A · Self-Supervised Learning

What is the problem with large-scale training?
- We need a lot of labeled data
Is there a way we can train neural networks without the need for huge manually labeled datasets?
Slide from Stanford CS231n (2026), Lecture 12, p. 10. Figure: Krizhevsky et al. (2012); the deck adds “Figures reproduced with permission”.
A · Self-Supervised Learning

1. Pretext Task
2. Downstream Task
Slide from Stanford CS231n (2026), Lecture 12, p. 11.
A · Self-Supervised Learning

Slide from Stanford CS231n (2026), Lecture 12, p. 12.
A · Self-Supervised Learning

Slide from Stanford CS231n (2026), Lecture 12, p. 13.
A · Self-Supervised Learning
Example: learn to predict image transformations / complete corrupted images

Slide from Stanford CS231n (2026), Lecture 12, p. 14.
A · Self-Supervised Learning
Slide from Stanford CS231n (2026), Lecture 12, p. 15. The deck shows this list in one column.
A · Self-Supervised Learning

1. Learn good feature extractors from self-supervised pretext tasks, e.g., predicting image rotations
2. Attach a shallow network on the feature extractor; train the shallow network on the target task with small amount of labeled data
Slide from Stanford CS231n (2026), Lecture 12, pp. 16–17. One animation; its final state.
A · Self-Supervised Learning
Today’s lecture

Slide from Stanford CS231n (2026), Lecture 12, p. 18. Works shown: Doersch et al. (2015); OpenAI (2023); van den Oord et al. (2016); Florence et al. (2018).
Part A · How learning gets rid of labels
A · Pretext tasks from image transformations
Guide your model to understand visual semantics (representations of meanings)
A · Pretext tasks from image transformations

Hypothesis: a model could recognize the correct rotation of an object only if it has the “visual commonsense” of what the object should look like unperturbed.
Slide from Stanford CS231n (2026), Lecture 12, p. 21. Figure: Gidaris et al. (2018).
A · Pretext tasks from image transformations

Self-supervised learning by rotating the entire input images.
The model learns to predict which rotation is applied (4-way classification)
Slide from Stanford CS231n (2026), Lecture 12, pp. 22–23. One animation; its final state. Figure: Gidaris et al. (2018).
A · Pretext tasks from image transformations

Self-supervised learning on CIFAR10 (entire training set).
Freeze conv1 + conv2
Learn conv3 + linear layers with subset of labeled CIFAR10 data (classification).
Slide from Stanford CS231n (2026), Lecture 12, p. 24. Figure: Gidaris et al. (2018).
A · Pretext tasks from image transformations

Full supervised learning
No pretraining
Self-supervised learning on ImageNet (entire training set [1,281,167 images]) with AlexNet.
Finetune on labeled data from Pascal VOC 2007 [trainval: 5,011 images].
Slide from Stanford CS231n (2026), Lecture 12, p. 25. Figure: Gidaris et al. (2018). Image counts in brackets (added): ILSVRC 2012 training set, image-net.org; VOC 2007 trainval, Ultralytics VOC dataset page. Retrieved 3 Nov 2026.
A · Pretext tasks from image transformations

Self-supervised learned attention has better interpretation.
A · Pretext tasks from image transformations

Slide from Stanford CS231n (2026), Lecture 12, p. 27. Figure: Doersch et al. (2015).
A · Pretext tasks from image transformations

The model predicts the arrangement of the shuffled tiles (the index of the permutation -- learn the relation of patches in the image), not the object category.
Slide from Stanford CS231n (2026), Lecture 12, p. 28. Figure: Noroozi & Favaro (2016), Fig. 3
A · Pretext tasks from image transformations

Context Encoders: Feature Learning by Inpainting (Pathak et al., 2016)
Slide from Stanford CS231n (2026), Lecture 12, p. 30. Figure: Pathak et al. (2016).
A · Pretext tasks from image transformations
Auto Encoder

Learning to reconstruct the missing pixels
Slide from Stanford CS231n (2026), Lecture 12, p. 31. Figure: Pathak et al. (2016).
A · Pretext tasks from image transformations

Slide from Stanford CS231n (2026), Lecture 12, p. 32. Figure: Pathak et al. (2016).
A · Pretext tasks from image transformations
(We will talk about adversarial learning in the next lecture)
Loss = reconstruction + adversarial learning
\(*\): Element wise multiplication
\(M = \begin{cases} 0 & \text{not masked} \\ 1 & \text{masked} \end{cases}\)
\(F_\theta\): Encoder
Added: \(x\) is the input image. \(F_\theta\) maps the image with a hole, \((1 - M) * x\), to a full image: image → image. It is the network that we train.
Added: \(D\) is a second network, the discriminator. It takes an image and outputs the probability that the image is real, not inpainted.
Adversarial loss between “real” images and inpainted images
Slide from Stanford CS231n (2026), Lecture 12, p. 33. Source: Pathak et al. (2016). The two blue notes on \(F_\theta\) and \(D\) are added.
A · Pretext tasks from image transformations

Slide from Stanford CS231n (2026), Lecture 12, p. 34. Figure: Pathak et al. (2016).
A · Pretext tasks from image transformations

Self-supervised learning on ImageNet training set, transfer to classification (Pascal VOC 2007), detection (Pascal VOC 2007), and semantic segmentation (Pascal VOC 2012)
Slide from Stanford CS231n (2026), Lecture 12, p. 35. Table: Pathak et al. (2016).
A · Pretext tasks from image transformations

Slide from Stanford CS231n (2026), Lecture 12, p. 36. The deck credits “Richard Zhang / Phillip Isola”. See Zhang et al. (2016).
A · Pretext tasks from image transformations

Slide from Stanford CS231n (2026), Lecture 12, p. 37. The deck credits “Richard Zhang / Phillip Isola”. See Zhang et al. (2016).
A · Pretext tasks from image transformations

F1 sees only the grayscale channel and predicts the colour channels. F2 sees only the colour channels and predicts the grayscale channel. Each network predicts what the other one sees.
Slide from Stanford CS231n (2026), Lecture 12, p. 38. The deck credits “Richard Zhang / Phillip Isola”. Figure: Zhang et al. (2017). The caption under the figure is added.
A · Pretext tasks from image transformations
Idea: cross-channel predictions

Slide from Stanford CS231n (2026), Lecture 12, p. 39. The deck credits “Richard Zhang / Phillip Isola”. Figure: Zhang et al. (2017).
A · Pretext tasks from image transformations

Self-supervised learning on ImageNet (entire training set).
Use concatenated features from F₁ and F₂
Labeled data is from the Places (Zhou 2016).
Slide from Stanford CS231n (2026), Lecture 12, p. 40. Figure: Zhang et al. (2017). Places: Zhou et al. (2016).
A · Pretext tasks from image transformations

Slide from Stanford CS231n (2026), Lecture 12, p. 41. The deck credits “Richard Zhang / Phillip Isola”. See Zhang et al. (2016).
A · Pretext tasks from image transformations

Slide from Stanford CS231n (2026), Lecture 12, p. 42. The deck credits “Richard Zhang / Phillip Isola”. See Zhang et al. (2016).
A · Pretext tasks from image transformations
Scaling inpainting idea: Reconstruction with a larger masked portion

Slide from Stanford CS231n (2026), Lecture 12, p. 48. Figure: He et al. (2022)
A · Pretext tasks from image transformations

Slide from Stanford CS231n (2026), Lecture 12, p. 49. Figure: He et al. (2022), Fig. 1.
A · Pretext tasks from image transformations

Slide from Stanford CS231n (2026), Lecture 12, p. 50. Figure: He et al. (2022), Fig. 1 (the page labels it “Figure 6”).
A · Pretext tasks from image transformations

Slide from Stanford CS231n (2026), Lecture 12, p. 51. Figure: He et al. (2022), Fig. 1 (the page labels it “Figure 6”).
A · Pretext tasks from image transformations

Slide from Stanford CS231n (2026), Lecture 12, p. 52. Figure: He et al. (2022), Fig. 1 (the page labels it “Figure 6”).
A · Pretext tasks from image transformations
Slide from Stanford CS231n (2026), Lecture 12, p. 53.
A · Pretext tasks from image transformations

Slide from Stanford CS231n (2026), Lecture 12, p. 54; diagrams cropped from the same page.
A · Pretext tasks from image transformations
So many modeling/

Slide from Stanford CS231n (2026), Lecture 12, p. 55 (first of two slides). Figure: He et al. (2022), Fig. 5 (ViT-L).
A · Pretext tasks from image transformations


Slide from Stanford CS231n (2026), Lecture 12, p. 55 (second of two slides). Figures: He et al. (2022), Table 1f and Fig. 6 (ViT-L; ft: fine-tuning, lin: linear probing).
A · Pretext tasks from image transformations


Slide from Stanford CS231n (2026), Lecture 12, p. 56. Figures: He et al. (2022), Fig. 2 (first column; 80% masked) and Table 3.
A · Pretext tasks from image transformations
Slide from Stanford CS231n (2026), Lecture 12, pp. 57–58. One animation; its final state.
Part A · How learning gets rid of labels
A · Contrastive representation learning

Learned representations may be tied to a specific pretext task!
Can we come up with a more general pretext task?
Slide from Stanford CS231n (2026), Lecture 12, p. 59.
A · Contrastive representation learning

Slide from Stanford CS231n (2026), Lecture 12, pp. 60–61. One animation; its final state.
A · Contrastive representation learning

Slide from Stanford CS231n (2026), Lecture 12, pp. 62, 64. Page 64 repeats page 62 after the deck’s agenda page.
Part A
Coming up in Part B: contrastive representation learning, from the formulation to DINO.
Part B
B · Intuition and formulation

Slide from Stanford CS231n (2026), Lecture 12, p. 65.
B · Intuition and formulation
What we want:
x: reference sample; x+ positive sample; x- negative sample
Given a chosen score function, we aim to learn an encoder function f that yields high score for positive pairs (x, x+) and low scores for negative pairs (x, x-).
Slide from Stanford CS231n (2026), Lecture 12, p. 66.
B · Intuition and formulation
Loss function given 1 positive sample and N - 1 negative samples:
This seems familiar …
Cross entropy loss for a N-way softmax classifier!
I.e., learn to find the positive sample from the N samples
Slide from Stanford CS231n (2026), Lecture 12, pp. 67–70 (one build, final state). Photos cropped from p. 68.
B · Intuition and formulation
Loss function given 1 positive sample and N - 1 negative samples:
Commonly known as the InfoNCE loss (van den Oord et al., 2018)
A lower bound on the mutual information between f(x) and f(x+)
The larger the negative sample size (N), the tighter the bound
Detailed derivation: Poole et al., 2019
Slide from Stanford CS231n (2026), Lecture 12, p. 71.
Part B · Contrastive representation learning
B · Instance contrastive learning: SimCLR and MOCO
Cosine similarity as the score function:
Use a projection network g(·) to project features to a space where contrastive learning is applied
Generate positive samples through data augmentation:

Slide from Stanford CS231n (2026), Lecture 12, p. 72. Source: Chen et al., 2020. Figure: Chen et al. (2020), Fig. 2.
B · Instance contrastive learning: SimCLR and MOCO

Slide from Stanford CS231n (2026), Lecture 12, p. 73. Source: Chen et al., 2020. Figure: Chen et al. (2020), Fig. 4 (photo: CC BY, Von.grzanka).
B · Instance contrastive learning: SimCLR and MOCO
Generate a positive pair by sampling data augmentation functions
Iterate through and use each of the 2N sample as reference, compute average loss
InfoNCE loss:
Use all non-positive samples in the batch as x-
Slide from Stanford CS231n (2026), Lecture 12, pp. 74–76 (one build, final state). Source: Chen et al., 2020. Figure: Algorithm 1.
B · Instance contrastive learning: SimCLR and MOCO

Train feature encoder on ImageNet (entire training set) using SimCLR.
Freeze feature encoder, train a linear classifier on top with labeled data.
Slide from Stanford CS231n (2026), Lecture 12, p. 77. Source: Chen et al., 2020. Figure: Chen et al. (2020), Fig. 1.
B · Instance contrastive learning: SimCLR and MOCO

Train feature encoder on ImageNet (entire training set) using SimCLR.
Finetune the encoder with 1% / 10% of labeled data on ImageNet.
Slide from Stanford CS231n (2026), Lecture 12, p. 78. Source: Chen et al., 2020. Table: Chen et al. (2020), Table 7.
B · Instance contrastive learning: SimCLR and MOCO


Linear / non-linear projection heads improve representation learning.
A possible explanation:
Slide from Stanford CS231n (2026), Lecture 12, p. 79. Source: Chen et al., 2020. Figures: Chen et al. (2020), Figs. 8 and 2.
B · Instance contrastive learning: SimCLR and MOCO

Large training batch size is crucial for SimCLR!
Large batch size causes large memory footprint during backpropagation: requires distributed training on TPUs (ImageNet experiments)
Slide from Stanford CS231n (2026), Lecture 12, p. 80. Source: Chen et al., 2020. Figure: Chen et al. (2020), Fig. 9.
B · Instance contrastive learning: SimCLR and MOCO

Key differences to SimCLR:
Slide from Stanford CS231n (2026), Lecture 12, pp. 81–82. One animation; its final state. Math retyped. Figure: He et al. (2020), Fig. 1.
B · Instance contrastive learning: SimCLR and MOCO
Generate a positive pair by sampling data augmentation functions
No gradient through the key
Update the FIFO negative sample queue
# f_q, f_k: encoder networks for query and key# queue: dictionary as a queue of K keys (CxK)# m: momentum# t: temperature f_k.params = f_q.params # initializefor x in loader: # load a minibatch x with N samples x_q = aug(x) # a randomly augmented version x_k = aug(x) # another randomly augmented version q = f_q.forward(x_q) # queries: NxC k = f_k.forward(x_k) # keys: NxC k = k.detach() # no gradient to keys # positive logits: Nx1 l_pos = bmm(q.view(N,1,C), k.view(N,C,1)) # negative logits: NxK l_neg = mm(q.view(N,C), queue.view(C,K)) # logits: Nx(1+K) logits = cat([l_pos, l_neg], dim=1) # contrastive loss, Eqn.(1) labels = zeros(N) # positives are the 0-th loss = CrossEntropyLoss(logits/t, labels) # SGD update: query network loss.backward() update(f_q.params) # momentum update: key network f_k.params = m*f_k.params+(1-m)*f_q.params # update dictionary enqueue(queue, k) # enqueue the current minibatch dequeue(queue) # dequeue the earliest minibatch
Use the running queue of keys as the negative samples
InfoNCE loss
Update f_k through momentum
Slide from Stanford CS231n (2026), Lecture 12, p. 83. The code is retyped from He et al. (2020), Algorithm 1; each label points to the line that it names.
B · Instance contrastive learning: SimCLR and MOCO

A hybrid of ideas from SimCLR and MoCo:
Slide from Stanford CS231n (2026), Lecture 12, p. 84. Paper: X. Chen et al. (2020).
B · Instance contrastive learning: SimCLR and MOCO

Key takeaways:
Slide from Stanford CS231n (2026), Lecture 12, p. 85. Table: X. Chen et al. (2020), Table 1.
B · Instance contrastive learning: SimCLR and MOCO

Key takeaways:
Slide from Stanford CS231n (2026), Lecture 12, p. 86. Table: X. Chen et al. (2020), Table 2.
B · Instance contrastive learning: SimCLR and MOCO

Key takeaways:
Slide from Stanford CS231n (2026), Lecture 12, p. 87. Table: X. Chen et al. (2020), Table 3.
Part B · Contrastive representation learning
B · Sequence contrastive learning: CPC

Instance-level contrastive learning:
contrastive learning based on positive & negative instances.
Examples: SimCLR, MoCo

Sequence-level contrastive learning:
contrastive learning based on sequential / temporal orders.
Example: Contrastive Predictive Coding (CPC)
Slide from Stanford CS231n (2026), Lecture 12, p. 88. Right figure: van den Oord et al. (2018), Fig. 1.
B · Sequence contrastive learning: CPC

Contrastive: contrast between “right” and “wrong” sequences using contrastive learning.
Predictive: the model has to predict future patterns given the current context.
Coding: the model learns useful feature vectors, or “code”, for downstream tasks, similar to other self-supervised methods.
Slide from Stanford CS231n (2026), Lecture 12, p. 89. Figure: van den Oord et al. (2018); the deck adds a “Figure source” link.
B · Sequence contrastive learning: CPC

Slide from Stanford CS231n (2026), Lecture 12, pp. 90–92. One animation; its final state (“The original paper uses GRU-RNN here.” is on p. 91 only). Math retyped. Figure: van den Oord et al. (2018).
B · Sequence contrastive learning: CPC

Slide from Stanford CS231n (2026), Lecture 12, p. 93. Figure: van den Oord et al. (2018), Fig. 1.
B · Sequence contrastive learning: CPC

Slide from Stanford CS231n (2026), Lecture 12, p. 94. Figure and table: van den Oord et al. (2018), Fig. 2 and Table 1.
B · Sequence contrastive learning: CPC
Idea: split image into patches, model rows of patches from top to bottom as a sequence. I.e., use top rows as context to predict bottom rows.

Slide from Stanford CS231n (2026), Lecture 12, p. 95. Figure: van den Oord et al. (2018), Fig. 4.
Part B · Contrastive representation learning
B · Self-Distillation Without Labels, DINO

Slide from Stanford CS231n (2026), Lecture 12, p. 96. Figure: Caron et al. (2021), Fig. 1.
B · Self-Distillation Without Labels, DINO

Teacher update (EMA): \(\theta_{teacher} \leftarrow \tau\,\theta_{teacher} + (1-\tau)\,\theta_{student}\). At each step the teacher weights move slowly toward the student weights. \(\tau\) is close to 1 (0.996). Only the student is trained.
Why: the teacher is an average of past students, so its targets are steadier and better than the student's own. Copying the student weights into the teacher fails to converge.
B · Self-Distillation Without Labels, DINO

Slide from Stanford CS231n (2026), Lecture 12, p. 98. Figure: Caron et al. (2021), Algorithm 1.
B · Self-Distillation Without Labels, DINO

Slide from Stanford CS231n (2026), Lecture 12, p. 99. The deck credits Caron et al. (2021); the figure is Oquab et al. (2024), Fig. 1 (DINOv2).
B · Self-Distillation Without Labels, DINO
A general formulation for contrastive learning:
InfoNCE loss: N-way classification among positive and negative samples
Commonly known as the InfoNCE loss (van den Oord et al., 2018)
A lower bound on the mutual information between f(x) and f(x+)
Slide from Stanford CS231n (2026), Lecture 12, p. 100. Math retyped from the slide.
B · Self-Distillation Without Labels, DINO
SimCLR: a simple framework for contrastive representation learning

Slide from Stanford CS231n (2026), Lecture 12, p. 101. Figure: T. Chen et al. (2020), Fig. 2.
B · Self-Distillation Without Labels, DINO
MoCo (v1, v2): contrastive learning using momentum sample encoder

Slide from Stanford CS231n (2026), Lecture 12, p. 102. Figure: He et al. (2020), Fig. 1.
B · Self-Distillation Without Labels, DINO
CPC: sequence-level contrastive learning

Slide from Stanford CS231n (2026), Lecture 12, p. 103. Figure: van den Oord et al. (2018).
Part B
Coming up in Part C: today's best vision encoders, and how to choose one.
Part C
Today's best encoders combine the three methods, at scale. Which one to use depends on the task.
C
Slide from Georgia Tech CS 8803-VLM (2026), Lecture 3, p. 41.
C
| Family | Idea | Examples | Strong at | Weak at |
|---|---|---|---|---|
| Contrastive / joint embedding | Two augmented views of the same image attract; different images repel | SimCLR, MoCo | Global, linear-probe semantics | Needs negatives & large batches; dense features mediocre |
| Masked image modeling (MIM) | Hide most patches, reconstruct the missing pixels / tokens | BEiT, MAE, iBOT | Scalable pretraining; strong after finetuning | Frozen / linear-probe features are weak |
| Self-distillation (no negatives) | Student matches the output distribution of an EMA teacher across views | BYOL, DINO, DINOv2/v3 | Frozen features; emergent object structure | Collapse must be prevented explicitly |
DINOv2 / DINOv3 deliberately combine the last two: an image-level self-distillation loss for global semantics + a patch-level masked loss for dense features.
Slide from Georgia Tech CS 8803-VLM (2026), Lecture 3, p. 42.
C

Slide from Georgia Tech CS 8803-VLM (2026), Lecture 3, p. 48. Figure: Siméoni et al. (2025), Fig. 6; panels cropped from the deck page.
C

Slide from Georgia Tech CS 8803-VLM (2026), Lecture 3, p. 49. Page cites Siméoni et al. (2025), §4.2 and Fig. 10. Panels cropped; formulas retyped.
C
| Backbone | Supervision | ADE20k mIoU ↑ | NYUv2 depth RMSE ↓ | DAVIS track J&F ↑ | IN1k linear top-1 ↑ |
|---|---|---|---|---|---|
| SigLIP 2 g/16 | image–text (weak) | 42.7 | 0.494 | 62.3 | 89.1 |
| PE core G/14 | image–text (weak) | 38.9 | 0.590 | 53.1 | 89.3 |
| DINOv2 g/14 | self-supervised | 49.5 | 0.372 | 73.6 | 87.3 |
| PE spatial G/14 | image–text + distill | 49.3 | 0.362 | 74.5 | — |
| DINOv3 7B/16 | self-supervised | 55.9 | 0.309 | 79.7 | 88.4 |
Slide from Georgia Tech CS 8803-VLM (2026), Lecture 3, p. 51. Page note: Siméoni et al., DINOv3, 2025, Tabs. 3, 5, 7. Linear probes on frozen features; DAVIS at medium resolution.
C
| Method | Data processing | Compute |
|---|---|---|
| MAE | ImageNet; a random crop is enough | Encoder reads 25% of patches. ViT-L: 31 h, 128 TPU v3 cores |
| SimCLR, MoCo | Crop and colour change are necessary | Many negatives: batch 4,096, or a queue of 65,536 |
| CLIP | 400M web image–text pairs | Batch 32,768. ViT-L/14: 12 days, 256 V100 GPUs |
| DINO, DINOv2 | Multi-crop; 1.2B web images curated to 142M | Two networks. DINOv2 g/14: 22,016 A100 GPU-hours |
Each paper's own set-up, so not a controlled comparison. MAE: Table 1e, §4.2 (1,600 epochs). SimCLR: Fig. 5, §2.2. MoCo: §4. CLIP: §2.2, §2.5. DINO: §3.2. DINOv2: §3, Table 14.
C
No labels. Image–text models: CLIP, SigLIP 2, or DINOv3 with dino.txt.
Few labels, little compute. Needs good features as they are: DINOv3, CLIP.
Train small low-rank updates; keep the weights frozen. Lecture 1·02, slides 149–158.
Most labels and compute. MAE is strong here.
The test can change the winner. ViT-L, frozen: MoCo v3 77.6, MAE 73.5. Tune one block: MAE 81.0, MoCo v3 79.9.
⚖️ LO5 No labels does not mean no data problems. Nobody checks billions of web images for consent or bias. DINOv2 removed unsafe images and blurred faces first.
MAE against MoCo v3 (Chen et al., 2021): He et al. (2022), Fig. 9, values read from the plot labels. Zero-shot: Radford et al. (2021); k-NN: Caron et al. (2021); LoRA: Hu et al. (2022). Face blurring: Oquab et al. (2024), §3.
Part C
Close · Summary
| Method | What the model predicts | How it can cheat | What stops the cheat |
|---|---|---|---|
| MAE | The hidden image patches | Copy from the neighbouring patches | Hide 75% of the patches |
| SimCLR | Which other view belongs to this image | Give every image the same vector | The other images in the batch (negatives) |
| DINO | The teacher's output for another view of the image | The same output for every image | Centring, sharpening, and a slow teacher |
Where we got to
Three answers, each with one design choice that stops the cheat: predict the hidden patches (MAE: hide most of them); match another view of the same image (SimCLR: the other images in the batch are negatives); match a slow teacher (DINO: centring and sharpening).
Captions help with the names of things; images alone give better features for each patch. Today's encoders combine both. Next week: connect an encoder to an LLM.
What comes next · Wed 4 Nov
CLIP can rank labels you give it, but it cannot answer a question about the image. How do we connect its image encoder to an LLM?