Introduction to Generative AI with Foundation Models
CEG5305 · Department of Electrical and Computer Engineering, National University of Singapore · 4 MC, graduate level.
This site holds interactive companions to the lectures. Announcements, slides, homework, submissions and grades are on Canvas, which takes precedence wherever the two differ.
Part 2 lectures
Part 2 covers Topics 4–7, foundation model engineering. Each lecture is one syllabus topic; the six-hour topics run over two sessions.
Lecture 1
Architecture and training of LLMs
Session 01 → Session 02 →
Introduction to the fundamental architecture of LLMs and FMs, the training objectives of LLMs and FMs, the scaling law, parameter-efficient tuning with LoRA, and pre-training of LLMs and FMs.
Session 01: from n-grams to the Transformer, the components inside a GPT, ablation. Session 02: training, data, scaling laws, parameter-efficient fine-tuning.
Lecture 2
Post-training and prompting
Session 01 → Session 02 →
Understanding the process of current major post-training techniques for LLMs and FMs, including instruction tuning and reinforcement learning with human feedback (RLHF). Analyze the strengths and weaknesses of traditional prompting and advanced chain-of-thought (CoT) prompting.
Session 01: why we need post-training, how it is done (instruction tuning, RLHF, DPO, GRPO), and what it enables (reasoning, tool use, agents, safety). Session 02: prompting, from one call to a loop: a brief history of 19 prompting methods, chain-of-thought and reasoning models, agents and harnesses.
Lecture 3Self-supervised foundation modelsSession 01 →
Formulating self-supervised learning as the key paradigm for training Foundation Models with minimal human annotation. Derivation of the core objectives and design principles behind DINO-style self-distillation, CLIP-style contrastive pretraining, and MAE-style masked reconstruction, emphasizing representation learning, transferability, robustness, and practical trade-offs in data processing, computation, and downstream adaptation.
How a model learns from images without labels: pretext tasks, contrastive learning (SimCLR, MoCo, CPC), self-distillation (DINO) and today's vision encoders.
Lecture 4Extending LLMs to large multi-modal modelsSession 01 →
Exploration of extending LLMs and prompting towards other data domains, especially visual data with large language-vision models (LLVMs). Introduction to the applications of LLVMs and other large multi-modal models.
How LLMs learned to see (CLIP, Flamingo, BLIP-2, Qwen-VL), building a vision-language model step by step (LLaVA, with code from nanoVLM), and advanced topics: Qwen3-VL, computer-use agents, robot actions, world models and safety.
Course structure
| Part | Topic | Hours |
|---|---|---|
| Part 1 Core generative modelling foundations | 1 · Foundations and recap: CNNs, GANs → Transformers and language models | 6 |
| 2 · Autoregressive models and latent variable models | 6 | |
| 3 · Diffusion models | 6 | |
| Part 2 Foundation model engineering | 4 · Architecture and training of LLMs | 6 |
| 5 · Post-training and prompting | 6 | |
| 6 · Self-supervised foundation models | 3 | |
| 7 · Extending LLMs to large multi-modal models | 3 |
The schedule, assessment and reading list are published on Canvas.