CEG5305 · Generative AI with Foundation Models
AY2026/27 · Semester 1

Introduction to Generative AI with Foundation Models

CEG5305 · Department of Electrical and Computer Engineering, National University of Singapore · 4 MC, graduate level.

Lecturers · Liu Xingyu (Part 1) · Junyuan Hong (Part 2)
This site holds interactive companions to the lectures. Announcements, slides, homework, submissions and grades are on Canvas, which takes precedence wherever the two differ.

Part 2 lectures

Part 2 covers Topics 4–7, foundation model engineering. Each lecture is one syllabus topic; the six-hour topics run over two sessions.

Lecture 1 Architecture and training of LLMs Session 01 → Session 02 → Introduction to the fundamental architecture of LLMs and FMs, the training objectives of LLMs and FMs, the scaling law, parameter-efficient tuning with LoRA, and pre-training of LLMs and FMs. Session 01: from n-grams to the Transformer, the components inside a GPT, ablation. Session 02: training, data, scaling laws, parameter-efficient fine-tuning.
Lecture 2 Post-training and prompting Session 01 → Session 02 → Understanding the process of current major post-training techniques for LLMs and FMs, including instruction tuning and reinforcement learning with human feedback (RLHF). Analyze the strengths and weaknesses of traditional prompting and advanced chain-of-thought (CoT) prompting. Session 01: why we need post-training, how it is done (instruction tuning, RLHF, DPO, GRPO), and what it enables (reasoning, tool use, agents, safety). Session 02: prompting, from one call to a loop: a brief history of 19 prompting methods, chain-of-thought and reasoning models, agents and harnesses.
Lecture 3Self-supervised foundation modelsSession 01 → Formulating self-supervised learning as the key paradigm for training Foundation Models with minimal human annotation. Derivation of the core objectives and design principles behind DINO-style self-distillation, CLIP-style contrastive pretraining, and MAE-style masked reconstruction, emphasizing representation learning, transferability, robustness, and practical trade-offs in data processing, computation, and downstream adaptation. How a model learns from images without labels: pretext tasks, contrastive learning (SimCLR, MoCo, CPC), self-distillation (DINO) and today's vision encoders.
Lecture 4Extending LLMs to large multi-modal modelsSession 01 → Exploration of extending LLMs and prompting towards other data domains, especially visual data with large language-vision models (LLVMs). Introduction to the applications of LLVMs and other large multi-modal models. How LLMs learned to see (CLIP, Flamingo, BLIP-2, Qwen-VL), building a vision-language model step by step (LLaVA, with code from nanoVLM), and advanced topics: Qwen3-VL, computer-use agents, robot actions, world models and safety.

Course structure

PartTopicHours
Part 1
Core generative modelling foundations
1 · Foundations and recap: CNNs, GANs → Transformers and language models6
2 · Autoregressive models and latent variable models6
3 · Diffusion models6
Part 2
Foundation model engineering
4 · Architecture and training of LLMs6
5 · Post-training and prompting6
6 · Self-supervised foundation models3
7 · Extending LLMs to large multi-modal models3

The schedule, assessment and reading list are published on Canvas.