The plan
| Block | What it covers |
|---|---|
| Opening | Why study prompting · Recap: Lecture 2 session 01 · Today's question |
| Part A | A brief history of prompting: from question to instruction · think · prompt engineering: from manual to automatic · tool use · context engineering and agent · what's next? |
| Part B | Prompt to think: chain-of-thought and reasoning models |
| Break | 10 minutes |
| Part C | Prompt as System: Agents and harness: how capable are agents today? · what is an agent? · harness or training? · agents are systems · context and safety · multi-agent: When to invoke more agents? |
| Close | Strengths and weaknesses · homework |
A brief history of prompting
How did prompting grow from one question to agents?
Nineteen methods on one math question (GSM8K, Cobbe et al., 2021). Each method fixes the limitation of the one before. Methods 1–10 change the text that the model reads. Methods 11–19 add tools, loops, context and memory. The figure shows 19 ways to ask LLMs, from 2020 to 2026.
The nineteen methods
The numbers are the method numbers on the slides. Most methods have a slide on what they fix, a slide with code, and a slide on where they are used and what limit remains.
- 1Direct question: end the prompt with "Answer:"
- 2Instruction: specify the task and the answer format
- 3In-context learning: show how to solve (Brown et al., 2020)
- 4Chain-of-thought: the model writes the steps first (Wei et al., 2022)
- 5Best-of-n: try n times, and one try often wins (Cobbe et al., 2021)
- 6Self-consistency: take the answer most chains agree on (Wang et al., 2023)
- 7System prompt: the developer's set-up gets its own role (OpenAI, 2023)
- 8Prompt engineering: prompt as program
- 9APE: an LLM writes the instruction, a score picks it (Zhou et al., 2022)
- 10aOPRO: the LLM reads scored prompts, writes a better one (Yang et al., 2024)
- 10bTextGrad: written feedback works like a gradient (Yuksekgonul et al., 2024)
- 11Tool use: the model asks for a function, our code runs it (Schick et al., 2023)
- 12aReAct: the LLM thinks and acts, the world observes (Yao et al., 2023)
- 12bReflexion: after a failed try, write a lesson, try again (Shinn et al., 2023)
- 13MCP: one standard plug between agents and tools (Anthropic, 2024)
- 14Context engineering: choose all that the model reads (Karpathy, 2025)
- 15ACE: the agent edits its own playbook (Zhang et al., 2025)
- 16Agent design: instructions, tools and a loop, in a library
- 17Multi-agent systems: split the work, check each other (Hadfield et al., 2025)
- 18Long-horizon agents: work for hours, keep memory (Packer et al., 2023)
- 19What's next: agents that improve agents (Zhang et al., 2025)
From question to instruction
The weights do not change: the model learns the task from k examples in its input.
1 · Direct answer: easy to use, hard to read
- Limitation
- Any form: "$18", "18 dollars", a paragraph. Hard to check.
- Unclear task: answer, explain, or continue the story?
2 · Instruction: says what to do, not how to do it
- Where it is used
- Almost every request to a chat model
- Extraction ("return JSON with fields …"), classification labels, summaries
- Limitation
- An instruction says what to do, not how.
- Some tasks are hard to define via description.
3 · In-context learning: copies the pattern, not the reasoning
- Where it is used
- GPT-3 apps, 2020–2022: translation, classification, formatting
- Today: fixing an output format or a style with two or three examples
Think
No checker and no training: the chains vote.
4 · Chain-of-thought: one chain, one chance
- Where it is used
- Math, code, science questions, planning
- Reasoning models (o1, 2024) are trained to write the chain themselves: Part B
- Limitation
- We sample one chain. One wrong step (16 − 3 − 4 = 8) makes the answer wrong.
- Small models gain little from CoT (Wei et al., 2022).
5 · Best-of-n: needs something that knows the right answer
- Where it is used
- Code generation: run the unit tests, keep a program that passes
- Math with a checker; image generation (pick the best image)
- Limitation
- To know which try won, you need a checker: the gold answer, tests or a trained verifier.
- For a new math question there is no gold answer.
6 · Self-consistency: more reliable, but costly and hand-built
- Where it is used
- Math and multiple-choice questions
- Code: group programs by their outputs, then vote (AlphaCode): Part B
Prompt engineering: from manual to automatic
Practice: 1. write the prompt as a template, 2. test each change on many questions, 3. keep the better version.
7 · System prompt: a compound instruction, written by hand
- Where it is used
- Every chat product: tutors, customer service, coding assistants
- Published system messages, e.g. the one of Codex (OpenAI, 2025)
- Limitation
- A good system prompt comes from trial and error.
- Small changes in wording change accuracy, and the best wording differs per model.
8 · Prompt engineering: a person searches by hand
- Where it is used
- Prompt templates kept in code, with versions and tests
- Prompting guides from the model vendors
- Limitation
- A person writes and tests each version.
- The space of wordings is huge, and the best wording differs per model.
9 · APE: one round of guessing, no learning from scores
- Where it is used
- Automatic prompt search for classification, extraction and math
- Prompt optimizers in libraries such as DSPy (Khattab et al., 2023)
- Limitation
- APE samples candidates and keeps the best. It does not learn from the scores which direction helps.
- It uses no feedback about why a prompt failed.
10 · LLM optimizers: better text, but still only text
- Where it is used
- Prompt optimization in apps; DSPy optimizers
- TextGrad also optimizes code, molecules and treatment plans (Yuksekgonul et al., 2024)
Tool use
Anthropic released the Model Context Protocol on 25 Nov 2024 as an open standard (Anthropic, 2024). Write a tool server once, and every MCP client can use it.
11 · Tool use: one call, one step, our code decides
- Where it is used
- Calculators and code interpreters: math, data analysis
- Search and retrieval for current facts; email, calendar and booking APIs
- Limitation
- One tool step per round: our code decides when to call the model again.
- A task with many steps (search → read → compute → check) needs a loop, and the model cannot learn from its failed tries.
12 · ReAct, Reflexion: strong loops, custom plumbing
- Where it is used
- QA with search (HotpotQA), web shopping (WebShop), household tasks (ALFWorld)
- Coding agents: run the tests, reflect, fix
- Limitation
- Every app writes its own tool wrappers: search, files, databases, each for each model.
- N apps × M tools: the same integration is written again and again.
13 · MCP: many tools, and a context that fills up
- Where it is used
- Coding agents connected to GitHub, databases and file systems
- Connectors in chat apps: mail, calendars, drives
Context engineering and agent
ACE grows a detailed playbook instead of rewriting a short prompt (Zhang et al., 2025). Harness engineering changes the system around the model: prompts, tools, memory, and control flow.
14 · Context engineering: designed once, by a person
- Where it is used
- Coding agents: which files and which tool results to show
- Long chats: summarize (compact) old turns; retrieval chatbots
- Limitation
- A person writes the context rules once.
- The agent does not improve them from its own successes and failures.
15 · ACE: a better context, but a hand-written loop
- Where it is used
- Agent benchmarks (AppWorld) and finance tasks (Zhang et al., 2025)
- Rules files that agents keep for a project, e.g. AGENTS.md (OpenAI, 2025)
- Limitation
- Each app still writes its own loop, tool handling and stop rules.
- Real agents also need guardrails, hand-offs and traces, and these are easy to get wrong.
16 · One agent: one context does every job
- Where it is used
- Customer-service and data-analysis agents
- Limitation
- One agent plans, solves and checks, all in one context.
- Long tasks overflow that context, and nobody checks its work.
17 · Multi-agent systems: smart within a session
- Where it is used
- Deep-research products: a lead agent and searching subagents
- Coding: planner, coder, reviewer
- Limitation
- Each session starts fresh: the agents forget what they learned yesterday.
- Tasks of hours or days need memory and state across sessions.
18 · Long-horizon agents: people still design them
- Where it is used
- Software engineering: Claude Code, Codex
- Personal assistants on your own machine: OpenClaw
What's next?
Darwin Gödel Machine: an agent rewrites its own code and keeps an archive of variants (Zhang et al., 2025).
19 · What's next: open problems
- Where it is used
- ML experiments (autoresearch), coding agents (Darwin Gödel Machine)
- Agents that keep and edit their own playbooks (ACE)
Prompt to think
When does writing more before the answer help?
This part follows the lecture Inference-time techniques for LLM reasoning by X. Chen (2025). The model writes more before it answers: reasoning models, chain-of-thought prompting, search over many chains, and revision. Chen's three steps: one longer chain from a better prompt; many chains, then a vote or a verifier; a revised chain, which needs external feedback.
Reasoning: test-time scaling
OpenAI o1 (the first large-scale reasoning model) started to achieve impressive performance across various challenging reasoning tasks in math, coding, STEM, etc. (OpenAI, 2024). But the gain gets smaller. Core idea: trigger the LLM to generate long chain-of-thought (CoT).
Longer: one longer chain from a better prompt
How to improve the CoT performance at inference time: few-shot prompting with labeling of thoughts (Wei et al., 2022); instruction prompting to trigger CoT generation (Kojima et al., 2022); instruct the LLM to automate the prompt design, as in analogical prompting (Yasunaga et al., 2024).
Chain-of-thought prompting: variable computation of the thought process adapting to tasks of different difficulty levels. More complex questions -> more reasoning steps in the chain-of-thought. Least-to-most prompting: easy-to-hard generalization via decomposition (Zhou et al., 2023). Without CoT, the depth L limits the serial steps (Li et al., 2024).
Wider: many chains, then a vote or a verifier
Exploring multiple branches allows the LLM to recover from mistakes in a single generation. Challenge: how to select the best response from multiple candidates? In most cases, we do not have an oracle scorer at inference time. Self-consistency: select the response with the most consistent final answer (Wang et al., 2023). Universal self-consistency: ask the LLM to perform consistency-based selection (Chen et al., 2023). Consistency-based selection: a simple, effective and general principle.
Deeper: a revised chain, which needs external feedback
Inference-time self-improvement: LLM iteratively improves its own response for the given task. Reflexion and Self-Refine (Shinn et al., 2023; Madaan et al., 2023). Self-debugging is a natural workflow for code generation (Chen et al., 2024). Self-reflection and self-refinement work with good (external) evaluation. Oracle: utilize the ground truth answer for correction. Without oracle feedback, LLMs need to judge the response correctness themselves.
The Bitter Lesson from Richard Sutton (again) is an important guideline for designing reasoning techniques, including both inference-time and training-time algorithms (Sutton, 2019).
CoT helps mainly on math and symbolic tasks
Large gains from CoT come mainly on math, logic and symbolic tasks. Elsewhere, the gains are small (Sprague et al., 2025). The chain need not show the real reason (Turpin et al., 2023).
Prompt as System: Agents and harness
What do the loop and the harness add?
When one call is not enough: the model in a loop, the system around it, and how to secure it. This part follows the lecture What are agents? And how do they work? by Fried and Neubig (2026a) and uses pages from two later lectures of the same course (Neubig, 2026; Fried and Neubig, 2026b).
How capable are agents today?
A web agent reads the screen, then clicks. Agents are capable but naughty. What should an agent be allowed to do autonomously? Autonomous, ask first, never.
Challenges. Agent basics: How to empower an LLM with abilities beyond chat and training? Better control: How to control an agent so that humans do not have to be in the loop frequently?
What is an agent?
Environment: a repository, website, application, or workflow. State/Observations: messages, file contents, webpages, screenshots, and tool results. Actions: replies, file edits, shell commands, API calls, and mouse or keyboard events. The agent senses its environment and acts on it (Russell and Norvig, 2020).
From LLMs to agents: from one shot to loop. In the agent (thought-act-observe loop), a Thought is to reason and plan, an Action is a command, and an Observation is text sent back (Yao et al., 2023). Tool definition: a typed interface exposed to the model, with a name and description and a JSON Schema for arguments. The chat template renders it as text. The id links the result to its call. An agent is a model in a loop. The context holds the instruction, the tool result and the conversation history. It is the state. The figure shows one turn of the loop.
Harness or training?
A usable agent should be reliable: accurate tool calling, coherence over long context, customizability, complex task management, environment understanding, safety. How to build these capabilities? Two ways: LLM training and harness engineering.
The table gives, for each capability, the failures, the harness engineering and the LLM training. The six slides follow.
| Capability | Failures | Harness engineering | LLM training |
|---|---|---|---|
| Accurate tool calling | Wrong tool · Wrong type · Broken format · Irrelevant result | Grammar-constrained decoding. Checks: tool exists? types right? result relevant? | SFT: train on tool-calling traces. |
| Coherence over long context | Overflows the window · Forgets an early rule · Clutters with a side task · Drifts from the goal | Context compression/compaction. Dynamic memory lookup. Sub-agent delegation. Task checklist. | SFT: long-context training. |
| Customizability | Ignores user preferences · Lacks our workflow · Cannot reach our own tools | Agent memory. Skills. Custom tools. | RL: learning from user feedback. |
| Complex task management | Skips a step · Overloads one thread · Trusts an unchecked result | Provide planning/decomposition tools. Sub-agent delegation. Verify results with tests. | RL: train on complex, long-horizon tasks. |
| Environment understanding | Lacks domain knowledge · Cannot reach the data source · Breaks the output shape · Misreads FAILED test_b | Learn skills corresponding to domain knowledge. MCP: connect tools and data. | SFT: train on data w/ expected observation shape. RL: train in domain-specific environments. |
| Safety | Deletes the inbox · Leaks a credential · Edits outside the repo · Unseen for 100 steps | Sandboxing tools. Limit access to credentials. Monitor trajectories. | RL: safety-aware RL. |
Capability 1: tool calling
Capability 2: long context
Capability 3: customizability
Capability 4: complex tasks
Capability 5: the environment
Capability 6: safety
Agents are systems, not just models
The figure shows the sandbox, the harness, inference, the model, monitoring and training. The table gives the shared functionality of each part.
| Part | Shared functionality |
|---|---|
| Harness engineering | Manage state, tools, memory, and control flow. Validate actions and handle errors. Enforce permissions and safety boundaries. |
| Sandbox | Isolate code and tool execution. Limit compute, network, and filesystem access. Create reproducible environments. |
| LM inference | Serve model generations reliably. Batch requests and reuse the KV cache. Manage streaming, parallelism, and throughput. |
| Training systems | Prepare data and collect rollouts. Coordinate distributed workers. Checkpoint, evaluate, and reproduce runs. |
| Observability and monitoring | Capture traces and metrics. Track quality, cost, and failures. Compare trajectories and evaluations. |
Context and safety
Context
Agent can accumulate a lot of context in long run. Quadratic input from linear history growth: the input at call t is the fixed prefix plus the history before call t. Context compaction: compaction as state estimation has three parts. Behavioral fidelity, bounded representation, and recoverability: copy exact anchors; retain pointers to source evidence. Repeated compaction drift: each update inherits the previous checkpoint. Copy anchors exactly, keep the recent tail, retrieve source evidence.
Threat models
| Threat model | What happens |
|---|---|
| Malicious user | The user asks the agent to do something it should not do. |
| Prompt injection | The user has a legitimate task for the agent. An attacker puts instructions in an email, webpage, file, or tool result. The agent follows them while carrying out the user’s task. |
| Agent error | The agent tries to do something useful, but makes a mistake. |
| Improper means | The agent pursues a useful goal using an inappropriate method. |
What is a sandbox? A sandbox is an isolated environment that limits what a program can access and do. Isolation boundaries: filesystem and processes, network, and resource limits.
Multi-agent: When to invoke more agents?
When to fold subagents into your main agent loop (Sarkis, 2026): a single agent with skills frequently outperformed subagents; enterprise deployments benefited from folding subagents in; the model no longer needed most of this structure. Common pattern: 5 layers. What we often recommend: 1 loop. Each handoff loses context when agents split by problem type. Merge subagents, call them as tools, or fold into skills and tools. Measure, change one thing at a time, and keep what holds up.
Where we got to
The lecture ends with a look back at each setting. Each has a strength and a weakness.
How far can a fixed model go? Answered today: far, when the input and the loop give what the weights lack: examples, time to reason, tools, feedback and a check.
Practice homework (no course credit)
Demo code: two_agent_demo.py · Colab notebook.
Two agents talk. Each agent is a model with its own instructions, and the reply of one is the input of the other. The demo code plays a patient and a counsellor in a first counselling session. It is a class role-play with two simulated roles, not medical advice: do not type your own problems into it.
- Run the demo. Get two_agent_demo.py, or open the Colab notebook. It uses the OpenAI Agents SDK and needs an API key for an OpenAI-compatible model; set
OPENAI_BASE_URLto use another server. How the course gives access will be posted here. - Predict, then run. Write your predictions with probabilities before you run. For example: will the patient stay in role? How does the talk end?
- Choose one piece to change and change only that:
- The scenario: patient and counsellor, customer and support agent, interviewer and candidate, tutor and student, or two neighbours. Or write your own.
- The turn limit and the stop word: when does the talk end?
- The history: does each agent see all earlier messages, or only the last two?
- A guardrail: a check on one agent's message before the other replies.
- One sentence of an agent's instructions.
- Compare. Run each version 3 times. Compare the number of messages, how the talk ended, role breaks and tokens. Use the same scenario, settings and number of runs for every version. Check that the difference is larger than the spread across your 3 runs. A role break is when an agent steps out of its role; read the transcript, because the count is only a hint. If a result looks too good, read the whole conversation.
What you produce, in at most 300 words: your predictions with probabilities, made before you ran; transcripts of the base run and of your change; counts per version over 3 runs; three findings, ranked; and one result you did not predict, explained.
Resources
Reading
- Wei et al. (2022), chain-of-thought prompting. The basis of Parts A and B.
- Yao et al. (2023), ReAct. The loop behind Parts A and C.
- Huang et al. (2024), why a model cannot yet correct its own reasoning without an external check (Part B).
- Sarkis (2026), when to fold subagents into the main agent loop (Part C).
Code
- two_agent_demo.py and the Colab notebook: the two-agent demo of the homework.
Slides
- The slides of this session. Every slide has a teaching script: press N.
References
Every work cited on this page, in APA style with shortened author lists. Titles link to the work.
Next lectures
Our next lectures go beyond language and into more modalities.
Self-supervised foundation models
Wed 28 Oct 2026, 18:00 SGT.
Large multi-modal models
Wed 4 Nov 2026.
Previous session
Session 01: Post-training Large Language Models changed the weights: instruction tuning, learning from preferences and reinforcement learning.