CEG5305 · Generative AI with Foundation Models

Home › Part 2 › Lecture 2 › Session 02

Part 2 - Lecture 2

Prompting Large Language Models

Topic 5, Post-training and Prompting. Session 02: Prompting — from one call to a loop.

Session 02 · three partsLecturer · Junyuan HongSlides · open the slidesPrevious · Session 01
How far can a fixed model go?

Session 01 changed the weights. Today, with weights unchanged, we investigate only the input and the loop around the model.

Part A

A brief history of prompting

Different prompting methods on one math question: from a direct answer to agents.

Part B

Prompt to think

Chain-of-thought and reasoning models: when writing more helps, and when it does not.

Part C

Prompt as System: Agents and harness

The loop and the harness around the model, to make them ready for production.

Every slide has a teaching script: open the slides and press N.

The plan

BlockWhat it covers
OpeningWhy study prompting · Recap: Lecture 2 session 01 · Today's question
Part AA brief history of prompting: from question to instruction · think · prompt engineering: from manual to automatic · tool use · context engineering and agent · what's next?
Part BPrompt to think: chain-of-thought and reasoning models
Break10 minutes
Part CPrompt as System: Agents and harness: how capable are agents today? · what is an agent? · harness or training? · agents are systems · context and safety · multi-agent: When to invoke more agents?
CloseStrengths and weaknesses · homework
PART A

A brief history of prompting

How did prompting grow from one question to agents?

Nineteen methods on one math question (GSM8K, Cobbe et al., 2021). Each method fixes the limitation of the one before. Methods 1–10 change the text that the model reads. Methods 11–19 add tools, loops, context and memory. The figure shows 19 ways to ask LLMs, from 2020 to 2026.

From question to instruction
1Direct answer2020
2Instruction2020
3In-context learning2020
Think
4Chain-of-thought2022
5Best-of-n2021
6Self-consistency2022
Prompt engineering: from manual to automatic
7System prompt2023
8Prompt engineering2023
9
10
Automatic prompt engineering2022–24
Tool use
11Tool use, function calling2023
12ReAct/Reflexion Agent2022–23
13MCP2024
Context engineering and agent
14Context engineering2025
15Agentic context engineering2025
16Develop agent2025
17Multi-agent systems2023–25
18Agent in long run2025
What's next?
19Agents improve agents2025–26
The nineteen methods, grouped by segment, with the year of each.

The nineteen methods

The numbers are the method numbers on the slides. Most methods have a slide on what they fix, a slide with code, and a slide on where they are used and what limit remains.

  1. 1Direct question: end the prompt with "Answer:"
  2. 2Instruction: specify the task and the answer format
  3. 3In-context learning: show how to solve (Brown et al., 2020)
  4. 4Chain-of-thought: the model writes the steps first (Wei et al., 2022)
  5. 5Best-of-n: try n times, and one try often wins (Cobbe et al., 2021)
  6. 6Self-consistency: take the answer most chains agree on (Wang et al., 2023)
  7. 7System prompt: the developer's set-up gets its own role (OpenAI, 2023)
  8. 8Prompt engineering: prompt as program
  9. 9APE: an LLM writes the instruction, a score picks it (Zhou et al., 2022)
  10. 10aOPRO: the LLM reads scored prompts, writes a better one (Yang et al., 2024)
  11. 10bTextGrad: written feedback works like a gradient (Yuksekgonul et al., 2024)
  12. 11Tool use: the model asks for a function, our code runs it (Schick et al., 2023)
  13. 12aReAct: the LLM thinks and acts, the world observes (Yao et al., 2023)
  14. 12bReflexion: after a failed try, write a lesson, try again (Shinn et al., 2023)
  15. 13MCP: one standard plug between agents and tools (Anthropic, 2024)
  16. 14Context engineering: choose all that the model reads (Karpathy, 2025)
  17. 15ACE: the agent edits its own playbook (Zhang et al., 2025)
  18. 16Agent design: instructions, tools and a loop, in a library
  19. 17Multi-agent systems: split the work, check each other (Hadfield et al., 2025)
  20. 18Long-horizon agents: work for hours, keep memory (Packer et al., 2023)
  21. 19What's next: agents that improve agents (Zhang et al., 2025)

From question to instruction

The weights do not change: the model learns the task from k examples in its input.

PromptJanet's ducks lay 16 eggs per day. She eats three … and bakes muffins … with four. She sells the remainder … for $2 per egg. How much does she make every day? Answer:
Example replyShe makes $18 every day at the market, since she has 9 eggs left to sell …
1 · Direct question: end the prompt with "Answer:"
PromptSolve the math word problem. Give only the final number in brackets, like [12]. Janet's ducks lay 16 eggs per day. She eats three … and bakes muffins … with four. She sells the remainder … for $2 per egg. How much does she make every day?
Example reply[18]
2 · Instruction: specify the task and the answer format
PromptSolve the math word problem. Give only the final number in brackets, like [12]. Q: Tom has 3 boxes of 4 pens and gives away 5. How many are left? A: [7] Q: A shop has 20 apples and sells 8, then 6. How many are left? A: [6] Q: Janet's ducks lay 16 eggs a day … How much does she make every day? A:
Example reply[18]
3 · In-context learning: show how to solve

1 · Direct answer: easy to use, hard to read

  • Limitation
    • Any form: "$18", "18 dollars", a paragraph. Hard to check.
    • Unclear task: answer, explain, or continue the story?

2 · Instruction: says what to do, not how to do it

  • Where it is used
    • Almost every request to a chat model
    • Extraction ("return JSON with fields …"), classification labels, summaries
  • Limitation
    • An instruction says what to do, not how.
    • Some tasks are hard to define via description.

3 · In-context learning: copies the pattern, not the reasoning

  • Where it is used
    • GPT-3 apps, 2020–2022: translation, classification, formatting
    • Today: fixing an output format or a style with two or three examples
FIXESThe reply had no fixed form, and the task was not stated. The instruction said what to do, but not how.
LIMITATIONThe examples show only answers.

Think

No checker and no training: the chains vote.

PromptSolve the math word problem. Give only the final number in brackets, like [12]. Q: Tom has 3 boxes of 4 pens. He gives away 5. How many pens are left? A: 3 × 4 = 12 pens. 12 − 5 = 7. [7] Q: Janet's ducks lay 16 eggs a day … How much does she make every day? A:
Example replyShe has 16 − 3 − 4 = 9 eggs. 9 × 2 = 18. [18]
4 · Chain-of-thought: the model writes the steps first
Question score [18] 0.92 [18] 0.87 [17] 0.31 [18] 0.95 [20] 0.12 [18] 0.90 [9] 0.08 [18] 0.89 verifierrφ [18]

Assume an verifier is available.

5 · Best-of-n: try n times, and one try often wins
Question [18] [18] [17] [18] [20] [18] [9] [18] votes per answer 19 117 518 120 [18]
6 · Self-consistency: take the answer most chains agree on

4 · Chain-of-thought: one chain, one chance

  • Where it is used
    • Math, code, science questions, planning
    • Reasoning models (o1, 2024) are trained to write the chain themselves: Part B
  • Limitation
    • We sample one chain. One wrong step (16 − 3 − 4 = 8) makes the answer wrong.
    • Small models gain little from CoT (Wei et al., 2022).

5 · Best-of-n: needs something that knows the right answer

  • Where it is used
    • Code generation: run the unit tests, keep a program that passes
    • Math with a checker; image generation (pick the best image)
  • Limitation
    • To know which try won, you need a checker: the gold answer, tests or a trained verifier.
    • For a new math question there is no gold answer.

6 · Self-consistency: more reliable, but costly and hand-built

  • Where it is used
    • Math and multiple-choice questions
    • Code: group programs by their outputs, then vote (AlphaCode): Part B
FIXESThe examples showed only answers, so the model jumped to the answer. One chain gave one chance. Best-of-n needed a checker that knows the answer.
LIMITATIONIt costs n times the tokens, and a shared mistake wins the vote. The set-up — instruction, examples, format — is still pasted into the user’s message on every call.

Prompt engineering: from manual to automatic

Practice: 1. write the prompt as a template, 2. test each change on many questions, 3. keep the better version.

Questions& answers LLM proposesdev-set accuracy Instruction 1 0.62 Instruction 2 0.71 Instruction 3 0.55 … (20 in all) … bestinstruction
Best found"Let's work this out in a step by step way to be sure we have the right answer."
9 · APE: an LLM writes the instruction, a score picks it
Objective evaluatorcode or a dev set scores New instructions LLM as optimizer Meta-promptsolution–score pairstask description return top solutions when finished
Best found (PaLM 2-L, GSM8K)"Take a deep breath and work on this problem step-by-step."
10a · OPRO: the LLM reads scored prompts, writes a better one
Prompta variable LLM Response Critic LLMthe text loss Feedbackin words Optimizerrewrites the prompt forward backward
Loss, in wordsa critic LLM: "Step 2 subtracts the muffin eggs twice; state each use once."
10b · TextGrad: written feedback works like a gradient

7 · System prompt: a compound instruction, written by hand

  • Where it is used
    • Every chat product: tutors, customer service, coding assistants
    • Published system messages, e.g. the one of Codex (OpenAI, 2025)
  • Limitation
    • A good system prompt comes from trial and error.
    • Small changes in wording change accuracy, and the best wording differs per model.

8 · Prompt engineering: a person searches by hand

  • Where it is used
    • Prompt templates kept in code, with versions and tests
    • Prompting guides from the model vendors
  • Limitation
    • A person writes and tests each version.
    • The space of wordings is huge, and the best wording differs per model.

9 · APE: one round of guessing, no learning from scores

  • Where it is used
    • Automatic prompt search for classification, extraction and math
    • Prompt optimizers in libraries such as DSPy (Khattab et al., 2023)
  • Limitation
    • APE samples candidates and keeps the best. It does not learn from the scores which direction helps.
    • It uses no feedback about why a prompt failed.

10 · LLM optimizers: better text, but still only text

  • Where it is used
    • Prompt optimization in apps; DSPy optimizers
    • TextGrad also optimizes code, molecules and treatment plans (Yuksekgonul et al., 2024)
FIXESThe set-up was mixed into the user’s message on every call. A person had to write and test every prompt. APE did not learn from the scores.
LIMITATIONMethods 1–10 change only the text. The model still computes in its head and can make arithmetic errors on big numbers. It cannot look up facts after its training cut-off, and it cannot act.

Tool use

Anthropic released the Model Context Protocol on 25 Nov 2024 as an open standard (Anthropic, 2024). Write a tool server once, and every MCP client can use it.

Our code Model Calculator 1 messages + tool list 2 tool call: calculator(…) 3 run the tool 4 result: 18 5 send the result back 6 answer: [18]
11 · Tool use: the model asks for a function, our code runs it
the LLM generatesinput from outsidenot sent to the LLM
Thoughtreason, plan Actiona command Environmentruns the actionexcluded from LLM input Observationtext sent back history is addedto the next input
12a · ReAct: the LLM thinks and acts, the world observes
the LLM generatesinput from outsidememory
Agent Self-reflectionLM Experiencelong-term memory EvaluatorLM ActorLM Trajectoryshort-term memory Environmentoutside the agent Action Obs / Reward
12b · Reflexion: a failed try becomes a lesson in memory

11 · Tool use: one call, one step, our code decides

  • Where it is used
    • Calculators and code interpreters: math, data analysis
    • Search and retrieval for current facts; email, calendar and booking APIs
  • Limitation
    • One tool step per round: our code decides when to call the model again.
    • A task with many steps (search → read → compute → check) needs a loop, and the model cannot learn from its failed tries.

12 · ReAct, Reflexion: strong loops, custom plumbing

  • Where it is used
    • QA with search (HotpotQA), web shopping (WebShop), household tasks (ALFWorld)
    • Coding agents: run the tests, reflect, fix
  • Limitation
    • Every app writes its own tool wrappers: search, files, databases, each for each model.
    • N apps × M tools: the same integration is written again and again.

13 · MCP: many tools, and a context that fills up

  • Where it is used
    • Coding agents connected to GitHub, databases and file systems
    • Connectors in chat apps: mail, calendars, drives
FIXESThe model computed in its head, and could not act. The loop did not learn from a failed try. Each app wrote its own wrapper for each tool.
LIMITATIONAn agent now has many tools, a long history and many documents. The context window fills up. More text is not better: at each step the model needs the right information, not all of it.

Context engineering and agent

ACE grows a detailed playbook instead of rewriting a short prompt (Zhang et al., 2025). Harness engineering changes the system around the model: prompts, tools, memory, and control flow.

Query Contextplaybook GeneratorLLM Trajectory ReflectorLLM Insights CuratorLLM Delta items iterativerefinement merged by code, not by an LLM
A delta item the Curator adds (example)• List every use of the eggs before you multiply.
15 · ACE: the agent edits its own playbook
Input Runner: loop until done Agent Model Instructions Tools Run tools Finalanswer tool calls results are added done, or max_turns
16 · Agent design: instructions, tools and a loop, in a library
Input Managerdecides who to call SolverCoT + calculator Checkerrecomputes Answer solve(…) check(…) agents used as tools
17 · Multi-agent systems: split the work, check each other
Main context (the prompt) System instructionsread-only Working contextthe model can write FIFO queuerecent messages LLM External context Recall storageall past messages Archival storagedocuments, notes search, append, page in / out memory pressure: old messages move to recall storage
Our task, nowRun the solver on all 1,319 GSM8K test questions, sort the errors by type, write report.md.
18 · Long-horizon agents: work for hours, keep memory

14 · Context engineering: designed once, by a person

  • Where it is used
    • Coding agents: which files and which tool results to show
    • Long chats: summarize (compact) old turns; retrieval chatbots
  • Limitation
    • A person writes the context rules once.
    • The agent does not improve them from its own successes and failures.

15 · ACE: a better context, but a hand-written loop

  • Where it is used
    • Agent benchmarks (AppWorld) and finance tasks (Zhang et al., 2025)
    • Rules files that agents keep for a project, e.g. AGENTS.md (OpenAI, 2025)
  • Limitation
    • Each app still writes its own loop, tool handling and stop rules.
    • Real agents also need guardrails, hand-offs and traces, and these are easy to get wrong.

16 · One agent: one context does every job

  • Where it is used
    • Customer-service and data-analysis agents
  • Limitation
    • One agent plans, solves and checks, all in one context.
    • Long tasks overflow that context, and nobody checks its work.

17 · Multi-agent systems: smart within a session

  • Where it is used
    • Deep-research products: a lead agent and searching subagents
    • Coding: planner, coder, reviewer
  • Limitation
    • Each session starts fresh: the agents forget what they learned yesterday.
    • Tasks of hours or days need memory and state across sessions.

18 · Long-horizon agents: people still design them

  • Where it is used
    • Software engineering: Claude Code, Codex
    • Personal assistants on your own machine: OpenClaw
FIXESA context that filled up with tools, history and documents. A person wrote the context rules, and they did not improve. Each app wrote its own loop and tool handling. One agent did every job in one context. Each session started fresh.
LIMITATIONPeople still design the agent: its prompt, tools and harness. The agent does not improve its own design, and small errors add up over long tasks.

What's next?

Darwin Gödel Machine: an agent rewrites its own code and keeps an archive of variants (Zhang et al., 2025).

Archive Self-modifyedits its own code Evaluateon a benchmark select a parent new agent (child) add it to the archive
Our task, nextFind a better prompt and harness for GSM8K overnight; measure on held-out questions.
19 · What's next: agents that improve agents

19 · What's next: open problems

  • Where it is used
    • ML experiments (autoresearch), coding agents (Darwin Gödel Machine)
    • Agents that keep and edit their own playbooks (ACE)
FIXESPeople designed every agent by hand.
LIMITATIONThe agent optimizes the metric it is given, so a bad metric gives a bad agent. Cost, safety and checking the results remain open.
Key message. From text completion to agent, the tech of prompting is more and more complicated to resolve the complicated tasks.
PART B

Prompt to think

When does writing more before the answer help?

This part follows the lecture Inference-time techniques for LLM reasoning by X. Chen (2025). The model writes more before it answers: reasoning models, chain-of-thought prompting, search over many chains, and revision. Chen's three steps: one longer chain from a better prompt; many chains, then a vote or a verifier; a revised chain, which needs external feedback.

Reasoning: test-time scaling

OpenAI o1 (the first large-scale reasoning model) started to achieve impressive performance across various challenging reasoning tasks in math, coding, STEM, etc. (OpenAI, 2024). But the gain gets smaller. Core idea: trigger the LLM to generate long chain-of-thought (CoT).

Longer: one longer chain from a better prompt

How to improve the CoT performance at inference time: few-shot prompting with labeling of thoughts (Wei et al., 2022); instruction prompting to trigger CoT generation (Kojima et al., 2022); instruct the LLM to automate the prompt design, as in analogical prompting (Yasunaga et al., 2024).

Chain-of-thought prompting: variable computation of the thought process adapting to tasks of different difficulty levels. More complex questions -> more reasoning steps in the chain-of-thought. Least-to-most prompting: easy-to-hard generalization via decomposition (Zhou et al., 2023). Without CoT, the depth L limits the serial steps (Li et al., 2024).

FIXESEncourage longer CoT for complex tasks. Support reasoning strategies required for the task.
WHAT IS MISSING SO FARWe should not limit the LLM to generate only one solution per problem!

Wider: many chains, then a vote or a verifier

Exploring multiple branches allows the LLM to recover from mistakes in a single generation. Challenge: how to select the best response from multiple candidates? In most cases, we do not have an oracle scorer at inference time. Self-consistency: select the response with the most consistent final answer (Wang et al., 2023). Universal self-consistency: ask the LLM to perform consistency-based selection (Chen et al., 2023). Consistency-based selection: a simple, effective and general principle.

FIXESSampling multiple solutions can reduce mistakes from a single prediction.
LIMITATIONIt is still suboptimal. No feedback loop to correct the mistakes after a complete solution is generated.

Deeper: a revised chain, which needs external feedback

Inference-time self-improvement: LLM iteratively improves its own response for the given task. Reflexion and Self-Refine (Shinn et al., 2023; Madaan et al., 2023). Self-debugging is a natural workflow for code generation (Chen et al., 2024). Self-reflection and self-refinement work with good (external) evaluation. Oracle: utilize the ground truth answer for correction. Without oracle feedback, LLMs need to judge the response correctness themselves.

The Bitter Lesson from Richard Sutton (again) is an important guideline for designing reasoning techniques, including both inference-time and training-time algorithms (Sutton, 2019).

FIXESLLM self-refines its output given both internal feedback and external evaluation.
LIMITATIONSelf-correction without oracle feedback hurts the reasoning performance (Huang et al., 2024). LLMs can wrongly judge the correctness of its predictions, leading to worse performance after self-correction.

CoT helps mainly on math and symbolic tasks

Large gains from CoT come mainly on math, logic and symbolic tasks. Elsewhere, the gains are small (Sprague et al., 2025). The chain need not show the real reason (Turpin et al., 2023).

Key message. A reasoning trajectory cannot be trained token by token, thus prompting is the essential tool to steer the reasoning.
PART C

Prompt as System: Agents and harness

What do the loop and the harness add?

When one call is not enough: the model in a loop, the system around it, and how to secure it. This part follows the lecture What are agents? And how do they work? by Fried and Neubig (2026a) and uses pages from two later lectures of the same course (Neubig, 2026; Fried and Neubig, 2026b).

How capable are agents today?

A web agent reads the screen, then clicks. Agents are capable but naughty. What should an agent be allowed to do autonomously? Autonomous, ask first, never.

CAPABLEAgents today finish long real tasks, such as a C compiler,
BUTand also make costly errors, such as deleting an inbox.

Challenges. Agent basics: How to empower an LLM with abilities beyond chat and training? Better control: How to control an agent so that humans do not have to be in the loop frequently?

What is an agent?

Environment: a repository, website, application, or workflow. State/Observations: messages, file contents, webpages, screenshots, and tool results. Actions: replies, file edits, shell commands, API calls, and mouse or keyboard events. The agent senses its environment and acts on it (Russell and Norvig, 2020).

From LLMs to agents: from one shot to loop. In the agent (thought-act-observe loop), a Thought is to reason and plan, an Action is a command, and an Observation is text sent back (Yao et al., 2023). Tool definition: a typed interface exposed to the model, with a name and description and a JSON Schema for arguments. The chat template renders it as text. The id links the result to its call. An agent is a model in a loop. The context holds the instruction, the tool result and the conversation history. It is the state. The figure shows one turn of the loop.

2 × (3 + 4)prompt tokensQueryModelReasoningEnv / userContextStateinstructionYou are a coding agent.Tools: read_file(path)tool resultdef add(a, b): return a + bconv. historyuser: Read cal.pymodel: read_file("cal.py")
An agent is a model in a loop: what comes back from the environment goes into the context, which is the state.

Harness or training?

A usable agent should be reliable: accurate tool calling, coherence over long context, customizability, complex task management, environment understanding, safety. How to build these capabilities? Two ways: LLM training and harness engineering.

LLM training
  • Change the model’s behavior through pretraining, SFT, or RL.
  • Teach reusable patterns for reasoning, tool use, and recovery.
  • Capabilities become part of the model.
AgentModel🔥 updatePrompt❄️ freezeEnvCollectdataOptimize
Harness engineering
  • Change the system around the model: prompts, tools, memory, and control flow.
  • Provide context, validation, retries, and safety boundaries.
  • Capabilities emerge from the model–harness combination.
AgentModel❄️ freezeContextHarness🔥 updateEnvCollectdataOptimize
Training updates the model and freezes the prompt; harness engineering updates the harness and freezes the model.

The table gives, for each capability, the failures, the harness engineering and the LLM training. The six slides follow.

CapabilityFailuresHarness engineeringLLM training
Accurate tool callingWrong tool · Wrong type · Broken format · Irrelevant resultGrammar-constrained decoding. Checks: tool exists? types right? result relevant?SFT: train on tool-calling traces.
Coherence over long contextOverflows the window · Forgets an early rule · Clutters with a side task · Drifts from the goalContext compression/compaction. Dynamic memory lookup. Sub-agent delegation. Task checklist.SFT: long-context training.
CustomizabilityIgnores user preferences · Lacks our workflow · Cannot reach our own toolsAgent memory. Skills. Custom tools.RL: learning from user feedback.
Complex task managementSkips a step · Overloads one thread · Trusts an unchecked resultProvide planning/decomposition tools. Sub-agent delegation. Verify results with tests.RL: train on complex, long-horizon tasks.
Environment understandingLacks domain knowledge · Cannot reach the data source · Breaks the output shape · Misreads FAILED test_bLearn skills corresponding to domain knowledge. MCP: connect tools and data.SFT: train on data w/ expected observation shape. RL: train in domain-specific environments.
SafetyDeletes the inbox · Leaks a credential · Edits outside the repo · Unseen for 100 stepsSandboxing tools. Limit access to credentials. Monitor trajectories.RL: safety-aware RL.

Capability 1: tool calling

Inaccurate calls for the task “fix test.py”
✗ Wrong toolweb_search
✗ Wrong typepath: 42
✗ Broken formatread_file(path
✗ Irrelevant resultWeather: 31°C
Harness engineering
  • Grammar-constrained decoding.
LLM training
  • SFT: train on tool-calling traces.
LLMTool callExecutionTool result/ERROR!tool exists? types right?result relevant?
Accurate tool calling

Capability 2: long context

Failures in a 200-turn session
✗ Overflowsthe window
✗ Forgetsan early rule
✗ Clutterswith a side task
✗ Driftsfrom the goal
Harness engineering
  • Context compression/compaction.
  • Dynamic memory lookup.
  • Sub-agent delegation.
  • Task checklist.
LLM training
  • SFT: long-context training.
HistoryLLMNext callTask donewindow nearly fullMemory compactionhow to run tests?Memory lookupsearch 50 filesSub-agentbug 1 fixedChecklist
Coherence over long context

Capability 3: customizability

Failures without customization
✗ Ignoresuser preferences
✗ Lacksour workflow
✗ Cannot reachour own tools
Harness engineering
  • Agent memory.
  • Skills.
  • Custom tools.
LLM training
  • RL: learning from user feedback.
TaskLLMResultprefers type hintsMemorywrite a release noteSkill filedeploy to stagingCustom tool
Customizability

Capability 4: complex tasks

Failures on a long task
✗ Skipsa step
✗ Overloadsone thread
✗ Trustsan unchecked result
Harness engineering
  • Provide planning/decomposition tools.
  • Sub-agent delegation.
  • Verify results with tests.
LLM training
  • RL: train on complex, long-horizon tasks.
TaskLLMResultcode, tests, docsPlantests and docsDelegateall tests pass?Verify
Complex task management

Capability 5: the environment

Failures in a shell
✗ Lacksdomain knowledge
✗ Cannot reachthe data source
✗ Breaksthe output shape
✗ MisreadsFAILED test_b
Harness engineering
  • Learn skills corresponding to domain knowledge.
  • MCP: connect tools and data.
LLM training
  • SFT: train on data w/ expected observation shape.
  • RL: train in domain-specific environments.
LLMCommandShellOutputwhich test runner?Skill: domainread a ticketMCP server
Environment understanding

Capability 6: safety

Unsafe actions
✗ Deletesthe inbox
✗ Leaksa credential
✗ Editsoutside the repo
✗ Unseenfor 100 steps
Harness engineering
  • Sandboxing tools.
  • Limit access to credentials.
  • Monitor trajectories.
LLM training
  • RL: safety-aware RL.
LLMCredential requestHuman approvalTool callrm -rfGuardrail reviewExecutionaccess PhotosOut of sandboxResultinsulting textMonitor
Safety
ADVANTAGES OF HARNESS ENGINEERINGEfficient: Edit a prompt, a tool, or a skill. No annotation and no GPU hours. Cheap: No training run. Test a change on the model you already use. Fix corner cases: Add a rule, a check, or a skill for the case. No new data needed. Anyone can customize: Memory, skills, and custom tools are files that an individual can write today.
LIMITATION OF TRAININGInefficient: Human annotation and a lot of GPU hours. Expensive: Data collection and training. Hard to generalize: More data is needed for corner cases, and niche domains always have corner cases. Not possible to customize by individuals: An individual has no resources for it.

Agents are systems, not just models

The figure shows the sandbox, the harness, inference, the model, monitoring and training. The table gives the shared functionality of each part.

Sandbox Harness Inference Model Monitoring Training
An agent is a system: a harness inside a sandbox, a model, and inference, monitoring and training around them.
PartShared functionality
Harness engineeringManage state, tools, memory, and control flow. Validate actions and handle errors. Enforce permissions and safety boundaries.
SandboxIsolate code and tool execution. Limit compute, network, and filesystem access. Create reproducible environments.
LM inferenceServe model generations reliably. Batch requests and reuse the KV cache. Manage streaming, parallelism, and throughput.
Training systemsPrepare data and collect rollouts. Coordinate distributed workers. Checkpoint, evaluate, and reproduce runs.
Observability and monitoringCapture traces and metrics. Track quality, cost, and failures. Compare trajectories and evaluations.

Context and safety

Context

Agent can accumulate a lot of context in long run. Quadratic input from linear history growth: the input at call t is the fixed prefix plus the history before call t. Context compaction: compaction as state estimation has three parts. Behavioral fidelity, bounded representation, and recoverability: copy exact anchors; retain pointers to source evidence. Repeated compaction drift: each update inherits the previous checkpoint. Copy anchors exactly, keep the recent tail, retrieve source evidence.

Threat models

Threat modelWhat happens
Malicious userThe user asks the agent to do something it should not do.
Prompt injectionThe user has a legitimate task for the agent. An attacker puts instructions in an email, webpage, file, or tool result. The agent follows them while carrying out the user’s task.
Agent errorThe agent tries to do something useful, but makes a mistake.
Improper meansThe agent pursues a useful goal using an inappropriate method.

What is a sandbox? A sandbox is an isolated environment that limits what a program can access and do. Isolation boundaries: filesystem and processes, network, and resource limits.

DEFENCESFine-grained tokens: Limit which resources the token can access, which actions it permits, and how long it lasts. Monitoring agent behavior: Monitoring records and checks an agent’s behavior to identify mistakes and unsafe actions.
LIMITSKeeping an API key hidden does not prevent the agent from misusing the access it provides. An agent can produce a useful answer after taking an unsafe action.

Multi-agent: When to invoke more agents?

When to fold subagents into your main agent loop (Sarkis, 2026): a single agent with skills frequently outperformed subagents; enterprise deployments benefited from folding subagents in; the model no longer needed most of this structure. Common pattern: 5 layers. What we often recommend: 1 loop. Each handoff loses context when agents split by problem type. Merge subagents, call them as tools, or fold into skills and tools. Measure, change one thing at a time, and keep what holds up.

Key message. Harness is essential when an agent becomes a complicated system with constraints, tools, failure control, etc.

Where we got to

The lecture ends with a look back at each setting. Each has a strength and a weakness.

SettingStrengthWeakness
PromptingNo training. Works with any model you can call.The prompt's format alone changes accuracy by up to 76 points (Sclar et al., 2024).
Chain-of-thoughtLarge gains on math and symbolic tasks (Sprague et al., 2025).Little gain elsewhere, and it can hurt (Liu et al., 2024). The chain need not show the real reason (Turpin et al., 2023).
Reasoning modelImproves with more training and more thinking time (OpenAI, 2024).Few-shot prompts make DeepSeek-R1 worse (Guo et al., 2025). Reasoning models state a hint they used in 25% (Claude 3.7 Sonnet) to 39% (DeepSeek R1) of cases (Y. Chen et al., 2025).
Agent + harnessFacts and actions beyond the weights. Tool feedback can correct errors.Unreliable across runs (Yao et al., 2024). Errors add up on long tasks (Sinha et al., 2026). Prompt injection (Greshake et al., 2023).
Strengths and weaknesses of each setting.

How far can a fixed model go? Answered today: far, when the input and the loop give what the weights lack: examples, time to reason, tools, feedback and a check.

Practice homework (no course credit)

This is practice homework. It carries no course credit and is not marked.

Demo code: two_agent_demo.py · Colab notebook.

Two agents talk. Each agent is a model with its own instructions, and the reply of one is the input of the other. The demo code plays a patient and a counsellor in a first counselling session. It is a class role-play with two simulated roles, not medical advice: do not type your own problems into it.

  1. Run the demo. Get two_agent_demo.py, or open the Colab notebook. It uses the OpenAI Agents SDK and needs an API key for an OpenAI-compatible model; set OPENAI_BASE_URL to use another server. How the course gives access will be posted here.
  2. Predict, then run. Write your predictions with probabilities before you run. For example: will the patient stay in role? How does the talk end?
  3. Choose one piece to change and change only that:
    • The scenario: patient and counsellor, customer and support agent, interviewer and candidate, tutor and student, or two neighbours. Or write your own.
    • The turn limit and the stop word: when does the talk end?
    • The history: does each agent see all earlier messages, or only the last two?
    • A guardrail: a check on one agent's message before the other replies.
    • One sentence of an agent's instructions.
  4. Compare. Run each version 3 times. Compare the number of messages, how the talk ended, role breaks and tokens. Use the same scenario, settings and number of runs for every version. Check that the difference is larger than the spread across your 3 runs. A role break is when an agent steps out of its role; read the transcript, because the count is only a hint. If a result looks too good, read the whole conversation.

What you produce, in at most 300 words: your predictions with probabilities, made before you ran; transcripts of the base run and of your change; counts per version over 3 runs; three findings, ranked; and one result you did not predict, explained.

Resources

Reading

Code

Slides

References

Every work cited on this page, in APA style with shortened author lists. Titles link to the work.

    Next lectures

    Our next lectures go beyond language and into more modalities.

    Lecture 3

    Self-supervised foundation models

    Wed 28 Oct 2026, 18:00 SGT.

    Lecture 4

    Large multi-modal models

    Wed 4 Nov 2026.

    Previous session

    Session 01: Post-training Large Language Models changed the weights: instruction tuning, learning from preferences and reinforcement learning.