GSM8K test question 1Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?
The problem: How to ask LLM to answer the question *correctly*?
GSM8K: grade-school math (Cobbe et al., 2021).
A · From question to instruction
19 ways to ask LLMs -- From 2020 to 2026
From question to instruction
1Direct answer2020
2Instruction2020
3In-context learning2020
Think
4Chain-of-thought2022
5Best-of-n2021
6Self-consistency2022
Prompt engineering: from manual to automatic
7System prompt2023
8Prompt engineering2023
9 10Automatic prompt engineering2022–24
Tool use
11Tool use, function calling2023
12ReAct/Reflexion Agent2022–23
13MCP2024
Context engineering and agent
14Context engineering2025
15Agentic context engineering2025
16Develop agent2025
17Multi-agent systems2023–25
18Agent in long run2025
What's next?
19Agents improve agents2025–26
A · From question to instruction
The key milestones from a user's view
Image generated with ChatGPT (OpenAI), October 2026.
A · From question to instruction
1 · Direct question: end the prompt with "Answer:"
PromptJanet's ducks lay 16 eggs per day. She eats three … and bakes muffins … with four. She sells the remainder … for $2 per egg. How much does she make every day?Answer:
Example replyShe makes $18 every day at the market, since she has 9 eggs left to sell …
How it works
The model continues the text. The prompt stops at "Answer:", so the next words are the answer.
A · From question to instruction
1 · Multiple choice: put the options in the prompt
PromptJanet's ducks lay 16 eggs per day. She eats three … and bakes muffins … with four. She sells the remainder … for $2 per egg. How much does she make every day?A) 14 B) 18 C) 22 D) 26
Answer:
Example replyB
Benchmarks such as MMLU ask four-option questions (A–D) and read one letter (Hendrycks et al., 2021).
A · From question to instruction
1 · Direct answer: one user message, one reply
from openai import OpenAI
client = OpenAI() # reads OPENAI_API_KEY
MODEL = "gpt-4.1-mini"# a non-reasoning model
Q = "Janet's ducks lay 16 eggs per day. ..."# the full questionPROMPT = Q + "\nAnswer:"# direct questionMC = Q + "\nA) 14 B) 18 C) 22 D) 26\nAnswer:"# multiple choice
r = client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": PROMPT}])
print(r.choices[0].message.content)
A · From question to instruction
1 · Direct answer: easy to use, hard to read
Limitation
Any form: "$18", "18 dollars", a paragraph. Hard to check.
Unclear task: answer, explain, or continue the story?
Failed example 1 · the answer is hard to read… so she makes $18 a day. That is $126 a week.
Reading the last number gives 126 ✗
Failed example 2 · the task is unclearHer neighbour Tom also keeps ducks, and …
It continues the story; no answer ✗
Next →2 · Instruction specifies the task and the answer format.
Failed examples written for this slide to show the two limitations; not from a model run.
A · From question to instruction
2 · Instruction: specify the task and the answer format
Fixes the reply had no fixed form, and the task was not stated.
PromptSolve the math word problem. Give only the final number in brackets, like [12].Janet's ducks lay 16 eggs per day. She eats three … and bakes muffins … with four. She sells the remainder … for $2 per egg. How much does she make every day?
Example reply[18]
Brown et al. (2020) call this zero-shot: a description of the task and no examples. Models trained to follow instructions made it reliable (InstructGPT; Ouyang et al., 2022).
A · From question to instruction
2 · Instruction: add one line, then parse the answer
import re
INSTR = ("Solve the math word problem. ""Give only the final number in brackets, like [12].")
r = client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": INSTR + "\n\n" + Q}])
text = r.choices[0].message.content # e.g. "[18]"m = re.search(r"\[(-?\d+)\]", text)# read the number
answer = int(m.group(1)) if m else None
The format makes the answer machine-readable: we can now score many questions automatically.
A · From question to instruction
2 · Instruction: says what to do, not how to do it
Where it is used
Almost every request to a chat model
Extraction ("return JSON with fields …"), classification labels, summaries
Limitation
An instruction says what to do, not how.
Some tasks are hard to define via description.
Next →3 · In-context learning shows solved examples.
A · From question to instruction
3 · In-context learning: show how to solve
Fixes the instruction said what to do, but not how.
PromptSolve the math word problem. Give only the final number in brackets, like [12].
Q: Tom has 3 boxes of 4 pens and gives away 5. How many are left? A: [7]Q: A shop has 20 apples and sells 8, then 6. How many are left? A: [6]Q: Janet's ducks lay 16 eggs a day … How much does she make every day? A:
Example reply[18]
The weights do not change: the model learns the task from k examples in its input. Brown et al. (2020) named this in-context learning (zero-, one- and few-shot).
A · From question to instruction
3 · In-context learning: examples as earlier turns
SHOTS = [("Tom has 3 boxes of 4 pens. He gives away 5. ...", "[7]"),("A shop has 20 apples. It sells 8, then 6. ...", "[6]"),
]
messages = [{"role": "user", "content": INSTR}]
for q, a in SHOTS:# k demonstrationsmessages += [{"role": "user", "content": q},{"role": "assistant", "content": a}]
messages += [{"role": "user", "content": Q}]
r = client.chat.completions.create(model=MODEL, messages=messages)
Each example is a user turn and an assistant turn. The model copies the pattern.
A · From question to instruction
3 · In-context learning: copies the pattern, not the reasoning
4 · Chain-of-thought: the model writes the steps first
Fixes the examples showed only answers, so the model jumped to the answer.
PromptSolve the math word problem. Give only the final number in brackets, like [12].
Q: Tom has 3 boxes of 4 pens. He gives away 5. How many pens are left?
A: 3 × 4 = 12 pens. 12 − 5 = 7. [7]
Q: Janet's ducks lay 16 eggs a day … How much does she make every day?
A:
Example replyShe has 16 − 3 − 4 = 9 eggs. 9 × 2 = 18. [18]
PaLM 540B on GSM8K: 17.9% → 56.9% with chain-of-thought examples (Wei et al., 2022). "Let's think step by step" gives the effect with no examples (Kojima et al., 2022). For the first time, the model "thinks" in text.
A · Think
4 · Chain-of-thought: worked examples, or one sentence
SHOTS_COT = [
("Tom has 3 boxes of 4 pens. He gives away 5. ...",
"3 x 4 = 12 pens. 12 - 5 = 7. [7]"), # steps, then answer
]
# few-shot CoT: build messages from SHOTS_COT as in method 3# zero-shot CoT: no examples, one sentence at the end
messages = [{"role": "user",
"content": INSTR + "\n\n" + Q +
"\nLet's think step by step."}]
r = client.chat.completions.create(model=MODEL, messages=messages)
Same call; the prompt now asks for the steps before the bracketed answer.
A · Think
4 · Chain-of-thought: one chain, one chance
Where it is used
Math, code, science questions, planning
Reasoning models (o1, 2024) are trained to write the chain themselves: Part B
Limitation
We sample one chain. One wrong step (16 − 3 − 4 = 8) makes the answer wrong.
Small models gain little from CoT (Wei et al., 2022).
Next →5 · Best-of-n samples many chains.
A · Think
5 · Best-of-n: try n times, and one try often wins
Fixes one chain gave one chance.
Assume an verifier is available.
Best-of-N
Generate N candidates: \(y_1,\dots,y_N \overset{\text{i.i.d.}}{\sim} \pi_\theta(\cdot\mid x)\)
Score and select the best: \(y^* = \arg\max_{y_i,\,i=1,\dots,N} r_\phi(x,y_i)\)
Codex solved 28.8% of HumanEval problems with one sample, and 70.2% with 100 samples per problem (Chen et al., 2021).
A trained verifier that picks the best of n gave about the gain of a 30× larger model (Cobbe et al., 2021).
A · Think
5 · Best-of-n: ask for n samples in one call
r = client.chat.completions.create(
model=MODEL, messages=messages,
n=8, temperature=1.0) # 8 different samples
answers = [parse(c.message.content) for c in r.choices]
best = max(r.choices,# needs a verifierkey=lambda c: verifier_score(Q, c.message.content))
n returns n independent samples; temperature above 0 makes them differ.
A · Think
5 · Best-of-n: needs something that knows the right answer
Where it is used
Code generation: run the unit tests, keep a program that passes
Math with a checker; image generation (pick the best image)
Limitation
To know which try won, you need a checker: the gold answer, tests or a trained verifier.
For a new math question there is no gold answer.
Next →6 · Self-consistency lets the tries check each other.
A · Think
6 · Self-consistency: take the answer most chains agree on
Fixes best-of-n needed a checker that knows the answer.
PaLM 540B on GSM8K: 56.5% with one chain → 74.4% with a majority vote over 40 sampled chains (Wang et al., 2023). No checker and no training: the chains vote.
7 · System prompt: the developer's set-up gets its own role
Fixes the set-up was mixed into the user's message on every call.
systemYou are a careful math tutor. Solve step by step. End with the final number in brackets, like [12].Example: Tom has 3 boxes of 4 pens … [7]
userJanet's ducks lay 16 eggs a day … How much does she make every day?
Example reply16 − 3 − 4 = 9; 9 × 2 = 18. [18]
The system prompt is reused with every new question.
The ChatGPT API (1 Mar 2023) takes a list of messages with roles: system, user, assistant (OpenAI, 2023).
Models are trained to rank the system instructions above the user's (Wallace et al., 2024).
A · Prompt engineering: from manual to automatic
7 · System prompt: set once, reused on every call
SYSTEM = """You are a careful math tutor.
Solve the problem step by step.
End with the final number in brackets, like [12].
Example: Tom has 3 boxes of 4 pens. He gives away 5.
3 x 4 = 12. 12 - 5 = 7. [7]"""
messages = [{"role": "system", "content": SYSTEM},# developer
{"role": "user", "content": Q}] # changes
r = client.chat.completions.create(model=MODEL, messages=messages)
Newer OpenAI models name this role "developer"; the idea is the same.
A · Prompt engineering: from manual to automatic
7 · System prompt: a compound instruction, written by hand
Where it is used
Every chat product: tutors, customer service, coding assistants
Published system messages, e.g. the one of Codex (OpenAI, 2025)
Limitation
A good system prompt comes from trial and error.
Small changes in wording change accuracy, and the best wording differs per model.
Next →8 · Prompt engineering treats the prompt as a program.
A · Prompt engineering: from manual to automatic
8 · Prompt engineering: prompt as program
Fixes one hand-made prompt, not tested in a system way.
Practice: 1. write the prompt as a template, 2. test each change on many questions, 3. keep the better version.
A · Prompt engineering: from manual to automatic
ChatGPT's system prompt: identity, date, then rules per tool
system · ChatGPT (GPT-4) · 10 Nov 2023You are ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture.
Knowledge cutoff: 2023-04
Current date: 2023-11-10
Image input capabilities: Enabled
# Tools
## python
When you send a message containing Python code to python, it will be executed in a stateful Jupyter notebook environment. Python will respond with the output of the execution or time out after 60.0 seconds. The drive at '/mnt/data' can be used to save and persist user files. Internet access for this session is disabled. Do not make external web requests or API calls as they will fail.
continued## dalle
// 1. The prompt must be in English. Translate to English if needed.
// 3. DO NOT ask for permission to generate the image, just do it!
// 6. Do not create images of politicians or other public figures. Recommend other ideas instead.
// 7. Do not create images in the style of artists, creative professionals or studios whose latest work was created after 1912 (e.g. Picasso, Kahlo).
…
## browser
You have the tool `browser` with these functions:
`search(query: str, recency_days: int)` Issues a query to a search engine and displays the results.
`click(id: str)` Opens the webpage with the given id, displaying it.
…
Never write a summary with more than 80 words.
Do not repeat lyrics obtained from this tool.
ALWAYS include multiple distinct sources in your response, at LEAST 3-4.
A copy reported by users, not an official OpenAI document (LouisShark/chatgpt_system_prompt, GitHub, 10 Nov 2023). Lines are copied; "…" marks omissions.
A · Prompt engineering: from manual to automatic
8 · Prompt engineering: a template, tested on a dev set
def build_prompt(question, shots, fmt="[12]"):
lines = ["You are a careful math tutor.",
"Think step by step, then check each step.",
f"End with the final number in brackets, like {fmt}."]
for q, a in shots:
lines += [f"Q: {q}", f"A: {a}"]
return"\n".join(lines + [f"Q: {question}", "A:"])
def accuracy(build):# 100 dev questionsreturn sum(ask(build(q)) == gold for q, gold in DEV) / len(DEV)
Change one line, measure on the dev set, keep the version that scores higher.
A · Prompt engineering: from manual to automatic
8 · Prompt engineering: a person searches by hand
Where it is used
Prompt templates kept in code, with versions and tests
Prompting guides from the model vendors
Limitation
A person writes and tests each version.
The space of wordings is huge, and the best wording differs per model.
Next →9 · APE lets an LLM write the prompt.
A · Prompt engineering: from manual to automatic
9 · APE: an LLM writes the instruction, a score picks it
Fixes a person had to write and test every prompt.
Best found"Let's work this out in a step by step way to be sure we have the right answer."
APE (Automatic Prompt Engineer) found this prompt; it beat "Let's think step by step". Its instructions matched or beat human ones on 24/24 instruction-induction and 17/21 BIG-Bench tasks (Zhou et al., 2022).
The candidates and scores in the diagram are illustrations.
A · Prompt engineering: from manual to automatic
9 · APE: sample candidate prompts, keep the best on a dev set
req = ("Here are questions and their answers:\n" + PAIRS +
"\nWrite the instruction that produced these answers.")
r = client.chat.completions.create(
model=MODEL, n=20, temperature=1.0, # 20 candidates
messages=[{"role": "user", "content": req}])
cands = [c.message.content for c in r.choices]
def score(instr):# dev-set accuracyreturn sum(ask(instr, q) == g for q, g in DEV) / len(DEV)best = max(cands, key=score)
The LLM proposes; the dev set decides.
A · Prompt engineering: from manual to automatic
9 · APE: one round of guessing, no learning from scores
Where it is used
Automatic prompt search for classification, extraction and math
Prompt optimizers in libraries such as DSPy (Khattab et al., 2023)
Limitation
APE samples candidates and keeps the best. It does not learn from the scores which direction helps.
It uses no feedback about why a prompt failed.
Next →10 · OPRO and TextGrad use the scores and the feedback.
A · Prompt engineering: from manual to automatic
10a · OPRO: the LLM reads scored prompts, writes a better one
Fixes APE did not learn from the scores.
Best found (PaLM 2-L, GSM8K)"Take a deep breath and work on this problem step-by-step."
OPRO (Optimization by PROmpting): optimized prompts beat human-designed ones by up to 8% on GSM8K and up to 50% on Big-Bench Hard (Yang et al., 2024). Scores in the meta-prompt here are illustrations.
Diagram after Figure 2 of Yang et al. (2024), redrawn. Green: an LLM. Yellow: what OPRO adds. Scores are illustrations.
A · Prompt engineering: from manual to automatic
10a · OPRO: a loop of propose and score
history = [("Let's think step by step.", score(s0))]
for step in range(20):
ranked = sorted(history, key=lambda x: x[1])# worst first
meta = "Instructions and their scores:\n" + "\n".join(
f"{p} score={s:.2f}"for p, s in ranked)
meta += "\nWrite a new instruction with a higher score."new = ask_llm(meta)# the LLM proposeshistory.append((new, score(new)))# we measure
best = max(history, key=lambda x: x[1])
The LLM is the optimizer; the dev-set score is the objective.
A · Prompt engineering: from manual to automatic
10b · TextGrad: written feedback works like a gradient
Fixes a score says how good a prompt is, but not what to change.
Loss, in wordsa critic LLM: "Step 2 subtracts the muffin eggs twice; state each use once."
TextGrad (James Zou's group) applies "backpropagation" to text. On LeetCode-Hard it boosted gpt-4o and the best earlier method by 20% (Yuksekgonul et al., 2024).
Diagram after Figure 1 of Yuksekgonul et al. (2024), redrawn. Solid: forward pass. Dashed: backward pass.
A · Prompt engineering: from manual to automatic
10b · TextGrad: variables, a text loss, backward, step
import textgrad as tg
tg.set_backward_engine("gpt-4o", override=True) # the critic
prompt = tg.Variable(SYSTEM, requires_grad=True,
role_description="system prompt for math")
model = tg.BlackboxLLM("gpt-4o-mini", system_prompt=prompt)
opt = tg.TGD(parameters=[prompt])
q = tg.Variable(Q, requires_grad=False, role_description="question")
loss_fn = tg.TextLoss("Check each step. The answer must be 18.")loss = loss_fn(model(q))loss.backward()# feedback in wordsopt.step()# rewrite the prompt
Library textgrad (MIT); names checked against version 0.1.x.
A · Prompt engineering: from manual to automatic
10 · LLM optimizers: better text, but still only text
Where it is used
Prompt optimization in apps; DSPy optimizers
TextGrad also optimizes code, molecules and treatment plans (Yuksekgonul et al., 2024)
Limitation
Methods 1–10 change only the text. The model still computes in its head and can make arithmetic errors on big numbers.
It cannot look up facts after its training cut-off, and it cannot act.
Next →11 · Tool use lets the model call a function.
11 · Tool use: the model asks for a function, our code runs it
Fixes the model computed in its head, and could not act.
Toolformer taught a model to insert calls to a calculator and search (Schick et al., 2023); PAL let the model write Python that an interpreter runs (Gao et al., 2023). OpenAI added function calling to its API on 13 Jun 2023 (OpenAI, 2023).
Flow of function calling, after OpenAI (2023), redrawn. Green: written by the LLM. Yellow: what tool use adds.
A · Tool use
11 · Tool use: describe the function, run the call, send the result
ReAct interleaves reasoning traces and actions. With one or two examples it beat imitation and reinforcement-learning methods by 34% (ALFWorld) and 10% (WebShop) absolute success (Yao et al., 2023).
A · Tool use
12a · ReAct: the agent loop in ten lines
messages = [{"role": "system", "content": SYSTEM},
{"role": "user", "content": Q}]
for step in range(MAX_STEPS):# the agent loop
r = client.chat.completions.create(
model=MODEL, messages=messages, tools=tools)
msg = r.choices[0].message
messages.append(msg)
ifnot msg.tool_calls:# no action: final answerbreakfor call in msg.tool_calls:# act, then observemessages.append({"role": "tool", "tool_call_id": call.id,"content": run_tool(call)})
The model decides each next action; the loop stops at a final answer or after MAX_STEPS.
A · Tool use
12b · Reflexion: a failed try becomes a lesson in memory
the LLM generatesinput from outsidememory
ALFWorld example after Figure 1 of Shinn et al. (2023); the text is shortened and the diagram is redrawn. Green: written by the LLM. Grey: given from outside.
A · Tool use
12b · Reflexion: after a failed try, write a lesson, try again
Fixes the loop did not learn from a failed try.
Try 1Answer [26] → checker: wrong
Self-reflection, kept in memory"I subtracted only the breakfast eggs (16 − 3 = 13). List every use of the eggs before multiplying."
Try 2, with the lesson in the promptAnswer [18] → checker: correct
"Verbal reinforcement learning": the reward becomes text in memory, and the weights do not change. Reflexion reached 91% pass@1 on HumanEval, against 80% for GPT-4 (Shinn et al., 2023). Shunyu Yao co-authored both ReAct and Reflexion.
A · Tool use
12b · Reflexion: a loop over tries, with lessons as memory
lessons = []for trial in range(3):
ans = solve(Q, extra="\n".join(lessons))# ReAct or CoTok, feedback = check(ans)# tests, a checker, the envif ok:
breaklessons.append(ask_llm(f"Your answer {ans} failed: {feedback}. ""Write one lesson for the next try."))
The evaluator must give a signal: unit tests, a checker or the environment.
QA with search (HotpotQA), web shopping (WebShop), household tasks (ALFWorld)
Coding agents: run the tests, reflect, fix
Limitation
Every app writes its own tool wrappers: search, files, databases, each for each model.
N apps × M tools: the same integration is written again and again.
Next →13 · MCP makes one standard plug for tools.
A · Tool use
13 · MCP: one standard plug between agents and tools
Fixes each app wrote its own wrapper for each tool.
Anthropic released the Model Context Protocol on 25 Nov 2024 as an open standard (Anthropic, 2024). Write a tool server once, and every MCP client can use it.
A · Tool use
13 · MCP: a tool server, and an API call that uses it
from mcp.server.mcpserver import MCPServer # mcp 2.xserver = MCPServer("math")@server.tool()def calculator(expression: str) -> str:
"""Evaluate an arithmetic expression exactly."""return str(safe_eval(expression))
server.run(transport="streamable-http") # serve it
r = client.responses.create(model="gpt-5.4-mini", input=Q,
tools=[{"type": "mcp", "server_label": "math","server_url": MATH_SERVER_URL,"require_approval": "never"}])
Server: the official Python SDK (mcp 2.x). Client: the Responses API lists and calls the server's tools.
A · Tool use
13 · MCP: many tools, and a context that fills up
Where it is used
Coding agents connected to GitHub, databases and file systems
Connectors in chat apps: mail, calendars, drives
Limitation
An agent now has many tools, a long history and many documents. The context window fills up.
More text is not better: at each step the model needs the right information, not all of it.
Next →14 · Context engineering chooses what the model reads.
14 · Context engineering: build the context for each step
def build_context(task, history, memory, k=3):
return (
[{"role": "system",
"content": SYSTEM + "\nLessons:\n" + memory.top(task)}]+ similar_solved(task, k)# chosen per question+ compact(history, max_tokens=4000)# summarize old steps
+ [{"role": "user", "content": task}])
r = client.chat.completions.create(
model=MODEL, tools=tools,
messages=build_context(Q, history, memory))
Each model call gets a context built for that step.
A · Context engineering and agent
14 · Context engineering: designed once, by a person
Where it is used
Coding agents: which files and which tool results to show
Long chats: summarize (compact) old turns; retrieval chatbots
Limitation
A person writes the context rules once.
The agent does not improve them from its own successes and failures.
Next →15 · Agentic context engineering lets the agent edit its own context.
A · Context engineering and agent
15 · ACE: the agent edits its own playbook
Fixes a person wrote the context rules, and they did not improve.
A delta item the Curator adds (example)• List every use of the eggs before you multiply.
ACE grows a detailed playbook instead of rewriting a short prompt. This avoids "brevity bias" and "context collapse". +10.6% on agent tasks, with no weight updates (Zhang et al., 2025; James Zou is a co-author).
Diagram after Figure 4 of Zhang et al. (2025), redrawn. Green: an LLM. Yellow: what ACE adds. Dashed: the playbook, kept between questions.
A · Context engineering and agent
15 · ACE: generate, reflect, curate, repeat
playbook = [] # bullets: strategies, pitfallsfor q, gold in TRAIN:
trace = generate(q, context=playbook) # Generatorinsights = reflect(q, trace, gold)# Reflectorplaybook = curate(playbook, insights)# Curator:# add, merge or drop bullets
answer = generate(Q, context=playbook) # use it
Each role is an LLM call with its own prompt; code: ace-agent/ace (Apache-2.0).
A · Context engineering and agent
15 · ACE: a better context, but a hand-written loop
Where it is used
Agent benchmarks (AppWorld) and finance tasks (Zhang et al., 2025)
Rules files that agents keep for a project, e.g. AGENTS.md (OpenAI, 2025)
Limitation
Each app still writes its own loop, tool handling and stop rules.
Real agents also need guardrails, hand-offs and traces, and these are easy to get wrong.
Next →16 · Agent design uses a library for the loop.
A · Context engineering and agent
16 · Agent design: instructions, tools and a loop, in a library
Fixes each app wrote its own loop and tool handling.
OpenAI released the Agents SDK on 11 Mar 2025 (OpenAI, 2025). The Runner is the ReAct loop of method 12, written once and tested.
Diagram after the agent loop in OpenAI (2025), redrawn. Yellow: what the Agents SDK adds.
A · Context engineering and agent
16 · Agent design: the same question with the Agents SDK
from agents import Agent, Runner, function_tool
@function_tool
def calculator(expression: str) -> str:
"""Evaluate an arithmetic expression exactly."""return str(safe_eval(expression))
solver = Agent(name="Solver", instructions=SYSTEM,tools=[calculator], model="gpt-5.4-mini")result = Runner.run_sync(solver, Q, max_turns=6)
print(result.final_output) # "[18]"
Names checked against openai-agents 0.23.1.
A · Context engineering and agent
16 · One agent: one context does every job
Where it is used
Customer-service and data-analysis agents
Limitation
One agent plans, solves and checks, all in one context.
Long tasks overflow that context, and nobody checks its work.
Next →17 · Multi-agent systems split the work.
A · Context engineering and agent
17 · Multi-agent systems: split the work, check each other
Fixes one agent did every job in one context.
AutoGen: agents that talk to each other (Wu et al., 2023). Debate between agents improves math and strategy reasoning (Du et al., 2024). Anthropic's research system — a lead agent with parallel subagents — beat a single agent by 90.2% on its internal eval, and used about 15× the tokens of a chat (Hadfield et al., 2025).
Diagram after the manager pattern in OpenAI (2025), redrawn. Yellow: what multi-agent systems add.
A · Context engineering and agent
17 · Multi-agent: agents used as tools by a manager
checker = Agent(name="Checker", tools=[calculator],instructions="Recompute the answer. Reply OK or the error.")
manager = Agent(name="Manager",
instructions=("Ask solve for an answer, then check it. ""If check finds an error, ask solve again."),
tools=[solver.as_tool(tool_name="solve",tool_description="Solve a math word problem."),checker.as_tool(tool_name="check",tool_description="Check a solution.")])
result = Runner.run_sync(manager, Q)
Agent.as_tool turns an agent into a tool of another agent; handoffs passes control instead.
A · Context engineering and agent
17 · Multi-agent systems: smart within a session
Where it is used
Deep-research products: a lead agent and searching subagents
Coding: planner, coder, reviewer
Limitation
Each session starts fresh: the agents forget what they learned yesterday.
Tasks of hours or days need memory and state across sessions.
Next →18 · Long-horizon agents keep memory.
A · Context engineering and agent
18 · Long-horizon agents: work for hours, keep memory
Fixes each session started fresh.
Our task, nowRun the solver on all 1,319 GSM8K test questions, sort the errors by type, write report.md.
Claude Code (24 Feb 2025): an agent in the terminal (Anthropic, 2025). Codex (16 May 2025): a cloud agent that works on many tasks in parallel (OpenAI, 2025). OpenClaw (open source, Nov 2025): "your assistant, on your devices, in your chats".
Diagram after Figures 1–3 of Packer et al. (2023), redrawn. Yellow: external memory.
A · Context engineering and agent
18 · Long-horizon: a session that persists on disk
from agents import SQLiteSession
session = SQLiteSession("gsm8k-study", "memory.db")# on disk
task = ("Run the solver on all 1,319 GSM8K test questions, ""sort the errors by type, and write report.md.")
result = Runner.run_sync(analyst, task, session=session,max_turns=200)# tomorrow: the same session id continues with its historyresult = Runner.run_sync(analyst, "Continue.", session=session)
SQLiteSession stores the conversation; the agent can also keep notes files through its tools.
A · Context engineering and agent
18 · Long-horizon agents: people still design them
Where it is used
Software engineering: Claude Code, Codex
Personal assistants on your own machine: OpenClaw
Limitation
People still design the agent: its prompt, tools and harness.
The agent does not improve its own design, and small errors add up over long tasks.
Next →19 · What's next: agents that improve agents.
Our task, nextFind a better prompt and harness for GSM8K overnight; measure on held-out questions.
Darwin Gödel Machine: an agent rewrites its own code and keeps an archive of variants; SWE-bench 20.0% → 50.0% (Zhang et al., 2025). autoresearch (Karpathy, Mar 2026): an agent changes the training code, trains for 5 minutes, keeps or drops the change, and repeats overnight; people write the program.md that gives it its context.
Diagram after Figure 1 of Zhang et al. (2025), redrawn. Green: an LLM. Yellow: what is new. The dark circle is the selected parent.
A · What's next?
19 · What's next: Sakana AI's self-improving AI research
best = evaluate(code) # held-out accuracy
notes = []
for i in range(N): # e.g. overnightpatch = agent_propose(code, notes)# an LLM edits the code
score = evaluate(apply(code, patch)) # fixed budget per runif score > best:# keep, or dropcode, best = apply(code, patch), scorenotes.append((patch.summary, score))# memory of tries
The same loop as OPRO (method 10), now over code instead of a prompt.
A · What's next?
19 · What's next: open problems
Where it is used
ML experiments (autoresearch), coding agents (Darwin Gödel Machine)
Agents that keep and edit their own playbooks (ACE)
Limitation
The agent optimizes the metric it is given, so a bad metric gives a bad agent.
Cost, safety and checking the results remain open.
Next → Parts B and C go deeper: reasoning at inference (B) and agents and harnesses (C).
A · What's next?
Methods 1–10 change the text that the model reads
Method
What it changes
Its limitation
Milestone 1 · Text completion
1 Direct answer
the question only
no fixed answer format
2 Instruction
adds the task and format
says what, not how
3 In-context learning
adds solved examples
answers only, no steps
Milestone 2 · Chat
4 Chain-of-thought
asks for the steps
one chain, one chance
5 Best-of-n
samples n tries
needs a checker
6 Self-consistency
votes over the tries
set-up mixed into user text
7 System prompt
gives the set-up its own role
written by hand
8 Prompt engineering
tests prompts as programs
a person searches
9 APE
an LLM writes the prompt
no learning from scores
10 OPRO, TextGrad
an LLM optimizes the prompt
still only text
A · What's next?
Methods 11–19 add tools, loops, context and memory
Method
What it changes
Its limitation
Milestone 3 · Agents
11 Tool use
calls a function
one step per round
12 ReAct, Reflexion
loops: act, observe, reflect
custom tool plumbing
13 MCP
one standard for tools
the context fills up
14 Context engineering
chooses what the model reads
rules made by a person
15 ACE
the agent edits its context
hand-written loop
16 Agents SDK
the loop as a library
one context does every job
17 Multi-agent
splits and checks the work
no memory across sessions
18 Long-horizon
memory on disk, hours of work
designed by people
19 Next
agents improve agents
the metric decides
Part A — A brief history of prompting · summary
How did prompting grow from one question to agents?
Key message: From text completion to agent, the tech of prompting is more and more complicated to resolve the complicated tasks.
From question to instruction: the question, an answer format, then solved examples (ICL).
Think: chain-of-thought; best-of-n needs a checker; self-consistency votes.
Prompt engineering: from manual to automatic: system prompts by hand, then LLMs that optimize prompts (APE, OPRO, TextGrad).
Tool use: function calls, ReAct and Reflexion, MCP.
Context engineering and agent: ACE, agent libraries, multi-agent systems, memory.
What's next? agents that improve agents.
Next, Part B: chain-of-thought in depth — longer, wider and deeper chains, and reasoning models.
Part B
Prompt to think
The model writes more before it answers: reasoning models, chain-of-thought prompting, search over many chains, and revision.
B · Chain-of-thought and reasoning models
Reasoning: test-time scaling
OpenAI o1 (the first large-scale reasoning model) started to achieve impressive performance across various challenging reasoning tasks in math, coding, STEM, etc.
Chart redrawn from the numbers in OpenAI (2024), Appendix A; expert human 69.7 is read from that post's chart. Solid bars: one answer (pass@1); light bars: majority vote of 64 answers (cons@64). AIME: a US maths contest. Codeforces: programming contests. GPQA Diamond: hard science questions written by experts.
B · Chain-of-thought and reasoning models
Reasoning: test-time scaling
O3 achieved 87.5% accuracy on ARC-AGI, with >$1k test-time cost to solve each task
Other existing LLMs achieved <25% accuracy if no special inference-time techniques were used.
But the gain gets smaller. o1 low → medium → high: 25% → 31% → 32%.
Chart redrawn from Chollet (2024); costs read from its log axis. "Tuned": trained on 75% of the ARC-AGI public training set.
B · Chain-of-thought and reasoning models
Reasoning behind ChatGPT view
B · Chain-of-thought and reasoning models
Core idea: trigger the LLM to generate long chain-of-thought (CoT)
Thought
Approaches
Few-shot CoT prompting
Instruction prompting
Instruction tuning
Reinforcement learning
How to scale token budget effectively to improve performance?
B · Chain-of-thought and reasoning models
Background: standard prompting
Exemplar
Before the advancement in post-training techniques, standard prompting performance is poor on reasoning benchmarks.
Issue: standard few-shot exemplars only provide information on the final solution format, but not the rationale to derive the solution.
Table: Kojima et al. (2022), Table 1; text-davinci-002 (GPT-3). Each cell: accuracy with a task-specific prompt to extract the answer / with "The answer is".
B · Chain-of-thought and reasoning models
But zero-shot CoT is still worse than few-shot CoT
How to improve CoT performance w/o manually labeling exemplars?
Examples are not always there. How to improve zero-shot CoT?
B · Chain-of-thought and reasoning models
Analogical prompting: instruct the LLM to generate exemplars
Prompt the LLM to first recall relevant exemplars, before solving the test problem.
Model InputQ: What is the area of the square with the four vertices at (-2, 2), (2, -2), (-2, -6), and (-6, -2)?
# Instruction:
## Recall relevant exemplars:
## Solve the initial problem:
Model Output# Relevant exemplars:
Q: What is the area of the square with a side length of 5?
A: The area of a square is found by squaring the length of its side. So, the area of this square is 5^2 = 25. …
Analogical prompting: instruct the LLM to generate exemplars (cont.)
Model Output# Solve the initial problem:
To find the area of the square, we need to find the side length. The length is … \sqrt{(2 - (-2))^2 + (-2 - 2)^2} = \sqrt{32}. So, the area of the square is (\sqrt{32})^2 = 32.
Example: Yasunaga et al. (2024), Fig. 1 (right), written out; the model writes LaTeX, shown as it wrote it.
B · Chain-of-thought and reasoning models
Analogical prompting: instruct the LLM to generate exemplars
Benefits
Exemplars are self-generated by LLMs, no manual labeling.
Humans are not explicitly given demonstrations every time for a new task
Instead, humans intrinsically recall from past relevant experience
"Do you know a related problem?"
George Pólya, How to Solve It (1945)
, which shows the book's cover and a page; not reproduced here. The question is the heading of that passage (Pólya, 1945).
B · Chain-of-thought and reasoning models
Example: Codeforces (prompt)
Our Prompt: Self-generate Exemplars + KnowledgeYour goal is to write Python3 code to solve competitive programming problems. Given a problem, explain the core concepts in it and provide other relevant problems. Then solve the original problem.
# Problem:
You are given a sequence a_1, a_2, …, a_n. Each element of a is 1 or 2. Find out if an integer k exists so that the following conditions are met. 1 ≤ k ≤ n-1, and a_1 ⋅ a_2 … a_k = a_{k+1} … a_n. If there exist multiple k that satisfy the given condition, print the smallest.
The prompt is typed out in full from Yasunaga et al. (2024), Fig. 3 (top), over three slides. Codeforces: a website for programming contests.
B · Chain-of-thought and reasoning models
Example: Codeforces (prompt) (cont.)
# Instruction:## Algorithms:
Identify the core concepts or algorithms used to solve the problem.
## Tutorial:
Write a tutorial about these algorithms.
## Example Problems:
Provide three examples of relevant competitive programming problems that involve these algorithms. For each problem, describe the problem, explain the solution in detail, and then write the correct Python3 code.
The prompt is typed out in full from Yasunaga et al. (2024), Fig. 3 (top), over three slides. Codeforces: a website for programming contests.
B · Chain-of-thought and reasoning models
Example: Codeforces (prompt) (cont.)
## Python3 code to solve the original problem:
- Explanation of the solution:
- Python3 code to solve the problem:
Besides exemplars, the LLM can also self-generate high-level knowledge
The generated knowledge complements the problems with broader insights
The prompt is typed out in full from Yasunaga et al. (2024), Fig. 3 (top), over three slides. Codeforces: a website for programming contests.
Output by GPT3.5-turboSelf-generated Knowledge## Algorithms: The core algorithm for solving this problem is the prefix product algorithm.
## Tutorial: The prefix product algorithm is used to calculate the product of all the elements in a prefix of an array. The algorithm works by initializing a variable "prefix" to 1, and then iterating through the array from left to right. At each iteration, the current element is multiplied with the prefix, and the result is stored in the prefix variable. This way, the prefix variable keeps track of the product of all the elements in the prefix of the array. The time complexity of this algorithm is O(n), where n is the size of the input array. ......
Self-generated Exemplars## Example Problems:1. Problem: Given an array of integers, find the product of all the elements in each subarray of size k.
Solution: We can use the prefix product algorithm to solve this problem. We initialize a prefix variable to 1, and then iterate through the array from left to right. At each iteration, we multiply the current element with the prefix, ......
def product_of_subarrays(arr, k):
prefix = 1
result = []
for i in range(len(arr)):
prefix = prefix * arr[i]
if i >= k-1:
result.append(prefix)
prefix = prefix // arr[i-k+1]
return result
2. Problem: Given an array of integers, find the longest subarray with a product less than or equal to a given target.
Solution: ......
3. Problem: Given an array of integers, find the smallest k such that the product of all the elements in the prefix of the array is equal to the product of all the elements in the suffix of the array.
Solution: ......
## Python3 code to solve the original problem:
Explanation of the solution:
- Initialize prefix and suffix variables to 1, and k to -1.
- Calculate the product of all the elements in the sequence using the prefix product algorithm.
- Iterate through the sequence from right to left.
- Update the prefix and suffix variables.
- If the prefix is equal to the suffix, update k to the current index.
- Print the value of k.
Example: Codeforces (solving the initial problem) (cont.)
t = int(input())
for _ in range(t):
n = int(input())
arr = list(map(int, input().split()))
prefix, suffix = 1, 1
k = -1
for i in range(n):
prefix = prefix * arr[i]
for i in range(n-1, 0, -1):
prefix = prefix // arr[i]
suffix = suffix * arr[i]
if prefix == suffix:
k = i
print(k)
Analogical prompting outperforms 0-shot CoT and manual few-shot CoT
Chart redrawn from Yasunaga et al. (2024), Tables 1–3. Codeforces: GPT-3.5-turbo-16k; analogical = self-made knowledge and exemplars (exemplars only: 13%). Temporal sequences: a BIG-Bench task on what time a person could have gone somewhere.
B · Chain-of-thought and reasoning models
Stronger LLMs are better analogical reasoners
GSM8K for math reasoning
Weaker LLMs benefit less from analogical prompting, though it does not hurt the zero-shot performance
With stronger LLMs, analogical prompting outperforms CoT with manually-designed or retrieved exemplars
The generated CoT is more tailored to the underlying LLM
Table: Yasunaga et al. (2024), Table 4 (GSM8K). Retrieved: the 5 most similar solved problems from the training set.
B · Chain-of-thought and reasoning models
What CoT brings into LLM reasoning
Chain-of-thought prompting: variable computation of the thought process adapting to tasks of different difficulty levels
More complex questions -> more reasoning steps in the chain-of-thought
Reasoning strategies enabled by CoT
Decomposition
Planning
…
We can explicitly instruct the LLM with the desired reasoning strategies for problem solving
The next two slides show why more tokens give more computation.
B · Chain-of-thought and reasoning models: the math
Without CoT, the depth L limits the serial steps
One pass makes one token. Inside a pass, a chain of steps goes up at most L layers.
A thought token goes back in as input, and the next pass starts again at layer 1. Each token adds up to L more steps.
✋ Your turn
Parity: is the number of 1s odd? Let each thought token be "the parity so far". Write the 4 tokens.
Drawn for this lecture after Li et al. (2024), §1 and the proof sketch of their Theorem 3.3 (writing the output back "resets the depth"). Longest chain with T thought tokens ≤ L × (T + 1) layer steps (computed). See also Zhou (2025), p. 5.
B · Chain-of-thought and reasoning models: the math
What CoT brings into LLM reasoning
T thought tokens let a shallow Transformer run T gates
Each thought token computes one gate. Attention (blue) reads the gate's inputs; the MLP computes it (orange).
Theorem 3.3. T(n) gates need T(n) CoT steps, at constant depth:
\(\mathsf{SIZE}[T(n)] \subseteq \mathsf{CoT}[T(n), \log n, 1]\)
Theorem 3.1. No CoT, constant precision: only \(\mathsf{AC}^0\) (constant-depth circuits). Parity is not in it.
Li et al. (2024), Theorems 3.1 and 3.3, Fig. 2 (circuit redrawn): n = input length, embedding size O(log n), constant bits of precision. Zhou (2025), p. 5: "O(T) intermediate tokens". "No AC0 circuits can decide the parity": their p. 8. Parity tokens and gate values computed.
B · Chain-of-thought and reasoning models
How to improve reasoning?
Least-to-most prompting: easy-to-hard generalization via decomposition
B · Chain-of-thought and reasoning models
Summary
Chain-of-thought generation: variable computation of the thought process adapting to tasks of different difficulty levels
How to improve the CoT performance at inference time
Few-shot prompting with labeling of thoughts
Instruction prompting to trigger CoT generation
Instruct the LLM to automate the prompt design
Note: the best practice to interact with LLMs evolves over time
The principles of how to discover good prompting strategies for reasoning hold true
Encourage longer CoT for complex tasks
Support reasoning strategies required for the task
The deck's second half starts on p. 38.
B · Chain-of-thought and reasoning models
What is missing so far?
We should not limit the LLM to generate only one solution per problem!
Exploring multiple branches allows the LLM to recover from mistakes in a single generation
Generate multiple candidate solutions per problem
Generate multiple potential next reasoning steps given the current (partial) thought
Challenge: how to select the best response from multiple candidates?
In most cases, we do not have an oracle scorer at inference time
B · Chain-of-thought and reasoning models
Self-consistency: take the answer most chains agree on
Fixes best-of-n needed a checker that knows the answer.
PaLM 540B on GSM8K: 56.5% with one chain → 74.4% with a majority vote over 40 sampled chains (Wang et al., 2023). No checker and no training: the chains vote.
B · Chain-of-thought and reasoning models
Self-Consistency: select the response with the most consistent final answer
Note: the selection is only based on the final answer, the reasoning paths do not need to be the same across different sampled responses.
General principle of how to design effective reasoning techniques
The Bitter Lesson from Richard Sutton (again) is an important guideline for designing reasoning techniques, including both inference-time and training-time algorithms.
“One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great.”
“We want AI agents that can discover like we can, not which contain what we have discovered. Building in our discoveries only makes it harder to see how the discovering process can be done.”
Key message: A reasoning trajectory cannot be trained token by token, thus prompting is the essential tool to steer the reasoning.
Reasoning models such as o1 think first, and their accuracy grows with more compute at inference.
CoT prompting: worked examples or “Let’s think step by step” make a large model write steps first; small models gain little.
Chen's three steps: one longer chain from a better prompt; many chains, then a vote or a verifier; a revised chain, which needs external feedback.
Limits: CoT helps mainly on math and symbolic tasks, and the chain need not show the real reason.
Next, Part C: on many of these tasks a tool does better, so we give the model tools and a loop.
Invited talk · on Zoom
Scaling synthetic data and environments
Zeyu Qin
HKUST Member of technical staff, Kimi
The Zoom link is on Canvas.
About the speaker
Fourth-year Ph.D. candidate in Computer Science and Engineering at HKUST, advised by Minhao Cheng and Shuai Wang.
Member of technical staff at Kimi: RL and long-horizon agents (Kimi K2 thinking, K2.5, K2.6, K3).
Research: RL, long-horizon agents, reasoning, scalable oversight; scaling laws of synthetic data (COLM 2025).
Bio and photo from the speaker's homepage, alan-qin.github.io (read 20 Oct 2026).
NUS guest lectureWed 28 Oct 2026 · 8:30 PM SGT
Working with Increasingly Capable AI for Science
Notes on autoresearch and human–AI collaboration
AI agents can now take a research project from hypothesis to paper, and they are getting better fast. What can they actually do, and do they have research taste? How should scientists and agents split the work? And what should research look like when its readers are agents? I will share what we have learned building and measuring research agents, and my views on where autoresearch is going.
MeasureResearch ability and taste, on real papersResearchTree
CollaborateHumans at the forks, agents on the stepsOrchestra · AI-Research-SKILLs
BuildResearch for agents, not human readersAgent-Native Research Artifacts · NeurIPS 2026
NextSimulated labs for training agentsSimLab · in progress
Zechen ZhangHarvard University · Founder, Orchestra Research
Zechen builds tools for doing research with AI agents. He leads ResearchTree, is senior author of Agent-Native Research Artifacts (NeurIPS 2026), and created AI-Research-SKILLs, an open-source skill library with 13,000+ GitHub stars.
Content and style after the talk poster for this lecture (28 Oct 2026).
BREAK
10 minutesBack at 19:54
Coming up in Part C, Prompt as System: Agents and harness — the loop around the model, how to measure an agent, and how to keep it safe.
Part C
Prompt as System: Agents and harness
When one call is not enough: the model in a loop, the system around it, and how to measure and secure it.
Measure, change one thing at a time, and keep what holds up
As models evolve, our understanding of how to best use them and guidance around best practice evolve as well. That makes any one architecture less important than the principle behind it: measure, change one thing at a time, and keep what holds up.
This is one of those cases. As models get more capable and agents get longer running, more of your subagents can be skills and tools. You don’t have to make that move in one step, though. Build a small end-to-end eval, fold in one subagent, and keep it as a subagent only if the folded version scores worse, then rinse and repeat.
After Sarkis (2026), “Tools to accelerate this work”.
C · Multi-agent: When to invoke more agents?
Outline of the article
Agent building failure modes for large organizations
Using an agent loop with progressive disclosure
How to fold subagents into the main agent loop
Task-completion evals
Reducing subagents over time
Combining domain agents
Subagents called as tools
Folding subagents into the main agent via skills and tools
How several teams can contribute to a single agent
MCP tools
Skills
Knowledge bases
Tools to accelerate this work
Section headings of Sarkis (2026), in the article's order.
Part C — Prompt as System: Agents and harness · summary
What do the loop and the harness add?
Key message: Harness is essential when an agent becomes a complicated system with constraints, tools, failure control, etc.
Agents today finish long real tasks, such as a C compiler, and also make costly errors, such as deleting an inbox.
An agent is a model in a loop: the model writes a tool call, the harness runs it, and the result goes back into the context.
Harness or training: each agent capability can come from harness engineering or from training. An agent is a whole system. mini-swe-agent shows that a harness is little code.
Context and safety: the context grows with every call and can be compacted; threat models are a malicious user, prompt injection, agent error and improper means; a sandbox, fine-grained tokens and monitoring limit harm.
Next: the strengths and weaknesses of each setting, then the homework.
Close · Strengths and weaknesses
Prompting and CoT: no training, but fragile
Setting
Strength
Weakness
Prompting
No training. Works with any model you can call.
The prompt's format alone changes accuracy by up to 76 points (Sclar et al., 2024).
Improves with more training and more thinking time (OpenAI, 2024).
Few-shot prompts make DeepSeek-R1 worse (Guo et al., 2025). Reasoning models state a hint they used in 25% (Claude 3.7 Sonnet) to 39% (DeepSeek R1) of cases (Y. Chen et al., 2025).
Agent + harness
Facts and actions beyond the weights. Tool feedback can correct errors.
The scenario: patient and counsellor, customer and support agent, interviewer and candidate, tutor and student, two neighbours
The turn limit and the stop word: when does the talk end?
The history: all earlier messages, or only the last two?
A guardrail on one agent's message
One sentence of an agent's instructions
Check before you conclude
Same scenario, settings and number of runs for every version
Is the difference larger than the spread across your 3 runs?
Did an agent leave its role? Read the transcript; the count is only a hint
A result that looks too good: read the whole conversation
Optional practice, no course credit. The patient and the counsellor are simulated: a class role-play, not medical advice.
Lecture 2, session 02 · summary
How much can prompting change?
A · A brief history of prompting. From text completion to agent, the tech of prompting is more and more complicated to resolve the complicated tasks.
B · Prompt to think. A reasoning trajectory cannot be trained token by token, thus prompting is the essential tool to steer the reasoning.
C · Prompt as System: Agents and harness. Harness is essential when an agent becomes a complicated system with constraints, tools, failure control, etc.
Where we got to
How far can a fixed model go?
Answered today
Far, when the input and the loop give what the weights lack: examples, time to reason, tools, feedback and a check.
Next lectures
Our next lectures go beyond language and into more modalities. Lecture 3 · Self-supervised foundation models, Wed 28 Oct 2026, 18:00 SGT. Then Lecture 4 · Large multi-modal models, Wed 4 Nov 2026.
← → or PageUp/PageDown · F full screen · O overview · B black screen