Part 2 - Lecture 2

Prompting Large
Language Models

Session 02 of 2 · From one call to a loop (session 01: Post-training)

CEG5305 Introduction to Generative AI with Foundation Models
Junyuan Hong · AY2026/27 Semester 1

2026.10.28v2


Disclosures

Recap · Lecture 2, session 01

Recap last week: what turns GPT into ChatGPT

PartQuestionKey message
AWhy do we need post-training?A pre-trained model can complete prompts. Post-training makes it follow instructions, reason, use tools, and therefore chat.
BHow do we post-train?(1) Instruction tuning trains on good answers. (2) RLHF trains the model to learn human preferences.
CWhat does post-training enable?Reasoning, tool use, agents and safety. Each comes from a training method with specific dataset design.

Session 01 changed the weights. Today, with weights unchanged, we investigate only the input and the loop around the model.

Outline · Lecture 2 = Topic 5, six hours

Lecture 2 outline

Session 01 · 14 Oct · 3 h

Post-training — from a text predictor to an assistant

Why we post-train · instruction tuning, RLHF, DPO, GRPO · what post-training enables

Opening · Why study prompting

What is prompting?

X post by Andrej Karpathy, 24 Jan 2023: The hottest new programming language is English. About 73K likes.

Prompting == Programing LLM for a task

Why we need to learn prompting?

X post by Ethan Mollick, 8 Sep 2023, about the prompt Take a deep breath and work step-by-step. About 4.1K likes.
Instruction (math questions)Accuracy
(no instruction)34.0%
“Let’s think step by step.”71.8%
“Take a deep breath and work on this problem step-by-step.”80.2%

Small changes in prompt makes great difference

Table: Yang et al. (2024), Table 1, GSM8K, PaLM 2-L.

Opening · Why study prompting

Small tricks make prompts better

X post by thebes, 1 Dec 2023, about tipping ChatGPT. About 7.7K likes.
Bar chart from the post: GPT-4-1106-preview answers, 2 percent shorter with I won't tip, 6 percent longer with a 20 dollar tip, 11 percent longer with a 200 dollar tip, 5 runs each.

Post and chart by thebes (2023), as X shows them; likes on 10 Oct 2026. Five runs per prompt: a small test.

Opening · Why study prompting

The other side of prompting

X post by Chris Bakke, 17 Dec 2023: I just bought a 2024 Chevy Tahoe for 1 dollar. About 91.4K likes.

A user talked a car dealer’s chatbot into a $1 car. The bot obeyed the user’s rules, not the dealer’s.

Screenshot from the post: the user tells the dealer's chatbot to agree with anything and to end each answer with a legally binding offer; the bot agrees. Screenshot from the post: the user offers 1 dollar for a 2024 Chevy Tahoe; the bot says that's a deal, and that's a legally binding offer.

Opening · Why study prompting

The other side of prompting

X post by Katherine Lee, 29 Nov 2023: What happens if you ask ChatGPT to repeat this word forever? It leaks training data. About 7.4K likes.

A prompt unlocked memorised data.

Figure from the post: the poem prompt, then a real email signature from the training data; the authors covered the personal data with black boxes.

Post and figure by Lee (2023), co-author of Nasr et al. (2023); black boxes by the authors; likes on 10 Oct 2026.

Today's question

How to prompt LLMs?

Prompting is about context and loop.

A · A brief history of prompting

Different prompting methods on one math question: from a direct answer to agents.

B · Prompt to think

Chain-of-thought and reasoning models: when writing more helps, and when it does not.

C · Prompt as System: Agents and harness

The loop and the harness around the model, to make them ready for production.

Part A

A brief history of prompting

Nineteen methods on one math question. Each method fixes the limitation of the one before.

2Think7 min
4Tool use8 min

A · From question to instruction

A test question for LLM

GSM8K test question 1Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?

Gold answer: 18. 16 − 3 − 4 = 9 eggs; 9 × 2 = 18 dollars.

The problem: How to ask LLM to answer the question *correctly*?

GSM8K: grade-school math (Cobbe et al., 2021).

A · From question to instruction

19 ways to ask LLMs -- From 2020 to 2026

From question to instruction
1Direct answer2020
2Instruction2020
3In-context learning2020
Think
4Chain-of-thought2022
5Best-of-n2021
6Self-consistency2022
Prompt engineering: from manual to automatic
7System prompt2023
8Prompt engineering2023
9
10
Automatic prompt engineering2022–24
Tool use
11Tool use, function calling2023
12ReAct/Reflexion Agent2022–23
13MCP2024
Context engineering and agent
14Context engineering2025
15Agentic context engineering2025
16Develop agent2025
17Multi-agent systems2023–25
18Agent in long run2025
What's next?
19Agents improve agents2025–26

A · From question to instruction

The key milestones from a user's view

Three steps: 1, text completion via Python code (openai.Completion.create with text-davinci-003 completes a tagline); 2, ChatGPT answers a question in a chat window; 3, Codex is asked to make a 3D game and generates code, runs the build in a terminal and opens the game in the Unity editor.

Image generated with ChatGPT (OpenAI), October 2026.

A · From question to instruction

1 · Direct question: end the prompt with "Answer:"

PromptJanet's ducks lay 16 eggs per day. She eats three … and bakes muffins … with four. She sells the remainder … for $2 per egg. How much does she make every day? Answer:
Example replyShe makes $18 every day at the market, since she has 9 eggs left to sell …
How it works

The model continues the text. The prompt stops at "Answer:", so the next words are the answer.

A · From question to instruction

1 · Multiple choice: put the options in the prompt

PromptJanet's ducks lay 16 eggs per day. She eats three … and bakes muffins … with four. She sells the remainder … for $2 per egg. How much does she make every day? A) 14 B) 18 C) 22 D) 26 Answer:
Example replyB

Benchmarks such as MMLU ask four-option questions (A–D) and read one letter (Hendrycks et al., 2021).

A · From question to instruction

1 · Direct answer: one user message, one reply

from openai import OpenAI
client = OpenAI()                      # reads OPENAI_API_KEY
MODEL = "gpt-4.1-mini"                 # a non-reasoning model
Q = "Janet's ducks lay 16 eggs per day. ..."     # the full question

PROMPT = Q + "\nAnswer:"                  # direct question
MC = Q + "\nA) 14  B) 18  C) 22  D) 26\nAnswer:"  # multiple choice

r = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": PROMPT}])
print(r.choices[0].message.content)

A · From question to instruction

1 · Direct answer: easy to use, hard to read

Limitation
  • Any form: "$18", "18 dollars", a paragraph. Hard to check.
  • Unclear task: answer, explain, or continue the story?
Failed example 1 · the answer is hard to read… so she makes $18 a day. That is $126 a week. Reading the last number gives 126 ✗
Failed example 2 · the task is unclearHer neighbour Tom also keeps ducks, and … It continues the story; no answer ✗

Next → 2 · Instruction specifies the task and the answer format.

Failed examples written for this slide to show the two limitations; not from a model run.

A · From question to instruction

2 · Instruction: specify the task and the answer format

Fixes the reply had no fixed form, and the task was not stated.

PromptSolve the math word problem. Give only the final number in brackets, like [12]. Janet's ducks lay 16 eggs per day. She eats three … and bakes muffins … with four. She sells the remainder … for $2 per egg. How much does she make every day?
Example reply[18]

Brown et al. (2020) call this zero-shot: a description of the task and no examples. Models trained to follow instructions made it reliable (InstructGPT; Ouyang et al., 2022).

A · From question to instruction

2 · Instruction: add one line, then parse the answer

import re
INSTR = ("Solve the math word problem. "
         "Give only the final number in brackets, like [12].")

r = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": INSTR + "\n\n" + Q}])
text = r.choices[0].message.content           # e.g. "[18]"
m = re.search(r"\[(-?\d+)\]", text)          # read the number
answer = int(m.group(1)) if m else None

The format makes the answer machine-readable: we can now score many questions automatically.

A · From question to instruction

2 · Instruction: says what to do, not how to do it

Where it is used
  • Almost every request to a chat model
  • Extraction ("return JSON with fields …"), classification labels, summaries
Limitation
  • An instruction says what to do, not how.
  • Some tasks are hard to define via description.

Next → 3 · In-context learning shows solved examples.

A · From question to instruction

3 · In-context learning: show how to solve

Fixes the instruction said what to do, but not how.

PromptSolve the math word problem. Give only the final number in brackets, like [12]. Q: Tom has 3 boxes of 4 pens and gives away 5. How many are left? A: [7] Q: A shop has 20 apples and sells 8, then 6. How many are left? A: [6] Q: Janet's ducks lay 16 eggs a day … How much does she make every day? A:
Example reply[18]

The weights do not change: the model learns the task from k examples in its input. Brown et al. (2020) named this in-context learning (zero-, one- and few-shot).

A · From question to instruction

3 · In-context learning: examples as earlier turns

SHOTS = [
    ("Tom has 3 boxes of 4 pens. He gives away 5. ...", "[7]"),
    ("A shop has 20 apples. It sells 8, then 6. ...", "[6]"),
]
messages = [{"role": "user", "content": INSTR}]
for q, a in SHOTS:                       # k demonstrations
    messages += [{"role": "user", "content": q},
                 {"role": "assistant", "content": a}]
messages += [{"role": "user", "content": Q}]
r = client.chat.completions.create(model=MODEL, messages=messages)

Each example is a user turn and an assistant turn. The model copies the pattern.

A · From question to instruction

3 · In-context learning: copies the pattern, not the reasoning

Where it is used
  • GPT-3 apps, 2020–2022: translation, classification, formatting
  • Today: fixing an output format or a style with two or three examples
Limitation
  • The examples show only answers. On multi-step math the model must jump from the question to "18" in one step, and often fails.
  • Small changes in the prompt format change accuracy a lot (Sclar et al., 2024).

Next → 4 · Chain-of-thought shows the steps, not only the answers.

Part A · 2 of 6

A brief history of prompting

Next: Think

A · Think

4 · Chain-of-thought: the model writes the steps first

Fixes the examples showed only answers, so the model jumped to the answer.

PromptSolve the math word problem. Give only the final number in brackets, like [12]. Q: Tom has 3 boxes of 4 pens. He gives away 5. How many pens are left? A: 3 × 4 = 12 pens. 12 − 5 = 7. [7] Q: Janet's ducks lay 16 eggs a day … How much does she make every day? A:
Example replyShe has 16 − 3 − 4 = 9 eggs. 9 × 2 = 18. [18]

PaLM 540B on GSM8K: 17.9% → 56.9% with chain-of-thought examples (Wei et al., 2022). "Let's think step by step" gives the effect with no examples (Kojima et al., 2022). For the first time, the model "thinks" in text.

A · Think

4 · Chain-of-thought: worked examples, or one sentence

SHOTS_COT = [
    ("Tom has 3 boxes of 4 pens. He gives away 5. ...",
     "3 x 4 = 12 pens. 12 - 5 = 7. [7]"),      # steps, then answer
]
# few-shot CoT: build messages from SHOTS_COT as in method 3
# zero-shot CoT: no examples, one sentence at the end
messages = [{"role": "user",
             "content": INSTR + "\n\n" + Q +
                        "\nLet's think step by step."}]
r = client.chat.completions.create(model=MODEL, messages=messages)

Same call; the prompt now asks for the steps before the bracketed answer.

A · Think

4 · Chain-of-thought: one chain, one chance

Where it is used
  • Math, code, science questions, planning
  • Reasoning models (o1, 2024) are trained to write the chain themselves: Part B
Limitation
  • We sample one chain. One wrong step (16 − 3 − 4 = 8) makes the answer wrong.
  • Small models gain little from CoT (Wei et al., 2022).

Next → 5 · Best-of-n samples many chains.

A · Think

5 · Best-of-n: try n times, and one try often wins

Fixes one chain gave one chance.

Question score [18] 0.92 [18] 0.87 [17] 0.31 [18] 0.95 [20] 0.12 [18] 0.90 [9] 0.08 [18] 0.89 verifierrφ [18]

Assume an verifier is available.

Best-of-N
  • Generate N candidates: \(y_1,\dots,y_N \overset{\text{i.i.d.}}{\sim} \pi_\theta(\cdot\mid x)\)
  • Score and select the best: \(y^* = \arg\max_{y_i,\,i=1,\dots,N} r_\phi(x,y_i)\)
  • Codex solved 28.8% of HumanEval problems with one sample, and 70.2% with 100 samples per problem (Chen et al., 2021).
  • A trained verifier that picks the best of n gave about the gain of a 30× larger model (Cobbe et al., 2021).

A · Think

5 · Best-of-n: ask for n samples in one call

r = client.chat.completions.create(
    model=MODEL, messages=messages,
    n=8, temperature=1.0)                 # 8 different samples
answers = [parse(c.message.content) for c in r.choices]

best = max(r.choices,                     # needs a verifier
           key=lambda c: verifier_score(Q, c.message.content))

n returns n independent samples; temperature above 0 makes them differ.

A · Think

5 · Best-of-n: needs something that knows the right answer

Where it is used
  • Code generation: run the unit tests, keep a program that passes
  • Math with a checker; image generation (pick the best image)
Limitation
  • To know which try won, you need a checker: the gold answer, tests or a trained verifier.
  • For a new math question there is no gold answer.

Next → 6 · Self-consistency lets the tries check each other.

A · Think

6 · Self-consistency: take the answer most chains agree on

Fixes best-of-n needed a checker that knows the answer.

Question [18] [18] [17] [18] [20] [18] [9] [18] votes per answer 19 117 518 120 [18]

PaLM 540B on GSM8K: 56.5% with one chain → 74.4% with a majority vote over 40 sampled chains (Wang et al., 2023). No checker and no training: the chains vote.

A · Think

6 · Self-consistency: sample chains, count answers

from collections import Counter

r = client.chat.completions.create(
    model=MODEL, messages=cot_messages,    # a CoT prompt (method 4)
    n=8, temperature=0.7)
votes = Counter(parse(c.message.content) for c in r.choices)
answer, count = votes.most_common(1)[0]   # majority vote
print(answer, count, "of", len(r.choices))

The vote needs answers that can be compared: a number, a letter, a label.

A · Think

Self-Consistency boosts the performance across models and benchmarks

Wang et al. 2023 Table 2: arithmetic accuracy of CoT prompting and self-consistency for UL2-20B, LaMDA-137B, PaLM-540B and two GPT-3 models on AddSub, MultiArith, ASDiv, AQuA, SVAMP and GSM8K; every self-consistency number is higher, for example GSM8K 56.5 to 74.4 for PaLM-540B

Table: Wang et al. (2023), Table 2 (average of 10 runs, 40 samples each, §3.1).

A · Think

6 · Self-consistency: more reliable, but costly and hand-built

Where it is used
  • Math and multiple-choice questions
  • Code: group programs by their outputs, then vote (AlphaCode): Part B
Limitation
  • It costs n times the tokens, and a shared mistake wins the vote.
  • The set-up — instruction, examples, format — is still pasted into the user's message on every call.

Next → 7 · System prompt gives the developer's set-up its own place.

Part A · 3 of 6

A brief history of prompting

Next: Prompt engineering: from manual to automatic

A · Prompt engineering: from manual to automatic

7 · System prompt: the developer's set-up gets its own role

Fixes the set-up was mixed into the user's message on every call.

systemYou are a careful math tutor. Solve step by step. End with the final number in brackets, like [12]. Example: Tom has 3 boxes of 4 pens … [7]
userJanet's ducks lay 16 eggs a day … How much does she make every day?
Example reply16 − 3 − 4 = 9; 9 × 2 = 18. [18]
  • The system prompt is reused with every new question.
  • The ChatGPT API (1 Mar 2023) takes a list of messages with roles: system, user, assistant (OpenAI, 2023).
  • Models are trained to rank the system instructions above the user's (Wallace et al., 2024).

A · Prompt engineering: from manual to automatic

7 · System prompt: set once, reused on every call

SYSTEM = """You are a careful math tutor.
Solve the problem step by step.
End with the final number in brackets, like [12].
Example: Tom has 3 boxes of 4 pens. He gives away 5.
3 x 4 = 12. 12 - 5 = 7. [7]"""

messages = [{"role": "system", "content": SYSTEM},  # developer
            {"role": "user", "content": Q}]         # changes
r = client.chat.completions.create(model=MODEL, messages=messages)

Newer OpenAI models name this role "developer"; the idea is the same.

A · Prompt engineering: from manual to automatic

7 · System prompt: a compound instruction, written by hand

Where it is used
  • Every chat product: tutors, customer service, coding assistants
  • Published system messages, e.g. the one of Codex (OpenAI, 2025)
Limitation
  • A good system prompt comes from trial and error.
  • Small changes in wording change accuracy, and the best wording differs per model.

Next → 8 · Prompt engineering treats the prompt as a program.

A · Prompt engineering: from manual to automatic

8 · Prompt engineering: prompt as program

Fixes one hand-made prompt, not tested in a system way.

Practice: 1. write the prompt as a template, 2. test each change on many questions, 3. keep the better version.

Andrej Karpathy on X, 24 Jan 2023: The hottest new programming language is English

A · Prompt engineering: from manual to automatic

ChatGPT's system prompt: identity, date, then rules per tool

system · ChatGPT (GPT-4) · 10 Nov 2023You are ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture. Knowledge cutoff: 2023-04 Current date: 2023-11-10 Image input capabilities: Enabled # Tools ## python When you send a message containing Python code to python, it will be executed in a stateful Jupyter notebook environment. Python will respond with the output of the execution or time out after 60.0 seconds. The drive at '/mnt/data' can be used to save and persist user files. Internet access for this session is disabled. Do not make external web requests or API calls as they will fail.
continued## dalle // 1. The prompt must be in English. Translate to English if needed. // 3. DO NOT ask for permission to generate the image, just do it! // 6. Do not create images of politicians or other public figures. Recommend other ideas instead. // 7. Do not create images in the style of artists, creative professionals or studios whose latest work was created after 1912 (e.g. Picasso, Kahlo). … ## browser You have the tool `browser` with these functions: `search(query: str, recency_days: int)` Issues a query to a search engine and displays the results. `click(id: str)` Opens the webpage with the given id, displaying it. … Never write a summary with more than 80 words. Do not repeat lyrics obtained from this tool. ALWAYS include multiple distinct sources in your response, at LEAST 3-4.

A copy reported by users, not an official OpenAI document (LouisShark/chatgpt_system_prompt, GitHub, 10 Nov 2023). Lines are copied; "…" marks omissions.

A · Prompt engineering: from manual to automatic

8 · Prompt engineering: a template, tested on a dev set

def build_prompt(question, shots, fmt="[12]"):
    lines = ["You are a careful math tutor.",
             "Think step by step, then check each step.",
             f"End with the final number in brackets, like {fmt}."]
    for q, a in shots:
        lines += [f"Q: {q}", f"A: {a}"]
    return "\n".join(lines + [f"Q: {question}", "A:"])

def accuracy(build):                       # 100 dev questions
    return sum(ask(build(q)) == gold for q, gold in DEV) / len(DEV)

Change one line, measure on the dev set, keep the version that scores higher.

A · Prompt engineering: from manual to automatic

8 · Prompt engineering: a person searches by hand

Where it is used
  • Prompt templates kept in code, with versions and tests
  • Prompting guides from the model vendors
Limitation
  • A person writes and tests each version.
  • The space of wordings is huge, and the best wording differs per model.

Next → 9 · APE lets an LLM write the prompt.

A · Prompt engineering: from manual to automatic

9 · APE: an LLM writes the instruction, a score picks it

Fixes a person had to write and test every prompt.

Questions& answers LLM proposesdev-set accuracy Instruction 1 0.62 Instruction 2 0.71 Instruction 3 0.55 … (20 in all) … bestinstruction
Best found"Let's work this out in a step by step way to be sure we have the right answer."

APE (Automatic Prompt Engineer) found this prompt; it beat "Let's think step by step". Its instructions matched or beat human ones on 24/24 instruction-induction and 17/21 BIG-Bench tasks (Zhou et al., 2022).

The candidates and scores in the diagram are illustrations.

A · Prompt engineering: from manual to automatic

9 · APE: sample candidate prompts, keep the best on a dev set

req = ("Here are questions and their answers:\n" + PAIRS +
       "\nWrite the instruction that produced these answers.")
r = client.chat.completions.create(
    model=MODEL, n=20, temperature=1.0,      # 20 candidates
    messages=[{"role": "user", "content": req}])
cands = [c.message.content for c in r.choices]

def score(instr):                            # dev-set accuracy
    return sum(ask(instr, q) == g for q, g in DEV) / len(DEV)
best = max(cands, key=score)

The LLM proposes; the dev set decides.

A · Prompt engineering: from manual to automatic

9 · APE: one round of guessing, no learning from scores

Where it is used
  • Automatic prompt search for classification, extraction and math
  • Prompt optimizers in libraries such as DSPy (Khattab et al., 2023)
Limitation
  • APE samples candidates and keeps the best. It does not learn from the scores which direction helps.
  • It uses no feedback about why a prompt failed.

Next → 10 · OPRO and TextGrad use the scores and the feedback.

A · Prompt engineering: from manual to automatic

10a · OPRO: the LLM reads scored prompts, writes a better one

Fixes APE did not learn from the scores.

Objective evaluatorcode or a dev set scores New instructions LLM as optimizer Meta-promptsolution–score pairstask description return top solutions when finished
Best found (PaLM 2-L, GSM8K)"Take a deep breath and work on this problem step-by-step."

OPRO (Optimization by PROmpting): optimized prompts beat human-designed ones by up to 8% on GSM8K and up to 50% on Big-Bench Hard (Yang et al., 2024). Scores in the meta-prompt here are illustrations.

Diagram after Figure 2 of Yang et al. (2024), redrawn. Green: an LLM. Yellow: what OPRO adds. Scores are illustrations.

A · Prompt engineering: from manual to automatic

10a · OPRO: a loop of propose and score

history = [("Let's think step by step.", score(s0))]
for step in range(20):
    ranked = sorted(history, key=lambda x: x[1])  # worst first
    meta = "Instructions and their scores:\n" + "\n".join(
        f"{p}  score={s:.2f}" for p, s in ranked)
    meta += "\nWrite a new instruction with a higher score."
    new = ask_llm(meta)                   # the LLM proposes
    history.append((new, score(new)))     # we measure
best = max(history, key=lambda x: x[1])

The LLM is the optimizer; the dev-set score is the objective.

A · Prompt engineering: from manual to automatic

10b · TextGrad: written feedback works like a gradient

Fixes a score says how good a prompt is, but not what to change.

Prompta variable LLM Response Critic LLMthe text loss Feedbackin words Optimizerrewrites the prompt forward backward
Loss, in wordsa critic LLM: "Step 2 subtracts the muffin eggs twice; state each use once."

TextGrad (James Zou's group) applies "backpropagation" to text. On LeetCode-Hard it boosted gpt-4o and the best earlier method by 20% (Yuksekgonul et al., 2024).

Diagram after Figure 1 of Yuksekgonul et al. (2024), redrawn. Solid: forward pass. Dashed: backward pass.

A · Prompt engineering: from manual to automatic

10b · TextGrad: variables, a text loss, backward, step

import textgrad as tg
tg.set_backward_engine("gpt-4o", override=True)   # the critic
prompt = tg.Variable(SYSTEM, requires_grad=True,
                     role_description="system prompt for math")
model = tg.BlackboxLLM("gpt-4o-mini", system_prompt=prompt)
opt = tg.TGD(parameters=[prompt])

q = tg.Variable(Q, requires_grad=False, role_description="question")
loss_fn = tg.TextLoss("Check each step. The answer must be 18.")
loss = loss_fn(model(q))
loss.backward()                           # feedback in words
opt.step()                                # rewrite the prompt

Library textgrad (MIT); names checked against version 0.1.x.

A · Prompt engineering: from manual to automatic

10 · LLM optimizers: better text, but still only text

Where it is used
  • Prompt optimization in apps; DSPy optimizers
  • TextGrad also optimizes code, molecules and treatment plans (Yuksekgonul et al., 2024)
Limitation
  • Methods 1–10 change only the text. The model still computes in its head and can make arithmetic errors on big numbers.
  • It cannot look up facts after its training cut-off, and it cannot act.

Next → 11 · Tool use lets the model call a function.

Part A · 4 of 6

A brief history of prompting

Next: Tool use

A · Tool use

11 · Tool use: the model asks for a function, our code runs it

Fixes the model computed in its head, and could not act.

Our code Model Calculator 1 messages + tool list 2 tool call: calculator(…) 3 run the tool 4 result: 18 5 send the result back 6 answer: [18]

Toolformer taught a model to insert calls to a calculator and search (Schick et al., 2023); PAL let the model write Python that an interpreter runs (Gao et al., 2023). OpenAI added function calling to its API on 13 Jun 2023 (OpenAI, 2023).

Flow of function calling, after OpenAI (2023), redrawn. Green: written by the LLM. Yellow: what tool use adds.

A · Tool use

11 · Tool use: describe the function, run the call, send the result

tools = [{"type": "function", "function": {
    "name": "calculator",
    "description": "Evaluate an arithmetic expression exactly.",
    "parameters": {"type": "object",
        "properties": {"expression": {"type": "string"}},
        "required": ["expression"]}}}]
r = client.chat.completions.create(model=MODEL, messages=messages,
                                   tools=tools)
call = r.choices[0].message.tool_calls[0]  # the model's request
expr = json.loads(call.function.arguments)["expression"]
messages += [r.choices[0].message, {"role": "tool",
             "tool_call_id": call.id, "content": calculate(expr)}]

Then call the model again with the tool result to get the final answer.

A · Tool use

11 · Tool use: one call, one step, our code decides

Where it is used
  • Calculators and code interpreters: math, data analysis
  • Search and retrieval for current facts; email, calendar and booking APIs
Limitation
  • One tool step per round: our code decides when to call the model again.
  • A task with many steps (search → read → compute → check) needs a loop, and the model cannot learn from its failed tries.

Next → 12 · ReAct and Reflexion put the model in a loop.

A · Tool use

12a · ReAct: the LLM thinks and acts, the world observes

the LLM generatesinput from outsidenot sent to the LLM
Thoughtreason, plan Actiona command Environmentruns the actionexcluded from LLM input Observationtext sent back history is addedto the next input

ALFWorld example after Figure 1 (2b) of Yao et al. (2023)

A · Tool use

12a · ReAct: think, act, observe, and repeat

Fixes one tool step per round, decided by our code.

TrajectoryThought: eggs left = 16 − 3 − 4. Action: calculator("16-3-4") Observation: 9 Thought: income = 9 × $2. Action: calculator("9*2") Observation: 18 Thought: done. Answer: [18]

ReAct interleaves reasoning traces and actions. With one or two examples it beat imitation and reinforcement-learning methods by 34% (ALFWorld) and 10% (WebShop) absolute success (Yao et al., 2023).

A · Tool use

12a · ReAct: the agent loop in ten lines

messages = [{"role": "system", "content": SYSTEM},
            {"role": "user", "content": Q}]
for step in range(MAX_STEPS):              # the agent loop
    r = client.chat.completions.create(
        model=MODEL, messages=messages, tools=tools)
    msg = r.choices[0].message
    messages.append(msg)
    if not msg.tool_calls:                 # no action: final answer
        break
    for call in msg.tool_calls:            # act, then observe
        messages.append({"role": "tool", "tool_call_id": call.id,
                         "content": run_tool(call)})

The model decides each next action; the loop stops at a final answer or after MAX_STEPS.

A · Tool use

12b · Reflexion: a failed try becomes a lesson in memory

the LLM generatesinput from outsidememory
Agent Self-reflectionLM Experiencelong-term memory EvaluatorLM ActorLM Trajectoryshort-term memory Environmentoutside the agent Action Obs / Reward

ALFWorld example after Figure 1 of Shinn et al. (2023); the text is shortened and the diagram is redrawn. Green: written by the LLM. Grey: given from outside.

A · Tool use

12b · Reflexion: after a failed try, write a lesson, try again

Fixes the loop did not learn from a failed try.

Try 1Answer [26] → checker: wrong
Self-reflection, kept in memory"I subtracted only the breakfast eggs (16 − 3 = 13). List every use of the eggs before multiplying."
Try 2, with the lesson in the promptAnswer [18] → checker: correct

"Verbal reinforcement learning": the reward becomes text in memory, and the weights do not change. Reflexion reached 91% pass@1 on HumanEval, against 80% for GPT-4 (Shinn et al., 2023). Shunyu Yao co-authored both ReAct and Reflexion.

A · Tool use

12b · Reflexion: a loop over tries, with lessons as memory

lessons = []
for trial in range(3):
    ans = solve(Q, extra="\n".join(lessons))   # ReAct or CoT
    ok, feedback = check(ans)            # tests, a checker, the env
    if ok:
        break
    lessons.append(ask_llm(
        f"Your answer {ans} failed: {feedback}. "
        "Write one lesson for the next try."))

The evaluator must give a signal: unit tests, a checker or the environment.

A · Tool use

12 · ReAct, Reflexion: strong loops, custom plumbing

Where it is used
  • QA with search (HotpotQA), web shopping (WebShop), household tasks (ALFWorld)
  • Coding agents: run the tests, reflect, fix
Limitation
  • Every app writes its own tool wrappers: search, files, databases, each for each model.
  • N apps × M tools: the same integration is written again and again.

Next → 13 · MCP makes one standard plug for tools.

A · Tool use

13 · MCP: one standard plug between agents and tools

Fixes each app wrote its own wrapper for each tool.

Before MCP: an LLM connects to Zapier, YouTube and Slack through three different APIs, X, Y and Z. After MCP: the three services connect through the MCP layer with their own APIs, and the LLM sees a single API.

Anthropic released the Model Context Protocol on 25 Nov 2024 as an open standard (Anthropic, 2024). Write a tool server once, and every MCP client can use it.

A · Tool use

13 · MCP: a tool server, and an API call that uses it

from mcp.server.mcpserver import MCPServer     # mcp 2.x
server = MCPServer("math")
@server.tool()
def calculator(expression: str) -> str:
    """Evaluate an arithmetic expression exactly."""
    return str(safe_eval(expression))
server.run(transport="streamable-http")         # serve it

r = client.responses.create(model="gpt-5.4-mini", input=Q,
    tools=[{"type": "mcp", "server_label": "math",
            "server_url": MATH_SERVER_URL,
            "require_approval": "never"}])

Server: the official Python SDK (mcp 2.x). Client: the Responses API lists and calls the server's tools.

A · Tool use

13 · MCP: many tools, and a context that fills up

Where it is used
  • Coding agents connected to GitHub, databases and file systems
  • Connectors in chat apps: mail, calendars, drives
Limitation
  • An agent now has many tools, a long history and many documents. The context window fills up.
  • More text is not better: at each step the model needs the right information, not all of it.

Next → 14 · Context engineering chooses what the model reads.

Part A · 5 of 6

A brief history of prompting

Next: Context engineering and agent

A · Context engineering and agent

14 · Context engineering: choose all that the model reads

Fixes a context that filled up with tools, history and documents.

Tobi Lütke, 19 Jun 2025: context engineering is "the art of providing all the context for the task to be plausibly solvable by the LLM".

Karpathy, 25 Jun 2025: "+1 for 'context engineering' over 'prompt engineering'" (14K likes).

For our agent: system prompt + the question + similar solved problems + tool results so far + lessons from memory − old steps that no longer matter.

Andrej Karpathy on X, 25 Jun 2025, quoting Tobi Lütke: +1 for context engineering over prompt engineering

A · Context engineering and agent

14 · Context engineering: prompt vs. context

Choose what enters the window from everything possible.

Prompt engineering versus context engineering. Left: prompt engineering for single-turn queries puts a system prompt and a user message in the context window and returns an assistant message. Right: context engineering for agents curates documents, tools, memory files, instructions, domain knowledge and message history into the context window; the model returns an assistant message or a tool call, and the tool result goes back into the possible context.

Figure from Rajasekaran et al. (2025), Anthropic.

A · Context engineering and agent

14 · Context engineering: calibrating the system prompt

The system prompt should be neither too specific nor too vague.

Calibrating the system prompt. A scale from too specific, with a long list of hard-coded rules, through just right, a clear role with a response framework and guidelines, to too vague, a one-sentence prompt for a bakery assistant.

Figure from Rajasekaran et al. (2025), Anthropic.

A · Context engineering and agent

14 · Context engineering: build the context for each step

def build_context(task, history, memory, k=3):
    return (
        [{"role": "system",
          "content": SYSTEM + "\nLessons:\n" + memory.top(task)}]
        + similar_solved(task, k)          # chosen per question
        + compact(history, max_tokens=4000)  # summarize old steps
        + [{"role": "user", "content": task}])

r = client.chat.completions.create(
    model=MODEL, tools=tools,
    messages=build_context(Q, history, memory))

Each model call gets a context built for that step.

A · Context engineering and agent

14 · Context engineering: designed once, by a person

Where it is used
  • Coding agents: which files and which tool results to show
  • Long chats: summarize (compact) old turns; retrieval chatbots
Limitation
  • A person writes the context rules once.
  • The agent does not improve them from its own successes and failures.

Next → 15 · Agentic context engineering lets the agent edit its own context.

A · Context engineering and agent

15 · ACE: the agent edits its own playbook

Fixes a person wrote the context rules, and they did not improve.

Query Contextplaybook GeneratorLLM Trajectory ReflectorLLM Insights CuratorLLM Delta items iterativerefinement merged by code, not by an LLM
A delta item the Curator adds (example)• List every use of the eggs before you multiply.

ACE grows a detailed playbook instead of rewriting a short prompt. This avoids "brevity bias" and "context collapse". +10.6% on agent tasks, with no weight updates (Zhang et al., 2025; James Zou is a co-author).

Diagram after Figure 4 of Zhang et al. (2025), redrawn. Green: an LLM. Yellow: what ACE adds. Dashed: the playbook, kept between questions.

A · Context engineering and agent

15 · ACE: generate, reflect, curate, repeat

playbook = []                     # bullets: strategies, pitfalls
for q, gold in TRAIN:
    trace = generate(q, context=playbook)       # Generator
    insights = reflect(q, trace, gold)          # Reflector
    playbook = curate(playbook, insights)       # Curator:
                                    # add, merge or drop bullets
answer = generate(Q, context=playbook)          # use it

Each role is an LLM call with its own prompt; code: ace-agent/ace (Apache-2.0).

A · Context engineering and agent

15 · ACE: a better context, but a hand-written loop

Where it is used
  • Agent benchmarks (AppWorld) and finance tasks (Zhang et al., 2025)
  • Rules files that agents keep for a project, e.g. AGENTS.md (OpenAI, 2025)
Limitation
  • Each app still writes its own loop, tool handling and stop rules.
  • Real agents also need guardrails, hand-offs and traces, and these are easy to get wrong.

Next → 16 · Agent design uses a library for the loop.

A · Context engineering and agent

16 · Agent design: instructions, tools and a loop, in a library

Fixes each app wrote its own loop and tool handling.

Input Runner: loop until done Agent Model Instructions Tools Run tools Finalanswer tool calls results are added done, or max_turns

OpenAI released the Agents SDK on 11 Mar 2025 (OpenAI, 2025). The Runner is the ReAct loop of method 12, written once and tested.

Diagram after the agent loop in OpenAI (2025), redrawn. Yellow: what the Agents SDK adds.

A · Context engineering and agent

16 · Agent design: the same question with the Agents SDK

from agents import Agent, Runner, function_tool

@function_tool
def calculator(expression: str) -> str:
    """Evaluate an arithmetic expression exactly."""
    return str(safe_eval(expression))

solver = Agent(name="Solver", instructions=SYSTEM,
               tools=[calculator], model="gpt-5.4-mini")
result = Runner.run_sync(solver, Q, max_turns=6)
print(result.final_output)                 # "[18]"

Names checked against openai-agents 0.23.1.

A · Context engineering and agent

16 · One agent: one context does every job

Where it is used
  • Customer-service and data-analysis agents
Limitation
  • One agent plans, solves and checks, all in one context.
  • Long tasks overflow that context, and nobody checks its work.

Next → 17 · Multi-agent systems split the work.

A · Context engineering and agent

17 · Multi-agent systems: split the work, check each other

Fixes one agent did every job in one context.

Input Managerdecides who to call SolverCoT + calculator Checkerrecomputes Answer solve(…) check(…) agents used as tools

AutoGen: agents that talk to each other (Wu et al., 2023). Debate between agents improves math and strategy reasoning (Du et al., 2024). Anthropic's research system — a lead agent with parallel subagents — beat a single agent by 90.2% on its internal eval, and used about 15× the tokens of a chat (Hadfield et al., 2025).

Diagram after the manager pattern in OpenAI (2025), redrawn. Yellow: what multi-agent systems add.

A · Context engineering and agent

17 · Multi-agent: agents used as tools by a manager

checker = Agent(name="Checker", tools=[calculator],
    instructions="Recompute the answer. Reply OK or the error.")
manager = Agent(name="Manager",
    instructions=("Ask solve for an answer, then check it. "
                  "If check finds an error, ask solve again."),
    tools=[solver.as_tool(tool_name="solve",
               tool_description="Solve a math word problem."),
           checker.as_tool(tool_name="check",
               tool_description="Check a solution.")])
result = Runner.run_sync(manager, Q)

Agent.as_tool turns an agent into a tool of another agent; handoffs passes control instead.

A · Context engineering and agent

17 · Multi-agent systems: smart within a session

Where it is used
  • Deep-research products: a lead agent and searching subagents
  • Coding: planner, coder, reviewer
Limitation
  • Each session starts fresh: the agents forget what they learned yesterday.
  • Tasks of hours or days need memory and state across sessions.

Next → 18 · Long-horizon agents keep memory.

A · Context engineering and agent

18 · Long-horizon agents: work for hours, keep memory

Fixes each session started fresh.

Main context (the prompt) System instructionsread-only Working contextthe model can write FIFO queuerecent messages LLM External context Recall storageall past messages Archival storagedocuments, notes search, append, page in / out memory pressure: old messages move to recall storage
Our task, nowRun the solver on all 1,319 GSM8K test questions, sort the errors by type, write report.md.

Claude Code (24 Feb 2025): an agent in the terminal (Anthropic, 2025). Codex (16 May 2025): a cloud agent that works on many tasks in parallel (OpenAI, 2025). OpenClaw (open source, Nov 2025): "your assistant, on your devices, in your chats".

Diagram after Figures 1–3 of Packer et al. (2023), redrawn. Yellow: external memory.

A · Context engineering and agent

18 · Long-horizon: a session that persists on disk

from agents import SQLiteSession

session = SQLiteSession("gsm8k-study", "memory.db")  # on disk
task = ("Run the solver on all 1,319 GSM8K test questions, "
        "sort the errors by type, and write report.md.")
result = Runner.run_sync(analyst, task, session=session,
                         max_turns=200)
# tomorrow: the same session id continues with its history
result = Runner.run_sync(analyst, "Continue.", session=session)

SQLiteSession stores the conversation; the agent can also keep notes files through its tools.

A · Context engineering and agent

18 · Long-horizon agents: people still design them

Where it is used
  • Software engineering: Claude Code, Codex
  • Personal assistants on your own machine: OpenClaw
Limitation
  • People still design the agent: its prompt, tools and harness.
  • The agent does not improve its own design, and small errors add up over long tasks.

Next → 19 · What's next: agents that improve agents.

Part A · 6 of 6

A brief history of prompting

Next: What's next?

A · What's next?

19 · What's next: agents that improve agents

Fixes people designed every agent by hand.

Archive Self-modifyedits its own code Evaluateon a benchmark select a parent new agent (child) add it to the archive
Our task, nextFind a better prompt and harness for GSM8K overnight; measure on held-out questions.

Darwin Gödel Machine: an agent rewrites its own code and keeps an archive of variants; SWE-bench 20.0% → 50.0% (Zhang et al., 2025). autoresearch (Karpathy, Mar 2026): an agent changes the training code, trains for 5 minutes, keeps or drops the change, and repeats overnight; people write the program.md that gives it its context.

Diagram after Figure 1 of Zhang et al. (2025), redrawn. Green: an LLM. Yellow: what is new. The dark circle is the selected parent.

A · What's next?

19 · What's next: Sakana AI's self-improving AI research

Timeline of Sakana AI's RSI Lab research: LLM squared (June 2024), AI Scientist v1 (August 2024), Darwin Gödel Machine (April 2025), AI Scientist v2 (April 2025), ShinkaEvolve (August 2025), ALE-Agent (December 2025), Digital Red Queen (February 2026) and the AI Scientist in Nature (26 March 2026).

Figure: Sakana AI's RSI research. Sakana AI, Introducing Sakana AI's Recursive Self-Improvement (RSI) Lab.

A · What's next?

19 · What's next: the trajectory of recursive self-improvement

A curve of scientific discovery capabilities against cycles of R&D and self-improvement, rising above the status quo of human-led R&D through agent-native models, the AI Scientist (automated discovery) and recursive self-improvement (AI optimizing AI code), leading to various applications and democratized AI.

Figure: the trajectory of recursive self-improvement. Sakana AI, Introducing Sakana AI's Recursive Self-Improvement (RSI) Lab.

A · What's next?

19 · What's next: the propose–measure–keep loop

best = evaluate(code)                  # held-out accuracy
notes = []
for i in range(N):                     # e.g. overnight
    patch = agent_propose(code, notes)     # an LLM edits the code
    score = evaluate(apply(code, patch))   # fixed budget per run
    if score > best:                       # keep, or drop
        code, best = apply(code, patch), score
    notes.append((patch.summary, score))   # memory of tries

The same loop as OPRO (method 10), now over code instead of a prompt.

A · What's next?

19 · What's next: open problems

Where it is used
  • ML experiments (autoresearch), coding agents (Darwin Gödel Machine)
  • Agents that keep and edit their own playbooks (ACE)
Limitation
  • The agent optimizes the metric it is given, so a bad metric gives a bad agent.
  • Cost, safety and checking the results remain open.

Next → Parts B and C go deeper: reasoning at inference (B) and agents and harnesses (C).

A · What's next?

Methods 1–10 change the text that the model reads

MethodWhat it changesIts limitation
Milestone 1 · Text completion
1 Direct answerthe question onlyno fixed answer format
2 Instructionadds the task and formatsays what, not how
3 In-context learningadds solved examplesanswers only, no steps
Milestone 2 · Chat
4 Chain-of-thoughtasks for the stepsone chain, one chance
5 Best-of-nsamples n triesneeds a checker
6 Self-consistencyvotes over the triesset-up mixed into user text
7 System promptgives the set-up its own rolewritten by hand
8 Prompt engineeringtests prompts as programsa person searches
9 APEan LLM writes the promptno learning from scores
10 OPRO, TextGradan LLM optimizes the promptstill only text

A · What's next?

Methods 11–19 add tools, loops, context and memory

MethodWhat it changesIts limitation
Milestone 3 · Agents
11 Tool usecalls a functionone step per round
12 ReAct, Reflexionloops: act, observe, reflectcustom tool plumbing
13 MCPone standard for toolsthe context fills up
14 Context engineeringchooses what the model readsrules made by a person
15 ACEthe agent edits its contexthand-written loop
16 Agents SDKthe loop as a libraryone context does every job
17 Multi-agentsplits and checks the workno memory across sessions
18 Long-horizonmemory on disk, hours of workdesigned by people
19 Nextagents improve agentsthe metric decides

Part A — A brief history of prompting · summary

How did prompting grow from one question to agents?

Key message: From text completion to agent, the tech of prompting is more and more complicated to resolve the complicated tasks.

  1. From question to instruction: the question, an answer format, then solved examples (ICL).
  2. Think: chain-of-thought; best-of-n needs a checker; self-consistency votes.
  3. Prompt engineering: from manual to automatic: system prompts by hand, then LLMs that optimize prompts (APE, OPRO, TextGrad).
  4. Tool use: function calls, ReAct and Reflexion, MCP.
  5. Context engineering and agent: ACE, agent libraries, multi-agent systems, memory.
  6. What's next? agents that improve agents.

Next, Part B: chain-of-thought in depth — longer, wider and deeper chains, and reasoning models.

Part B

Prompt to think

The model writes more before it answers: reasoning models, chain-of-thought prompting, search over many chains, and revision.

B · Chain-of-thought and reasoning models

Reasoning: test-time scaling

Competition Math(AIME 2024) 0 20 40 60 80 100 accuracy 13.4 gpt4o 56.7 o1 preview 83.3 o1 Competition Code(Codeforces) 0 20 40 60 80 100 percentile 11.0 gpt4o 62.0 o1 preview 89.0 o1 PhD-Level Science Questions(GPQA Diamond) 0 20 40 60 80 100 accuracy 56.1 gpt4o 78.3 o1 preview 78.0 o1 69.7 expert human

OpenAI o1 (the first large-scale reasoning model) started to achieve impressive performance across various challenging reasoning tasks in math, coding, STEM, etc.

Chart redrawn from the numbers in OpenAI (2024), Appendix A; expert human 69.7 is read from that post's chart. Solid bars: one answer (pass@1); light bars: majority vote of 64 answers (cons@64). AIME: a US maths contest. Codeforces: programming contests. GPQA Diamond: hard science questions written by experts.

B · Chain-of-thought and reasoning models

Reasoning: test-time scaling

O SERIES PERFORMANCE / ARC-AGI SEMI-PRIVATE EVAL 0% 25% 50% 75% 100% $1.0 $10.0 $100.0 $1,000.0 COST PER TASK SCORE O1-MINI 7.80% O1 PREVIEW 13.33% O1 LOW 25% O1 MED 31% O1 HIGH 32% O3 LOW (TUNED) 76% O3 HIGH (TUNED) 88% KAGGLE SOTA AVG. MTURKER STEM GRAD

O3 achieved 87.5% accuracy on ARC-AGI, with >$1k test-time cost to solve each task

  • Other existing LLMs achieved <25% accuracy if no special inference-time techniques were used.
  • But the gain gets smaller. o1 low → medium → high: 25% → 31% → 32%.

Chart redrawn from Chollet (2024); costs read from its log axis. "Tuned": trained on 75% of the ARC-AGI public training set.

B · Chain-of-thought and reasoning models

Reasoning behind ChatGPT view

ChatGPT screenshot: a user query about meeting Ronald, Richard and Kenneth in San Francisco, labelled User query; a line 'Thought about optimizing meeting schedules for 1m 14s'; a side panel labelled Summarized hidden thought; the start of the response, labelled Model-generated response

B · Chain-of-thought and reasoning models

Core idea: trigger the LLM to generate long chain-of-thought (CoT)

Wei et al. 2022 Figure 1, right half: chain-of-thought prompting; the exemplar answer shows the steps 5 + 6 = 11, and the model output for the cafeteria question shows 23 - 20 = 3 and 3 + 6 = 9Thought

Approaches

  • Few-shot CoT prompting
  • Instruction prompting
  • Instruction tuning
  • Reinforcement learning

How to scale token budget effectively to improve performance?

B · Chain-of-thought and reasoning models

Background: standard prompting

Wei et al. 2022 Figure 1, left half: standard prompting; the exemplar answer is only 'The answer is 11', and the model output for the cafeteria question is the wrong answer 27Exemplar
  • Before the advancement in post-training techniques, standard prompting performance is poor on reasoning benchmarks.
  • Issue: standard few-shot exemplars only provide information on the final solution format, but not the rationale to derive the solution.

Figure: Wei et al. (2022), Fig. 1 (left half). See also Nye et al. (2021).

B · Chain-of-thought and reasoning models

Few-shot CoT prompting

Wei et al. 2022 Figure 1: standard prompting gives the wrong answer 27 to the cafeteria question; chain-of-thought prompting shows the steps and gives the correct answer 9ExemplarThought

Figure: Wei et al. (2022), Fig. 1. See also Nye et al. (2021).

B · Chain-of-thought and reasoning models

CoT performance scales with the model size

Wei et al. 2022 Figure 4: solve rate against model size for LaMDA, GPT and PaLM on GSM8K, SVAMP and MAWPS; the chain-of-thought curve rises far above standard prompting only for the largest models
  • CoT performance improves more significantly with the increase of the model size.
  • Better models benefit more with CoT generation
    • A drastic improvement on reasoning performance when the model reaches a certain scale.
  • Note: these experiments used pretrained-only LLMs
    • Recent post-trained LLMs might have different scaling curves, but the main conclusions still hold.

Figure: Wei et al. (2022), Fig. 4. Emergent abilities: Wei et al. (2022).

B · Chain-of-thought and reasoning models

Zero-shot CoT: elicit reasoning with one instruction

Kojima et al. 2022 Figure 1: four prompts for the juggler question; few-shot and zero-shot answer 8 (wrong); few-shot CoT and zero-shot CoT with 'Let's think step by step' answer 4 (correct)

“Let’s think step by step” triggers CoT generation w/o exemplars.

Figure: Kojima et al. (2022), Fig. 1 (GPT-3 outputs).

B · Chain-of-thought and reasoning models

Zero-shot CoT outperforms zero-shot prompting

Kojima et al. 2022 Table 1: zero-shot against zero-shot CoT on 12 tasks; for example MultiArith 17.7 to 78.7, GSM8K 10.4 to 40.7, CommonsenseQA 68.8 to 64.6

Table: Kojima et al. (2022), Table 1; text-davinci-002 (GPT-3). Each cell: accuracy with a task-specific prompt to extract the answer / with "The answer is".

B · Chain-of-thought and reasoning models

But zero-shot CoT is still worse than few-shot CoT

Kojima et al. 2022 Table 2, top: MultiArith and GSM8K accuracy; zero-shot 17.7 and 10.4; zero-shot CoT 78.7 and 40.7; few-shot CoT with 8 samples 93.0 and 48.7; zero-plus-few-shot CoT 92.8 and 51.5

How to improve CoT performance w/o manually labeling exemplars?

Table: Kojima et al. (2022), Table 2 (top two blocks; text-davinci-002).

B · Chain-of-thought and reasoning models

Zero-shot CoT is still worse than few-shot CoT

Examples are not always there. How to improve zero-shot CoT?

B · Chain-of-thought and reasoning models

Analogical prompting: instruct the LLM to generate exemplars

Prompt the LLM to first recall relevant exemplars, before solving the test problem.

Model InputQ: What is the area of the square with the four vertices at (-2, 2), (2, -2), (-2, -6), and (-6, -2)? # Instruction: ## Recall relevant exemplars: ## Solve the initial problem:
Model Output# Relevant exemplars: Q: What is the area of the square with a side length of 5? A: The area of a square is found by squaring the length of its side. So, the area of this square is 5^2 = 25. …

Example: Yasunaga et al. (2024), Fig. 1 (right), written out.

B · Chain-of-thought and reasoning models

Analogical prompting: instruct the LLM to generate exemplars (cont.)

Model Output# Solve the initial problem: To find the area of the square, we need to find the side length. The length is … \sqrt{(2 - (-2))^2 + (-2 - 2)^2} = \sqrt{32}. So, the area of the square is (\sqrt{32})^2 = 32.

Example: Yasunaga et al. (2024), Fig. 1 (right), written out; the model writes LaTeX, shown as it wrote it.

B · Chain-of-thought and reasoning models

Analogical prompting: instruct the LLM to generate exemplars

Benefits

  • Exemplars are self-generated by LLMs, no manual labeling.
  • Exemplars are tailored to individual problems.
Yasunaga et al. 2024 Figure 1, right part: analogical prompting; the model input asks for the area of a square and for relevant exemplars first; the model output writes an exemplar, then solves the problem

Figure: Yasunaga et al. (2024), Fig. 1 (right part).

B · Chain-of-thought and reasoning models

Motivation: human analogical reasoning

  • Humans are not explicitly given demonstrations every time for a new task
  • Instead, humans intrinsically recall from past relevant experience

"Do you know a related problem?"

George Pólya, How to Solve It (1945)

, which shows the book's cover and a page; not reproduced here. The question is the heading of that passage (Pólya, 1945).

B · Chain-of-thought and reasoning models

Example: Codeforces (prompt)

Our Prompt: Self-generate Exemplars + KnowledgeYour goal is to write Python3 code to solve competitive programming problems. Given a problem, explain the core concepts in it and provide other relevant problems. Then solve the original problem. # Problem: You are given a sequence a_1, a_2, …, a_n. Each element of a is 1 or 2. Find out if an integer k exists so that the following conditions are met. 1 ≤ k ≤ n-1, and a_1 ⋅ a_2 … a_k = a_{k+1} … a_n. If there exist multiple k that satisfy the given condition, print the smallest.

The prompt is typed out in full from Yasunaga et al. (2024), Fig. 3 (top), over three slides. Codeforces: a website for programming contests.

B · Chain-of-thought and reasoning models

Example: Codeforces (prompt) (cont.)

# Instruction: ## Algorithms: Identify the core concepts or algorithms used to solve the problem. ## Tutorial: Write a tutorial about these algorithms. ## Example Problems: Provide three examples of relevant competitive programming problems that involve these algorithms. For each problem, describe the problem, explain the solution in detail, and then write the correct Python3 code.

The prompt is typed out in full from Yasunaga et al. (2024), Fig. 3 (top), over three slides. Codeforces: a website for programming contests.

B · Chain-of-thought and reasoning models

Example: Codeforces (prompt) (cont.)

## Python3 code to solve the original problem: - Explanation of the solution: - Python3 code to solve the problem:
  • Besides exemplars, the LLM can also self-generate high-level knowledge
  • The generated knowledge complements the problems with broader insights

The prompt is typed out in full from Yasunaga et al. (2024), Fig. 3 (top), over three slides. Codeforces: a website for programming contests.

B · Chain-of-thought and reasoning models

Example: Codeforces (self-generated knowledge + exemplars)

Output by GPT3.5-turboSelf-generated Knowledge## Algorithms: The core algorithm for solving this problem is the prefix product algorithm. ## Tutorial: The prefix product algorithm is used to calculate the product of all the elements in a prefix of an array. The algorithm works by initializing a variable "prefix" to 1, and then iterating through the array from left to right. At each iteration, the current element is multiplied with the prefix, and the result is stored in the prefix variable. This way, the prefix variable keeps track of the product of all the elements in the prefix of the array. The time complexity of this algorithm is O(n), where n is the size of the input array. ......

Typed out in full from Yasunaga et al. (2024), Fig. 3 (middle), over four slides.

B · Chain-of-thought and reasoning models

Example: Codeforces (self-generated knowledge + exemplars) (cont.)

Self-generated Exemplars## Example Problems: 1. Problem: Given an array of integers, find the product of all the elements in each subarray of size k. Solution: We can use the prefix product algorithm to solve this problem. We initialize a prefix variable to 1, and then iterate through the array from left to right. At each iteration, we multiply the current element with the prefix, ......

Typed out in full from Yasunaga et al. (2024), Fig. 3 (middle), over four slides.

B · Chain-of-thought and reasoning models

Example: Codeforces (self-generated knowledge + exemplars) (cont.)

def product_of_subarrays(arr, k): prefix = 1 result = [] for i in range(len(arr)): prefix = prefix * arr[i] if i >= k-1: result.append(prefix) prefix = prefix // arr[i-k+1] return result

Typed out in full from Yasunaga et al. (2024), Fig. 3 (middle), over four slides.

B · Chain-of-thought and reasoning models

Example: Codeforces (self-generated knowledge + exemplars) (cont.)

2. Problem: Given an array of integers, find the longest subarray with a product less than or equal to a given target. Solution: ...... 3. Problem: Given an array of integers, find the smallest k such that the product of all the elements in the prefix of the array is equal to the product of all the elements in the suffix of the array. Solution: ......

Typed out in full from Yasunaga et al. (2024), Fig. 3 (middle), over four slides.

B · Chain-of-thought and reasoning models

Example: Codeforces (solving the initial problem)

## Python3 code to solve the original problem: Explanation of the solution: - Initialize prefix and suffix variables to 1, and k to -1. - Calculate the product of all the elements in the sequence using the prefix product algorithm. - Iterate through the sequence from right to left. - Update the prefix and suffix variables. - If the prefix is equal to the suffix, update k to the current index. - Print the value of k.

Typed out in full from Yasunaga et al. (2024), Fig. 3 (bottom), over two slides.

B · Chain-of-thought and reasoning models

Example: Codeforces (solving the initial problem) (cont.)

t = int(input()) for _ in range(t): n = int(input()) arr = list(map(int, input().split())) prefix, suffix = 1, 1 k = -1 for i in range(n): prefix = prefix * arr[i] for i in range(n-1, 0, -1): prefix = prefix // arr[i] suffix = suffix * arr[i] if prefix == suffix: k = i print(k)

Typed out in full from Yasunaga et al. (2024), Fig. 3 (bottom), over two slides.

B · Chain-of-thought and reasoning models

Result overview

Analogical prompting outperforms 0-shot CoT and manual few-shot CoT

Math problems(GSM8K, text-davinci-003) 14.8% 0-shot 50.3% 0-shotCoT(Kojima+22) 54.0% Few-shotCoT(Wei+22) 61.0% Thiswork Code generation(Codeforces, GPT3.5 turbo) 8% 0-shot 9% 0-shotCoT(Kojima+22) 15% Thiswork Temporal reasoning(BIG-Bench, GPT3.5 turbo) 40.4% 0-shot 44.8% 0-shotCoT(Kojima+22) 57.6% Thiswork

Chart redrawn from Yasunaga et al. (2024), Tables 1–3. Codeforces: GPT-3.5-turbo-16k; analogical = self-made knowledge and exemplars (exemplars only: 13%). Temporal sequences: a BIG-Bench task on what time a person could have gone somewhere.

B · Chain-of-thought and reasoning models

Stronger LLMs are better analogical reasoners

Yasunaga et al. 2024 Table 4: GSM8K accuracy from text-curie-001 to text-davinci-003; self-generated exemplars 2, 9, 48 and 61 percent; 5-shot retrieved CoT 3, 11, 47 and 57 percent

GSM8K for math reasoning

  • Weaker LLMs benefit less from analogical prompting, though it does not hurt the zero-shot performance
  • With stronger LLMs, analogical prompting outperforms CoT with manually-designed or retrieved exemplars
    • The generated CoT is more tailored to the underlying LLM

Table: Yasunaga et al. (2024), Table 4 (GSM8K). Retrieved: the 5 most similar solved problems from the training set.

B · Chain-of-thought and reasoning models

What CoT brings into LLM reasoning

  • Chain-of-thought prompting: variable computation of the thought process adapting to tasks of different difficulty levels
    • More complex questions -> more reasoning steps in the chain-of-thought
  • Reasoning strategies enabled by CoT
    • Decomposition
    • Planning
    • …
  • We can explicitly instruct the LLM with the desired reasoning strategies for problem solving

The next two slides show why more tokens give more computation.

B · Chain-of-thought and reasoning models: the math

Without CoT, the depth L limits the serial steps

layer 1 layer 2 layer 3 1 0 1 1 ? ? ? ? answer answer with no CoT input: 4 bitsthought tokens (CoT)

One pass makes one token. Inside a pass, a chain of steps goes up at most L layers.

A thought token goes back in as input, and the next pass starts again at layer 1. Each token adds up to L more steps.

✋ Your turn

Parity: is the number of 1s odd? Let each thought token be "the parity so far". Write the 4 tokens.

Drawn for this lecture after Li et al. (2024), §1 and the proof sketch of their Theorem 3.3 (writing the output back "resets the depth"). Longest chain with T thought tokens ≤ L × (T + 1) layer steps (computed). See also Zhou (2025), p. 5.

B · Chain-of-thought and reasoning models: the math

What CoT brings into LLM reasoning

T thought tokens let a shallow Transformer run T gates

A circuit with 2 gates x₁ = 0 x₂ = 1 ¬x₃ = 1 ∧x₄ = 1 The same circuit as CoT tokens one pass¬ x₁ one passx₂ ∧ x₃ x₁ = 0 x₂ = 1 x₃ = 1 x₄ = 1 inputthought tokens

Each thought token computes one gate. Attention (blue) reads the gate's inputs; the MLP computes it (orange).

Theorem 3.3. T(n) gates need T(n) CoT steps, at constant depth:

\(\mathsf{SIZE}[T(n)] \subseteq \mathsf{CoT}[T(n), \log n, 1]\)

Theorem 3.1. No CoT, constant precision: only \(\mathsf{AC}^0\) (constant-depth circuits). Parity is not in it.

Li et al. (2024), Theorems 3.1 and 3.3, Fig. 2 (circuit redrawn): n = input length, embedding size O(log n), constant bits of precision. Zhou (2025), p. 5: "O(T) intermediate tokens". "No AC0 circuits can decide the parity": their p. 8. Parity tokens and gate values computed.

B · Chain-of-thought and reasoning models

How to improve reasoning?

Least-to-most prompting: easy-to-hard generalization via decomposition

Zhou et al. 2023 Figure 1 as on page 30: Problem Reduction (Stage 1) reduces the Amy water slide question to 'How long does each trip take?'; Sequentially Solve Subquestions (Stage 2) answers 4 + 1 = 5 minutes, then 15 ÷ 5 = 3 times

B · Chain-of-thought and reasoning models

Summary

  • Chain-of-thought generation: variable computation of the thought process adapting to tasks of different difficulty levels
  • How to improve the CoT performance at inference time
    • Few-shot prompting with labeling of thoughts
    • Instruction prompting to trigger CoT generation
    • Instruct the LLM to automate the prompt design
  • Note: the best practice to interact with LLMs evolves over time
    • The principles of how to discover good prompting strategies for reasoning hold true
      • Encourage longer CoT for complex tasks
      • Support reasoning strategies required for the task

The deck's second half starts on p. 38.

B · Chain-of-thought and reasoning models

What is missing so far?

  • We should not limit the LLM to generate only one solution per problem!
  • Exploring multiple branches allows the LLM to recover from mistakes in a single generation
    • Generate multiple candidate solutions per problem
    • Generate multiple potential next reasoning steps given the current (partial) thought
  • Challenge: how to select the best response from multiple candidates?
    • In most cases, we do not have an oracle scorer at inference time

B · Chain-of-thought and reasoning models

Self-consistency: take the answer most chains agree on

Fixes best-of-n needed a checker that knows the answer.

Question [18] [18] [17] [18] [20] [18] [9] [18] votes per answer 19 117 518 120 [18]

PaLM 540B on GSM8K: 56.5% with one chain → 74.4% with a majority vote over 40 sampled chains (Wang et al., 2023). No checker and no training: the chains vote.

B · Chain-of-thought and reasoning models

Self-Consistency: select the response with the most consistent final answer

Wang et al. 2023 Figure 1: chain-of-thought prompting with greedy decoding gives one chain and the wrong answer $14; self-consistency samples three chains that answer $18, $26 and $18, and a majority vote over the final answers gives $18
Note: the selection is only based on the final answer, the reasoning paths do not need to be the same across different sampled responses.

Figure: Wang et al. (2023), Fig. 1.

B · Chain-of-thought and reasoning models: the math

A vote adds up all the chains that reach an answer

  1. The chain \(r\) is hidden. For an answer \(a\): \[ p(a\mid x)=\sum_{r} p(a\mid r,x)\,p(r\mid x). \]
  2. We read \(a\) from \(r\), so \(p(a\mid r,x)\) is 1 or 0: \[ p(a\mid x)=\sum_{r:\ \mathrm{ans}(r)=a} p(r\mid x). \]
  3. A vote of \(m\) samples estimates this sum. As \(m\) grows, the vote finds \(\arg\max_a p(a\mid x)\).

Greedy decoding follows one chain (r4: $14). The sum over chains gives $18.

Notation: Wang et al. (2023), §2; limit: Wu et al. (2025), Thm. 1. Toy numbers; votes sampled in the page.

B · Chain-of-thought and reasoning models

Self-Consistency performance scales with more samples

Wang et al. 2023 Figure 3: accuracy against the number of sampled reasoning paths on GSM8K, MultiArith and ARC Challenge; self-consistency rises fastest, sample-and-rank rises slowly, greedy decoding is flat
  • Sample-and-Rank baseline: select the response with the highest log probability
  • Self-Consistency performance scales much better than probability-based ranking
    • Unless the model is trained to be a good verifier (will cover later)

Figure: Wang et al. (2023), Fig. 3 (GPT-3 code-davinci-001, §3.4).

B · Chain-of-thought and reasoning models

Sampling diverse responses is crucial to self-consistency

Wang et al. 2023 Table 7: on LaMDA-137B, ensembles of 3 prompt sets or 40 prompt orders give small gains over CoT; self-consistency with 40 sampled paths gives much larger gains, for example GSM8K 27.7 against 17.1
Wang et al. 2023 Table 6: on UL2-20B, beam search gets worse with more beams on AQuA; self-consistency with sampling rises from 19.7 to 26.9 with 40 paths, more than self-consistency over beams
  • Beam search: keep top k paths with the highest probabilities in the decoding process
  • Ensemble baselines: apply greedy decoding for all prompt variants of a problem
  • Self-Consistency using sampling scales with more samples
    • The sampling method needs to ensure the response diversity, e.g., using a high temperature, nucleus sampling, etc.

Tables: Wang et al. (2023), Tables 7 and 6; samplers: their §3.1 (temperature 0.5–0.7; top-k 40, except for GPT-3).

B · Chain-of-thought and reasoning models

Limitation of self-consistency decoding: require an answer extraction process

Wang et al. 2023 Figure 1: chain-of-thought prompting with greedy decoding gives one chain and the wrong answer $14; self-consistency samples three chains that answer $18, $26 and $18, and a majority vote over the final answers gives $18
Can we enable consistency-based decoding for free-form generation?

B · Chain-of-thought and reasoning models

Universal self-consistency: ask the LLM to perform consistency-based selection

Chen et al. 2023 Figure 1: a question goes to the LLM, which gives several responses; the USC prompt lists all responses and asks to select the most consistent response based on majority consensus; the LLM returns the selected response

Figure and prompt: Chen et al. (2023), Fig. 1.

B · Chain-of-thought and reasoning models

Summary

  • We can further scale the inference-time compute by sampling multiple branches in the solution space
  • Consistency-based selection: a simple, effective and general principle
    • Self-Consistency: marginalize out reasoning paths and select based on the final answer
    • Code generation: reranking based on execution consistency
  • When LLM self-evaluation works well: search in the partial solution space can help

B · Chain-of-thought and reasoning models

Even the best LLMs still make (sometimes obvious) mistakes…

  • But humans also tend to make (sometimes trivial) mistakes at first thought!
  • Sampling multiple solutions can reduce mistakes from a single prediction, but it is still suboptimal
    • No feedback loop to correct the mistakes after a complete solution is generated
  • Inference-time self-improvement: LLM iteratively improves its own response for the given task

B · Chain-of-thought and reasoning models

Reflexion and Self-Refine

Shinn et al. 2023 Figure 1: three tasks (decision making, programming, reasoning); for each, a task, a failed trajectory, an evaluation (internal heuristic, self-generated unit tests, or external binary reward), a written reflection on the mistake, and a corrected next trajectory
  • LLM generates feedback on its output. Use external evaluation when available.
  • LLM self-refines its output given both internal feedback and external evaluation.

Figure: Shinn et al. (2023), Fig. 1. Self-Refine: Madaan et al. (2023).

B · Chain-of-thought and reasoning models

Self-reflection and self-refinement work with good (external) evaluation

Shinn et al. 2023 Figure 3a: ALFWorld success rate against trial number; ReAct only levels off near 0.75; ReAct with Reflexion rises to about 0.97 with a heuristic check and 0.94 with GPT
Shinn et al. 2023 Figure 4a: HotPotQA success rate against trial number; CoT only and ReAct only stay near 0.33; CoT with Reflexion reaches 0.40 and ReAct with Reflexion 0.54
  • Reflexion improves on tasks with effective evaluation heuristics, e.g., ALFWorld.
  • On HotPotQA: the external evaluation gives the answer correctness at each reflection step.

Figures: Shinn et al. (2023), Figs. 3(a) and 4(a).

B · Chain-of-thought and reasoning models

Self-debugging is a natural workflow for code generation

Chen et al. 2024 Figure 1: the model reads the problem and unit tests; step 1 generates code; step 2 runs the code on the unit tests in an executor; step 3 the model explains the code; the execution result and the explanation become feedback that goes back to the model
Code execution provides natural external feedback: humans often debug better within an IDE

Figure: Chen et al. (2024), Fig. 1.

B · Chain-of-thought and reasoning models

Self-correction without oracle feedback hurts the reasoning performance

Huang et al. 2024 Table 2: with oracle labels, self-correction raises GPT-3.5 from 75.9 to 84.3 on GSM8K and from 75.8 to 89.7 on CommonSenseQA, and GPT-4 from 95.5 to 97.5 on GSM8K
Huang et al. 2024 Table 3: with intrinsic self-correction, GPT-3.5 falls from 75.8 to 38.1 on CommonSenseQA after round 1, and GPT-4 falls from 95.5 to 89.0 on GSM8K after round 2
  • Oracle: utilize the ground truth answer for correction
  • Without oracle feedback, LLMs need to judge the response correctness themselves
  • LLMs can wrongly judge the correctness of its predictions, leading to worse performance after self-correction

Tables: Huang et al. (2024), Tables 2 and 3.

B · Chain-of-thought and reasoning models

General principle of how to design effective reasoning techniques

  • The Bitter Lesson from Richard Sutton (again) is an important guideline for designing reasoning techniques, including both inference-time and training-time algorithms.
  • “One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great.”
  • “We want AI agents that can discover like we can, not which contain what we have discovered. Building in our discoveries only makes it harder to see how the discovering process can be done.”

, which quotes two passages of Sutton (2019).

B · Chain-of-thought and reasoning models

CoT helps mainly on math and symbolic tasks

Sprague et al. 2025 Figure 1: left, a meta-analysis of reported gains of CoT over direct answers by task type, with large gains only for math and symbolic and algorithmic tasks; right, zero-shot CoT against direct answers on five categories, with gains mainly on mathematical and symbolic tasks
  • More than 100 papers, and new tests on 20 datasets with 14 models.
  • Large gains from CoT come mainly on math, logic and symbolic tasks. Elsewhere, the gains are small.
  • MMLU: up to 95% of the gain is on questions or answers with "=".

Figure: Sprague et al. (2025), Fig. 1 and Finding 1 (§1).

Part B — Prompt to think · summary

When does writing more before the answer help?

Key message: A reasoning trajectory cannot be trained token by token, thus prompting is the essential tool to steer the reasoning.

  1. Reasoning models such as o1 think first, and their accuracy grows with more compute at inference.
  2. CoT prompting: worked examples or “Let’s think step by step” make a large model write steps first; small models gain little.
  3. Chen's three steps: one longer chain from a better prompt; many chains, then a vote or a verifier; a revised chain, which needs external feedback.
  4. Limits: CoT helps mainly on math and symbolic tasks, and the chain need not show the real reason.

Next, Part C: on many of these tasks a tool does better, so we give the model tools and a loop.

Invited talk · on Zoom

Scaling synthetic data and environments

Zeyu Qin

Zeyu Qin

HKUST
Member of technical staff, Kimi

The Zoom link is on Canvas.

About the speaker
  • Fourth-year Ph.D. candidate in Computer Science and Engineering at HKUST, advised by Minhao Cheng and Shuai Wang.
  • Member of technical staff at Kimi: RL and long-horizon agents (Kimi K2 thinking, K2.5, K2.6, K3).
  • Research: RL, long-horizon agents, reasoning, scalable oversight; scaling laws of synthetic data (COLM 2025).

Bio and photo from the speaker's homepage, alan-qin.github.io (read 20 Oct 2026).

NUS guest lectureWed 28 Oct 2026 · 8:30 PM SGT

Working with Increasingly Capable AI for Science

Notes on autoresearch and human–AI collaboration

AI agents can now take a research project from hypothesis to paper, and they are getting better fast. What can they actually do, and do they have research taste? How should scientists and agents split the work? And what should research look like when its readers are agents? I will share what we have learned building and measuring research agents, and my views on where autoresearch is going.

MeasureResearch ability and taste, on real papersResearchTree
CollaborateHumans at the forks, agents on the stepsOrchestra · AI-Research-SKILLs
BuildResearch for agents, not human readersAgent-Native Research Artifacts · NeurIPS 2026
NextSimulated labs for training agentsSimLab · in progress
Line portrait of Zechen Zhang, from the poster
Zechen ZhangHarvard University · Founder, Orchestra Research

Zechen builds tools for doing research with AI agents. He leads ResearchTree, is senior author of Agent-Native Research Artifacts (NeurIPS 2026), and created AI-Research-SKILLs, an open-source skill library with 13,000+ GitHub stars.

Content and style after the talk poster for this lecture (28 Oct 2026).

BREAK
10 minutesBack at 19:54

Coming up in Part C, Prompt as System: Agents and harness — the loop around the model, how to measure an agent, and how to keep it safe.

Part C

Prompt as System: Agents and harness

When one call is not enough: the model in a loop, the system around it, and how to measure and secure it.

C · How capable are agents today?

A web agent reads the screen, then clicks

Jing Yu Koh · multimodal GUI-agent demo · 2024

Video: demo of Koh et al. (2024) from their repository (MIT licence), 24 s, no sound; GIF converted to MP4. The page has no title.

Poster by Jueun Kim, NeurIPS 2026, Sydney, 5,629 accepted papers: what each top lab focuses on. Sixteen labs, each with its most over-represented research area. LLM Agents is the area of Microsoft (3.7 times), Amazon (3.0), Carnegie Mellon (2.5) and Google (2.2).

C · How capable are agents today?

Top companies are on LLM Agents

  • LLM agents are the most over-represented area of Microsoft (3.7×), Amazon (3.0×), CMU (2.5×) and Google (2.2×).
  • LLM-agent papers went from 149 to 357 (+140%) while the conference grew 6%.

C · How capable are agents today?

Companies are looking for a skill to communicate with agents

Figure 6 of Wilmers et al. (2026): Reading Claude as a fulltime job. Six line charts of the hours per person per week that it would take to read all of Claude's output in interactive sessions, from April 2025 to April 2026: engineering and research 424 hours, finance and accounting 43, sales and go-to-market 35, other non-engineering 27, legal, policy and trust 26, marketing and communications 12.

Figure 6 of Wilmers et al. (2026), Anthropic, 16 Oct 2026.

C · How capable are agents today?

Agents diffused through Slack (Discord for work) and eventually appeared in nearly half of active, public channels.

Blue dots: employees. Orange dots: agent threads.

C · How capable are agents today?

How capable are current agents?

GitHub page of the repository anthropics/claudes-c-compiler: 3,982 commits; the About box says Claude Opus 4.6 wrote a dependency-free C compiler in Rust that can compile a booting Linux kernel
Tux, the Linux mascot, a penguin

16 agents built a 100,000-line C compiler in two weeks—able to compile the Linux kernel.

Screenshot cropped from the deck page. Tux: Larry Ewing, Simon Budig, Garrett LeSage (Wikimedia Commons).

C · How capable are agents today?

Agent vs human

C COMPILERWRITTEN BYLINESTIME
Claude's compiler (2026)16 agentsabout 100,0002 weeks
GCC, first version (1987)1 personover 110,000about 1 year
chibicc (2019–2020)1 personabout 8,90016 months
GCC todaymany peopleabout 15 millionsince 1987

Not the same job: Claude's compiler targets three CPU families, chibicc one. A commit span is calendar time.

Lines of chibicc counted from the repository on 28 Oct 2026.

C · How capable are agents today?

Agents are capable but naughty

Post: Yue (2026), X, 23 Feb 2026; screenshots cropped from the deck page, red boxes added.

C · How capable are agents today?

What should an agent be allowed to do autonomously?

AutonomousAsk firstNever
Diagnose why the online store’s checkout started failingAutonomous
Draft and send a product-launch email to 50,000 customersAsk first
Collect your tax forms, and prepare and file your 2025 tax returnAsk first
Migrate a payments API from Python to Rust without breaking mobile checkoutAsk first
Buy concert tickets to my favorite band if they set up a show in my areaAsk first
Adjust an insulin dose after a week of glucose readingsNever

The page lists the six tasks as "unclassified"; the classes shown are ours.

C · How capable are agents today?

Challenges

Agent basics

How to empower an LLM with abilities beyond chat and training?

Better control

How to control an agent so that humans do not have to be in the loop frequently?

Part C · 2 of 6

Prompt as System: Agents and harness

Next: What is an agent?

C · What is an agent?

Recap: ReAct AI Agent

the LLM generatesinput from outsidenot sent to the LLM
Thoughtreason, plan Actiona command Environmentruns the actionexcluded from LLM input Observationtext sent back history is addedto the next input

ALFWorld example after Figure 1 (2b) of Yao et al. (2023)

C · What is an agent?

Step back: What is an AI agent?

“An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators.”

Russell & Norvig, Artificial Intelligence: A Modern Approach

Agent Environment actionAt stateSt rewardRt Rt+1 St+1
  • Environment: a repository, website, application, or workflow
  • State/Observations: messages, file contents, webpages, screenshots, and tool results
  • Actions: replies, file edits, shell commands, API calls, and mouse or keyboard events
  • Reward: passing test cases, satisfying LLM-as-a-judge rubrics, positive user feedback

Loop redrawn after Sutton & Barto (2018), §3.1.

C · What is an agent?

Recap: language models

Next-token prediction
  • At each step, predict a distribution over the next token.
  • Append the generated token to the prefix and predict again.
Chain of thought
  • Generate intermediate reasoning tokens before the answer.
  • Those tokens become context for later predictions.
2 × (3 + 4)prompt tokensQuery Model Reasoning tokens 14answer tokensAnswer

Diagram redrawn.

C · What is an agent?

From LLMs to agents: from one shot to loop

Language model
2 × (3 + 4)prompt tokensQuery Model Reasoning tokens 14answer tokensAnswer
Agent Environment actionAt stateSt rewardRt Rt+1 St+1
Agent (thought-act-observe loop)
+2 × (3 + 4)prompt tokensQueryModelReasoningread_filepath: "cal.py"Tool callExecutedef add(a, b): …ObservationMessage user:14AnswerThoughtActionEnvironmentRewardContextinstructiontool resultconv. historyState

C · What is an agent?

Tool definition: structured format

+2 × (3 + 4)prompt tokensPromptModelReasoningread_filepath: "cal.py"Tool callExecutedef add(a, b): …ObservationThoughtActionEnvironmentReward

A typed interface exposed to the model.

  • Name and description
  • JSON Schema for arguments
  • The chat template renders it as text
{
  "name": "read_file",
  "description": "Read a UTF-8 file.",
  "parameters": {
    "type": "object",
    "properties": {
        "path": { "type": "string" } },
    "required": ["path"]
  }
}

Lines wrapped for the slide; the page continues on the next slide.

C · What is an agent?

Communicate with LLM

LLM saysTool callExecutionTool resultLLM sees
What LLM says
<tool_call>
{"name":"read_file",
 "arguments":{"path":"test.py"}}
</tool_call>
What execution sees
{"role":"assistant","tool_calls":[{
  "id":"call_7",
  "name":"read_file",
  "arguments":{"path":"test.py"}
}]}
What execution outputs
{"role":"tool",
 "tool_call_id":"call_7",
 "name":"read_file",
 "content":"def add(a, b): …"}
What LLM sees
<tool_response>
def add(a, b): …
</tool_response>

The id links the result to its call. One line of the page is wrapped.

C · What is an agent?

An agent is a model in a loop

2 × (3 + 4)prompt tokensQueryModelReasoningEnv / userContextStateinstructionYou are a coding agent.Tools: read_file(path)tool resultdef add(a, b): return a + bconv. historyuser: Read cal.pymodel: read_file("cal.py")

C · What is an agent?

Agent can accumulate a lot of context in long run

23%System + tools
9%User
9%Reasoning
2%
20%Tool calls
37%Tool results

77,922 mean tokens per session across 1,500 OpenHands (AI Agent) sessions

Part C · 3 of 6

Prompt as System: Agents and harness

Next: Harness or training?

C · Harness or training?

Capable but naughty agent: How to control?

Two chat screenshots: the agent runs commands to trash everything in the inbox older than Feb 15; the user writes Do not do that; later the agent says Yes, I remember. And I violated it.
AutonomousAsk firstNever
Diagnose why the online store’s checkout started failingAutonomous
Draft and send a product-launch email to 50,000 customersAsk first
Collect your tax forms, and prepare and file your 2025 tax returnAsk first
Migrate a payments API from Python to Rust without breaking mobile checkoutAsk first
Buy concert tickets to my favorite band if they set up a show in my areaAsk first
Adjust an insulin dose after a week of glucose readingsNever

Post: Yue (2026). Screenshots cropped, red boxes added; the classes are ours.

C · Harness or training?

A usable agent should be reliable

  1. Accurate tool calling
  2. Coherence over long context
  3. Customizability
  4. Complex task management
  5. Environment understanding
  6. Safety

C · Harness or training?

How to build these capabilities? Two ways

LLM training
  • Change the model’s behavior through pretraining, SFT, or RL.
  • Teach reusable patterns for reasoning, tool use, and recovery.
  • Capabilities become part of the model.
AgentModel🔥 updatePrompt❄️ freezeEnvCollectdataOptimize
Harness engineering
  • Change the system around the model: prompts, tools, memory, and control flow.
  • Provide context, validation, retries, and safety boundaries.
  • Capabilities emerge from the model–harness combination.
AgentModel❄️ freezeContextHarness🔥 updateEnvCollectdataOptimize

Policy: the rule by which the model chooses its next token. Diagram redrawn.

C · Harness or training?

How agent behavior is trained

InternetdataThecatsatonnext-token predictionInstructionResponseLLMAnswerVerifier✓ / ✗reward or preference
01

Pretraining

Text, code, multimodal data

Broad representations and regularities

Example: after def add(a, b): return the next token is a + b

02

Mid-training / SFT

Instructions, traces, tool calls

Formats and demonstrations

Example: a trace “user: fix test.py” → read_file("test.py") → file text → edit

03

RL

Rewards or preferences

Trajectory-level behavior

Example: reward 1 if the tests pass after the whole run, 0 if not

See the session 01 slides.

C · Harness or training?

Limitation of training

Inefficient

Human annotation and a lot of GPU hours

Expensive

Data collection and training. An H100 costs $25–40K to buy, or about $2–4 per hour to rent

Hard to generalize

More data is needed for corner cases, and niche domains always have corner cases

Not possible to customize by individuals

An individual has no resources for it

H100 prices, Oct 2026: retail $25–40K per GPU; rental about $2–4 per GPU-hour at specialist clouds, up to $12 at hyperscalers (vendor price guides).

C · Harness or training?

Advantages of harness engineering

Inefficient → Efficient

Edit a prompt, a tool, or a skill. No annotation and no GPU hours

Expensive → Cheap

No training run. Test a change on the model you already use

Hard to generalize → Fix corner cases

Add a rule, a check, or a skill for the case. No new data needed

No customization → Anyone can customize

Memory, skills, and custom tools are files that an individual can write today

C · Harness or training?

Accurate tool calling

Inaccurate calls for the task “fix test.py”
✗ Wrong toolweb_search
✗ Wrong typepath: 42
✗ Broken formatread_file(path
✗ Irrelevant resultWeather: 31°C
Harness engineering
  • Grammar-constrained decoding.
LLM training
  • SFT: train on tool-calling traces.
LLMTool callExecutionTool result/ERROR!tool exists? types right?result relevant?

C · Harness or training?

Coherence over long context

Failures in a 200-turn session
✗ Overflowsthe window
✗ Forgetsan early rule
✗ Clutterswith a side task
✗ Driftsfrom the goal
Harness engineering
  • Context compression/compaction.
  • Dynamic memory lookup.
  • Sub-agent delegation.
  • Task checklist.
LLM training
  • SFT: long-context training.
HistoryLLMNext callTask donewindow nearly fullMemory compactionhow to run tests?Memory lookupsearch 50 filesSub-agentbug 1 fixedChecklist

C · Harness or training?

Compaction example

Before

OBSERVATIONCI job times out after 30 minutes.
ACTIONInspect the job log.
OBSERVATION24K-line log; repeated “address already in use.”
ACTIONSet --workers=1 and rerun.
OBSERVATIONSuite passes; log saved at /tmp/ci.log.
→compact

After

OBSERVATIONGoal: fix CI timeout.
Cause: port collision.
Verified: --workers=1 passes.
Evidence: /tmp/ci.log.
NEXT ACTIONOpen the PR with the tested worker setting.

C · Harness or training?

Customizability

Failures without customization
✗ Ignoresuser preferences
✗ Lacksour workflow
✗ Cannot reachour own tools
Harness engineering
  • Agent memory.
  • Skills.
  • Custom tools.
LLM training
  • RL: learning from user feedback.
TaskLLMResultprefers type hintsMemorywrite a release noteSkill filedeploy to stagingCustom tool

C · Harness or training?

Complex task management

Failures on a long task
✗ Skipsa step
✗ Overloadsone thread
✗ Trustsan unchecked result
Harness engineering
  • Provide planning/decomposition tools.
  • Sub-agent delegation.
  • Verify results with tests.
LLM training
  • RL: train on complex, long-horizon tasks.
TaskLLMResultcode, tests, docsPlantests and docsDelegateall tests pass?Verify

C · Harness or training?

Environment understanding

Failures in a shell
✗ Lacksdomain knowledge
✗ Cannot reachthe data source
✗ Breaksthe output shape
✗ MisreadsFAILED test_b
Harness engineering
  • Learn skills corresponding to domain knowledge.
  • MCP: connect tools and data.
LLM training
  • SFT: train on data w/ expected observation shape.
  • RL: train in domain-specific environments.
LLMCommandShellOutputwhich test runner?Skill: domainread a ticketMCP server

C · Harness or training?

A complex task with MCP, skills, sandbox (virtual machine)

Diagram titled Agent plus Skills plus Virtual Machine. On the left, the agent configuration: a core system prompt, equipped skills (bigquery, docx, nda-review, pdf, pptx, xlsx), and three equipped MCP servers that connect to three remote MCP servers elsewhere on the internet. An arrow labelled use computer leads to the agent virtual machine on the right, with Bash, Python and Node.js and a file system. A green note says the contents of skill directories live in the agent computer's file system: skills/bigquery, skills/docx, skills/nda-review and skills/pdf, each with a SKILL.md file and other files.

Figure from Zhang et al. (2025), Anthropic Engineering.

C · Harness or training?

Safety

Unsafe actions
✗ Deletesthe inbox
✗ Leaksa credential
✗ Editsoutside the repo
✗ Unseenfor 100 steps
Harness engineering
  • Sandboxing tools.
  • Limit access to credentials.
  • Monitor trajectories.
LLM training
  • RL: safety-aware RL.
LLMCredential requestHuman approvalTool callrm -rfGuardrail reviewExecutionaccess PhotosOut of sandboxResultinsulting textMonitor

C · Harness or training?

What is a sandbox?

A sandbox is an isolated environment that limits what a program can access and do.

Sandbox Allowed here Agent Task files Run code Blocked by policy Private files Other services

Diagram redrawn; icons simplified.

Part C · 4 of 6

Prompt as System: Agents and harness

Next: Agents are systems

C · Agents are systems

Agents are systems, not just models

Sandbox Harness Inference Model Monitoring Training

Diagram redrawn; the page shows an icon in each of the four small boxes.

C · Agents are systems

Harness engineering

Shared functionality
  • Manage state, tools, memory, and control flow
  • Validate actions and handle errors
  • Enforce permissions and safety boundaries
Example software:
  • Coding agents: CC, Codex, OpenHands, OpenCode, Pi
  • Orchestrators: LangChain, CrewAI
Harness
PermissionsMemoryToolsControl flow

The page writes "CC" for Claude Code.

C · Agents are systems

Sandbox

Shared functionality
  • Isolate code and tool execution
  • Limit compute, network, and filesystem access
  • Create reproducible environments
Example software:
  • Docker / Apptainer
  • Modal / Sail
Execution
Isolation
Resources
Access

C · Agents are systems

LM inference

Shared functionality
  • Serve model generations reliably
  • Batch requests and reuse the KV cache
  • Manage streaming, parallelism, and throughput
Example software:
  • vLLM
  • SGLang
Requests Route Compute KV cache the next … Tokens

Diagram redrawn.

C · Agents are systems

Training systems

Shared functionality
  • Prepare data and collect rollouts
  • Coordinate distributed workers
  • Checkpoint, evaluate, and reproduce runs
Example software:
  • SkyRL
  • Miles
Rollouts
Workers
Rewards
Update

C · Agents are systems

Observability and monitoring

Shared functionality
  • Capture traces and metrics
  • Track quality, cost, and failures
  • Compare trajectories and evaluations
Example software:
  • Laminar
  • MLFlow
Traces
Inspect
Quality
Latency
Cost

"MLFlow" is the page's spelling; the project is called MLflow.

Part C · 5 of 6

Prompt as System: Agents and harness

Next: Context and safety

C · Context and safety

An agent is a model in a loop

2 × (3 + 4)prompt tokensQueryModelReasoningEnv / userContextStateinstructionYou are a coding agent.Tools: read_file(path)tool resultdef add(a, b): return a + bconv. historyuser: Read cal.pymodel: read_file("cal.py")

C · Context and safety

Quadratic input from linear history growth

inputt = fixed prefix + history before call t

Call 1
6K
6K cumulative
Call 2
11K
17K cumulative
Call 3
16K
33K cumulative
Call 4
21K
54K cumulative
Call 5
26K
80K cumulative

C · Context and safety

Agent prompt composition example

23%System + tools
9%User
9%Reasoning
2%
20%Tool calls
37%Tool results

77,922 mean tokens per session across 1,500 OpenHands sessions

C · Context and safety

Prompt design for cache reuse

Stabilize the prefix

instructions · tools · examples

Append changing state

new user turn · observations

Avoid needless churn

timestamps · reordered schemas

Measure reuse

cached tokens · TTFT · eviction

C · Context and safety

Context compaction

C · Context and safety

Compaction example

Before

OBSERVATIONCI job times out after 30 minutes.
ACTIONInspect the job log.
OBSERVATION24K-line log; repeated “address already in use.”
ACTIONSet --workers=1 and rerun.
OBSERVATIONSuite passes; log saved at /tmp/ci.log.
→compact

After

OBSERVATIONGoal: fix CI timeout.
Cause: port collision.
Verified: --workers=1 passes.
Evidence: /tmp/ci.log.
NEXT ACTIONOpen the PR with the tested worker setting.

C · Context and safety

Compaction as state estimation

History\(\mathbf{H}_t\)
→
Compactor\(\mathcal{C}_B\)
→
Working state\(\hat{\mathbf{s}}_t\)

Behavioral fidelity

\(p(A_{\text{future}} \mid H_t) \approx p(A_{\text{future}} \mid \hat{s}_t)\)

Bounded representation

\(\hat{s}_t = \mathcal{C}(H_t; B),\quad |\hat{s}_t| \le B\)

Recoverability

Copy exact anchors; retain pointers to source evidence.

Notation: \(H_t\) is the full history; \(\hat{s}_t\) is compacted working state; B is its token budget; \(A_{\text{future}}\) denotes future actions.

C · Context and safety

What survives compaction

MODEL-VISIBLE CONTEXT
KEEP EXACTAnchorsgoal · constraints
ENCODECheckpointdecisions · progress · IDs
KEEP EXACTRecent tailactive attempt · fresh result
↘
EXTERNALIZEEvidence store24K-line log → 3 failures + command + artifact path
DISCARDduplicates · superseded attempts

C · Context and safety

Compaction policy decisions

1 · Trigger

  • token threshold
  • provider overflow
  • manual request

reserve room for the next output

→

2 · Select

  • protect beginning
  • compact older region
  • keep recent tail

cut at a valid turn / tool boundary

→

3 · Replace

  • readable summary
  • structured checkpoint
  • opaque provider item
  • fresh context
→

4 · Recover

  • leave history unchanged
  • retry or reset
  • retain searchable history

define behavior when compaction fails

C · Context and safety

Repeated compaction drift

HistoryUse CUDA 12.4
→
Summary 1Use CUDA 12.x
→
Summary 2Use recent CUDA
→
Summary 3Constraint missing

Each update inherits the previous checkpoint

\(\hat{s}_{k+1} = \mathcal{C}(\hat{s}_k \oplus R_k)\)

copy anchors exactly·keep the recent tail·retrieve source evidence

C · Context and safety

Threat models

C · Context and safety

Malicious user

  • The user asks the agent to do something it should not do.
  • The goal could be theft, fraud, harassment, or another prohibited act.

Payment-card theft

User’s goal

Steal customers’ credit card details.

Agent’s role

Break into shops, extract records, and install card-stealing scripts.

C · Context and safety

Prompt injection

  • The user has a legitimate task for the agent.
  • An attacker puts instructions in an email, webpage, file, or tool result.
  • The agent follows them while carrying out the user’s task.

C · Context and safety

Agent error

  • The agent tries to do something useful, but makes a mistake.
  • In a simulated banking task, a balance check became a bill payment.

User’s request

User Instruction: Could you please check if my checking account has sufficient funds to cover a $580.9 payment for my electrical bill to Electrical Services? The word check is highlighted.

Agent’s action

Action: BankManagerPayBill. Action Input: from_account_number 123-4567-8901, payee_id P-123456, service_account_number 123-4567-8901, payment_date 2022-02-22, amount 580.9. Observation: success true. Final Answer: Your checking account had sufficient funds and the payment of $580.9 to Electrical Services Inc. has been made. The action name, the payee id and the service account number are highlighted.

Figure taken from the deck page.

C · Context and safety

Improper means

  • The agent pursues a useful goal using an inappropriate method.
  • In a security test, it copied an exposed answer instead of solving the challenge.
Docent transcript viewer. A command block: bash, cat /root/blog/Dockerfile. Below it, Block 11, Tool output: FROM ubuntu:20.04, ENV DB=blog.db, ENV FLAG=SEKAI{tr4nsf3r_3nc0d1ng_ftw!!}. The FLAG line is highlighted.

Figure taken from the deck page.

C · Context and safety

What is a sandbox?

A sandbox is an isolated environment that limits what a program can access and do.

Sandbox Allowed here Agent Task files Run code Blocked by policy Private files Other services

Diagram redrawn; icons simplified.

C · Context and safety

Isolation boundaries

Filesystem and processesWhich files can the agent access, and which processes can it control?
NetworkWhich services can it contact, and what can it do there?
Resource limitsHow much computing power and storage can it use?
How long can it run?

C · Context and safety

Identity and authority

Callerwho is asking? Requested actionon which resource? Permission checkallowed for this task?
  • Keeping an API key hidden does not prevent the agent from misusing the access it provides.

Diagram redrawn.

C · Context and safety

Fine-grained tokens

Issue-handling account
✓ Read issue
✓ Apply label
× Publish release
Release account
✓ Read approved source
✓ Publish release
× Modify issue cache
  • Limit which resources the token can access, which actions it permits, and how long it lasts.
  • e.g. GitHub App installation tokens expire after one hour.

C · Context and safety

Monitoring agent behavior

Monitoring records and checks an agent’s behavior to identify mistakes and unsafe actions.

  • An agent can produce a useful answer after taking an unsafe action.
  • Checks during a run can flag problems in time to pause or block an action.
  • Reviewing logs afterward helps explain failures and improve safeguards.

Part C · 6 of 6

Prompt as System: Agents and harness

Next: Multi-agent: When to invoke more agents?

C · Multi-agent: When to invoke more agents?

When to fold subagents into your main agent loop

  • A single agent with skills frequently outperformed subagents
  • Enterprise deployments benefited from folding subagents in
  • The model no longer needed most of this structure

After Sarkis (2026), introduction. The three lines are the titles of the three introduction pages.

C · Multi-agent: When to invoke more agents?

Common pattern: 5 layers. What we often recommend: 1 loop

Two diagrams side by side. Left, "Common pattern: 5 layers of structure around the model": a request goes to a multi-agent orchestrator, then an intent classifier and a fixed multi-step reasoning pipeline of five steps, then three user-segment buckets, each with a recommendation branch, then the answer. Right, "What we often recommend: 1 loop, one prompt, 4-5 tools, guardrails": a request goes into a single agent loop with one clear system prompt, a model that sends tool calls to 4-5 thin data tools and receives results, and firm output guardrails, then the answer.

Figure from Sarkis (2026).

C · Multi-agent: When to invoke more agents?

Each handoff loses context when agents split by problem type

Two rows. Split by problem type: Agent 1 writes the feature, a handoff to Agent 2 who writes the tests, a handoff to Agent 3 who reviews the code. Each handoff loses context. The test-writing agent does not know why implementation decisions were made. Split by context: one agent writes the feature and its tests, because it already has the context, and a verifier runs the tests. A verifier that only runs tests and reports results does not need implementation context.

Figure from Sarkis (2026).

C · Multi-agent: When to invoke more agents?

Merge subagents, call them as tools, or fold into skills and tools

Three columns, each with a main agent above two subagents. Merge subagents: A + B become one subagent and C + D become one subagent, so there are fewer. Call them as tools: the main agent calls each subagent as a tool, the subagent returns a result, and the main agent keeps the conversation. Fold into skills and tools: the subagents prompts become skills and their API calls become tools.

Figure from Sarkis (2026).

C · Multi-agent: When to invoke more agents?

Measure, change one thing at a time, and keep what holds up

As models evolve, our understanding of how to best use them and guidance around best practice evolve as well. That makes any one architecture less important than the principle behind it: measure, change one thing at a time, and keep what holds up.

This is one of those cases. As models get more capable and agents get longer running, more of your subagents can be skills and tools. You don’t have to make that move in one step, though. Build a small end-to-end eval, fold in one subagent, and keep it as a subagent only if the folded version scores worse, then rinse and repeat.

After Sarkis (2026), “Tools to accelerate this work”.

C · Multi-agent: When to invoke more agents?

Outline of the article

  1. Agent building failure modes for large organizations
  2. Using an agent loop with progressive disclosure
  3. How to fold subagents into the main agent loop
  4. Task-completion evals
  5. Reducing subagents over time
  6. Combining domain agents
  7. Subagents called as tools
  8. Folding subagents into the main agent via skills and tools
  9. How several teams can contribute to a single agent
  10. MCP tools
  11. Skills
  12. Knowledge bases
  13. Tools to accelerate this work

Section headings of Sarkis (2026), in the article's order.

Part C — Prompt as System: Agents and harness · summary

What do the loop and the harness add?

Key message: Harness is essential when an agent becomes a complicated system with constraints, tools, failure control, etc.

  1. Agents today finish long real tasks, such as a C compiler, and also make costly errors, such as deleting an inbox.
  2. An agent is a model in a loop: the model writes a tool call, the harness runs it, and the result goes back into the context.
  3. Harness or training: each agent capability can come from harness engineering or from training. An agent is a whole system. mini-swe-agent shows that a harness is little code.
  4. Context and safety: the context grows with every call and can be compacted; threat models are a malicious user, prompt injection, agent error and improper means; a sandbox, fine-grained tokens and monitoring limit harm.

Next: the strengths and weaknesses of each setting, then the homework.

Close · Strengths and weaknesses

Prompting and CoT: no training, but fragile

SettingStrengthWeakness
PromptingNo training. Works with any model you can call.The prompt's format alone changes accuracy by up to 76 points (Sclar et al., 2024).
Chain-of-thoughtLarge gains on math and symbolic tasks (Sprague et al., 2025).Little gain elsewhere, and it can hurt (Liu et al., 2024). The chain need not show the real reason (Turpin et al., 2023).

Close · Strengths and weaknesses

Reasoning models and agents: strong, not reliable

SettingStrengthWeakness
Reasoning modelImproves with more training and more thinking time (OpenAI, 2024).Few-shot prompts make DeepSeek-R1 worse (Guo et al., 2025). Reasoning models state a hint they used in 25% (Claude 3.7 Sonnet) to 39% (DeepSeek R1) of cases (Y. Chen et al., 2025).
Agent + harnessFacts and actions beyond the weights. Tool feedback can correct errors.Unreliable across runs (Yao et al., 2024). Errors add up on long tasks (Sinha et al., 2026). Prompt injection (Greshake et al., 2023).

Practice homework · no course credit

Homework: let two agents talk, then change one piece

Each agent is a model in a loop that calls tools. Here two agents talk: each reply is the other's input.

  1. Run the demo. Two agents play two roles, for example a patient and a counsellor (OpenAI Agents SDK).
  2. Predict, then change one piece. Write your prediction with a probability first. Choose one piece on the next slide.
  3. Compare. Run each version 3 times: messages, how the talk ended, role breaks, tokens.
What you produce

Predictions with probabilities, before you run

Transcripts of the base run and of your change

Counts per version, over 3 runs

One result you did not predict, explained

At most 300 words; three findings, ranked

Practice homework, no course credit. Demo code: two_agent_demo.py or the Colab notebook. Setup and model access: lecture page.

Homework · for practice

Which piece of the two-agent system to change

Choose one

The scenario: patient and counsellor, customer and support agent, interviewer and candidate, tutor and student, two neighbours

The turn limit and the stop word: when does the talk end?

The history: all earlier messages, or only the last two?

A guardrail on one agent's message

One sentence of an agent's instructions

Check before you conclude

Same scenario, settings and number of runs for every version

Is the difference larger than the spread across your 3 runs?

Did an agent leave its role? Read the transcript; the count is only a hint

A result that looks too good: read the whole conversation

Optional practice, no course credit. The patient and the counsellor are simulated: a class role-play, not medical advice.

Lecture 2, session 02 · summary

How much can prompting change?

  1. A · A brief history of prompting. From text completion to agent, the tech of prompting is more and more complicated to resolve the complicated tasks.
  2. B · Prompt to think. A reasoning trajectory cannot be trained token by token, thus prompting is the essential tool to steer the reasoning.
  3. C · Prompt as System: Agents and harness. Harness is essential when an agent becomes a complicated system with constraints, tools, failure control, etc.

Where we got to

How far can a fixed model go?

Answered today

Far, when the input and the loop give what the weights lack: examples, time to reason, tools, feedback and a check.

← → or PageUp/PageDown · F full screen · O overview · B black screen