Siddharth Sambharia

How LLMs Work, From Scratch

2026-07-12T00:00:00+00:00

When ChatGPT launched in late 2022, it opened a whole new set of possibilities for what computers can do. Fast forward to today, AI has already started contributing to the world in many ways, from writing code to speeding up scientific progress. Over the past few years AI models have kept getting better, and so has everything you can do with them.

And yet, most of us still don't know how this technology actually works.

A Large language model, is simply just a mathematical function: given some input text, it predicts the probability of what word comes next. I wanted to understand how LLMs work from scratch, not to become an expert, but to understand the basics of this technology.

In this post, I've broken down every step of how LLMs work from scratch, stripping away the complexity. By the end, you should have a simple mental model for how LLMs work fundamentally.

The roadmap:

Tokenization: text becomes integers
Embedding: integers become vectors with meaning and position
Attention: tokens read each other
MLP: each token thinks, and facts get stored
LayerNorm and residuals: plumbing that keeps it stable
The full forward pass: every piece wired together, and the vector becomes the next token
Inference: the model uses that next-token prediction to generate text, one token at a time

1. Tokenization

LLMs are mathematical functions; they don't understand words. We need a way to convert words to integers using a process called Tokenization.

Tokenization breaks down text into smaller units called tokens.
Each token is then mapped to a numerical value (Token-ID) that the model can process.

For instance: take a small dataset of 500 US names. You can map all characters in the alphabet to an integer:

Token ID	Character
0	a
1	b
2	c
…	…
25	z
26	special token

Now any name is a list of integers. SID → [19, 8, 3], JOHN → [10, 15, 8, 14].

This was a simple example, but in today's LLMs, instead of mapping each character to a token ID, models use subword tokenization techniques. Why?

Human language is fairly complex. Splitting tokens into subwords lets the model handle words it never saw in training by assembling them from pieces.
Models don't use character-to-token mapping because it makes sequences long and slow to compute.

Currently, Frontier Labs uses subword tokenization techniques like Byte Pair Encoding (BPE). BPE starts with individual characters and iteratively merges the most frequent pairs of symbols to form new tokens.

Subword tokenization: “symbiosis” splits into tokens, each mapped to a token ID

Alongside the regular tokens in the dataset, models add tokens for structure.

BOS (beginning of sequence) sits at the start, so the first real token has something before it to look at.
EOS (end of sequence) sits at the end. During inference, producing an EOS is the signal to stop. Without it, the model runs forever.

Some models use one token for both jobs. For example, a name will be represented as [BOS, S, I, D, EOS]

The model now has a predefined vocabulary of words with an integer value attached to each entry.

2. Embedding: giving tokens meaning

Token IDs are arbitrary integers; they don't have any meaning.

a = 0, b = 1…

The numbers mean nothing by themselves. We need a way to give meaning to these tokens.

Embeddings fix this.

Picture a library where each book (words) has a barcode (token-ID) stuck on it. The barcode says nothing about the book. What you actually want is a description: topic: food, difficulty: easy, language: english. An embedding is that description, written as a vector of numbers.

Embedding is a learned vector of numbers that stores information about the token, for the model to do math on it.

Word-Token-Embedding

Every model keeps an embedding matrix. Every token ID has a vector associated with it. For instance, take a small vocabulary with a 3-dimensional embedding vector; an embedding matrix for it looks like this (showing just a few tokens):

Token ID	Token	Embedding vector
0	"cat"	`[0.9, 0.1, 0.8]`
1	"kitten"	`[0.85, 0.15, 0.75]`
2	"car"	`[0.1, 0.9, 0.2]`

Tokens with similar meaning land close together in embedding space

For instance, in this example you can see:

“cat” [0.9, 0.1, 0.8] and “kitten” [0.85, 0.15, 0.75] are very close in the 3-D space. While “car” [0.1, 0.9, 0.2] is far away, even though its ID (2) is right next to cat's (0).

The embedding vector encodes meaning about the token that is learned in the training process. Maybe the first dimension captures “animal-ness,” the second “vehicle-ness.” Nobody set them by hand. The model learned them.

Because the vectors hold meaning, something surprising happens. After training, you can do meaningful arithmetic on tokens:

vec("king") - vec("man") + vec("woman") ≈ vec("queen")

Nobody coded that. It fell out of training.

This embedding matrix is called WTE (word token embedding) matrix. The dimension of the WTE matrix is (no. of token IDs, dimension of embedding), which is (50257, 768) for GPT-2.

Word-Position-Embedding

Let's take two sentences:

“The dog bit the man”
“The man bit the dog”

Both snippets have the same tokens, but the order of tokens changes the meaning completely.

The embedding vector for “man” in both cases is the same. We need a way to add the positional meaning to each vector.

The fix is a second matrix, WPE (word position embedding).

For instance, take a model with a context window (the maximum tokens an AI model can process in a single request) of 5 tokens, with a 3-dimensional position embedding vector; the WPE looks like this:

Position	Position vector
0	`[0.00, 0.10, 0.00]`
1	`[0.30, 0.20, 0.40]`
2	`[0.10, 0.40, 0.60]`
3	`[0.85, 0.12, 0.50]`
4	`[0.15, 0.45, 0.75]`

The dimension of the embedding vector is (context window × dimension of embedding vector). GPT-2's WPE is (1024, 768).

The input to the next transformer block is just the two added together:

X_i = WTE[token]  (what it is)
   + WPE[i]      (where it sits)

For “dog” sitting in position 1, the final embedding vector that gets passed to the next stage is:

X(dog) = [0.8, 0.2, 0.7]   (its WTE embedding)
       + [0.0, 0.1, 0.0]   (the position-1 vector from WPE)

Now “dog” in position 2 and “dog” in position 5 look different, and word order finally means something.

3. Attention

Now we have an input X per token that encodes its meaning and position.

Consider this example:

“She walked to the river bank to fish.”
“She called her investment bank about the loan.”

The token ‘bank’ has the exact same embedding vector, but it means completely different things in both examples. A fixed vector can't be both a riverbank and a financial one.

The model needs a way to update its vector to encode information from its surroundings. If the token ‘bank’ is near “river” and “fish,” it should lean geographic.

It's the job of the Attention Block in the Transformer where the tokens communicate with each other to update their values based on context. Figure out which earlier tokens are relevant and what to add to X(bank) to produce a refined X′(bank) that encodes the meaning of the words that came before it.

Attention is the trickiest concept to wrap your head around in this blog. Let's take the phrase “her investment bank”. Here's how Attention works in 3 parts:

Query (Q): “bank” asks, who can help me predict the next token.
Key (K): every other token before ‘bank’ in the model's context presents its offer.
Value (V): the actual information the other tokens in context hand over once picked.

Now the mechanics.

Attention introduces three learned matrices, W_q, W_k, W_v, and each token multiplies its X by them to produce three vectors.

A quick note before we dive in: an attention head is just one independent copy of this whole Q, K, V machinery. A model runs many heads in parallel, each learning to look for a different kind of relationship.

Note

A dot product is a single number that measures how much two lists of numbers agree. You multiply them position by position and add it up, so [1, 2] and [3, 1] give 3 + 2 = 5. The idea is that a bigger number means the two lists are more alike, which is how a token measures how relevant another token is to it.

Softmax is a function that converts raw scores into probabilities. It takes any list of numbers like [1, -2, 3] and turns it into a distribution like [0.12, 0.01, 0.88], where every value sits between 0 and 1 and they all sum to 1. Crucially, it preserves the relative order: the highest score stays the highest and the lowest stays the lowest, so nothing gets reshuffled, it just gets rescaled into "how much attention goes to each token."

Step 1: Project into Q, K, V.

Every token's embedding vector X(bank) gets multiplied by three learned weight matrices, W_q, W_k, W_v, each of size (embedding dimension × head dimension), e.g. in GPT-2, each head's matrices are (768 × 64):

Query = X · Wq
Key   = X · Wk
Value = X · Wv

We calculate the Q, K, V vectors for every token in the context.

Step 2: Compute attention scores: take the dot product of each Query with every Key to get a score matrix:

Q \ K	her	investment	bank
her	2	0	1
investment	0	2	1
bank	1	3	1

The highlighted row is the one we follow, the Query "bank" against every Key.

For “bank”, the raw scores are [1, 3, 1]: it aligns strongly with “investment”, weakly with “her” and with itself.

Step 3: Scale: these scores can get large, so we divide by √(d_k), the square root of the head dimension — the length of each Q/K/V vector. In this toy example each head is 3-dimensional, so d_k = 3 and √(d_k) = √3 = 1.73. This keeps gradients stable.

[1, 3, 1]  → ÷ √(d_k) = ÷√3 →  [0.57, 1.73, 0.57]

Step 4: Convert to probabilities: run the scaled scores through softmax so they sum to 1.

[0.57, 1.73, 0.57]  → softmax →  [0.2, 0.6, 0.2]

Each row shows how one query token splits its attention over the keys it can see; the “bank” row is the [0.2, 0.6, 0.2] from above

Step 5: Weighted sum of Values: multiply each token's Value vector by its attention weight and sum:

0.2 × V("her") + 0.6 × V("investment") + 0.2 × V("bank") = new context-aware vector for "bank"

The whole thing in one line:

Attention(Q, K, V) = softmax(Q · Kᵀ / √dk) · V

Before attention, “bank” was just “bank”, the same vector in a riverbank sentence or a finance one. After, it's “bank, mostly shaped by ‘investment’,” already leaning financial before the model predicts a single word. The vector wasn't replaced. It got nudged toward the meaning its neighbors imply.

Causal Masking (No peeking)

There's one flaw in Attention. A token may only attend to itself and what came before it. Otherwise it's cheating. We enforce this by setting future scores to negative infinity before softmax (e^-∞ = 0), which makes their probability zero:

Q \ K	her	investment	bank
her	0	−∞	−∞
investment	0	0	−∞
bank	1	3	1

“her” sees only itself. “investment” sees “her” and itself. “bank” sees everything before it. Every token predicts using only the past, exactly as it will when generating for real.

Multi-head attention

One head learns one kind of relationship. Language has many at once, so we run several in parallel, each with its own W_q, W_k, W_v, each hunting a different pattern.

The trick is to split the vector into equal chunks, one per head, instead of handing each head the full width. In GPT-2, for example, a 768-dimension embedding vector X splits across 12 heads, taking 64 dimensions each.

Take a 4-dimension X(bank) vector with 2 heads:

"bank": [0.9, 0.1, 0.8, 0.2]
Head 1: [0.9, 0.1]      Head 2: [0.8, 0.2]

Each head runs the five steps above on its own slice. One might lean on grammar (“bank” is a singular subject), the other on meaning (“bank” is related to finance). Say they come back with:

Head 1 → [0.67, 0.67]
Head 2 → [0.16, 0.58]

Glue them back together, up to full width again (4 here):

[0.67, 0.67, 0.16, 0.58]

One last matrix, W_o, mixes the glued heads together to get the final result.

[0.67, 0.67, 0.16, 0.58] × Wo = [0.8, 1.1, 0.4, 0.9]   ← final attention output for "bank"

4. Feed Forward Network (MLP)

The next block is the MLP (or feed-forward network), this is the part where LLMs store facts. In the previous step, the token ‘bank’ gathered context from the surrounding tokens, but it still hasn't reasoned through it.

Knowing “river is near” and “bank is near” isn't the same as concluding “this is geography.” To reason over the fact that river + bank = riverbank, models need a thinking layer, and that's where MLPs help.

MLP stands for multilayer perceptron. In this layer, each token's embedding vector goes through a series of operations where it expands to a larger dimension first and then compresses back to its initial dimension.

Unlike the previous step where each token ‘attends’ to one another, in this step each token thinks on its own. No token talks to another here; they all run the same operation in parallel.

Here's the working:

Expand (W₁). Blow the vector up into a bigger space (GPT-2 goes 768 → 3072) by multiplying it with the W₁ matrix.
Apply a nonlinearity like GeLU, ReLU, etc.
The simplest nonlinear function example is ReLU, which just zeroes out negatives and passes positives through: [0.8, 1.4, -0.2, 2.1] (ReLU) → [0.8, 1.4, 0, 2.1]
Compress (W₂). Multiply by the W₂ matrix to shrink back to the original size.

Mathematically:

MLP(x) = W2 · GELU(W1x + b1) + b2

W₁ expands the vector, the nonlinearity switches neurons on/off, W₂ compresses back

A nice way to read it: each row of W₁ asks one question (“is this river + bank?”). The answer after the nonlinearity is a neuron: positive for yes, zero for no. Each column of W₂ is the response to add back when that neuron fires. When “riverbank” lights up, W₂ stamps “geography, water, landscape” onto the token.

This is also where facts live. Most of a model's parameters sit in these layers, not in attention, and they aren't generic bookkeeping. Researchers have found single neurons that fire on the Eiffel Tower, or on past-tense verbs, or on a specific programming language. When a model “knows” Paris is the capital of France, that fact is spread across FFN weights in particular layers.

5. LayerNorm and Residuals

Two pieces of plumbing keep a deep stack trainable.

LayerNorm. After a few blocks, a token's numbers drift. Some hit 500, others 0.001. Softmax hates that ([500, 501, 499] collapses to nearly [0, 1, 0]), and training goes unstable. LayerNorm rescales each token's vector to mean 0 and spread 1, keeping the pattern but taming the scale. [10, 2, 6, 14] becomes [0.45, -1.34, -0.45, 1.34] Notice how the relative pattern is preserved but the scale is tamed.

Residuals. Each block adds its output back onto its input instead of replacing it:

new = old + block(old)

Without this, a later block can wipe out what an earlier one figured out, and “bank” could forget it was ever “bank.” With it, the original rides along and each block only has to add a small nudge. A block with nothing useful to contribute can output roughly 0 and get skipped for free.

For instance, “cat”'s vector is [0.8, 0.3] and attention produces the update [0.0, 1.1]:

Without residual: new vector = [0.0, 1.1], the 0.8 is gone forever, unrecoverable by later blocks.
With residual: new vector = [0.8, 0.3] + [0.0, 1.1] = [0.8, 1.4], the original survives, new context layered on top.

Residuals also save training. A deep stack has to send a correction signal backward from the last block to the first, and without a shortcut that signal fades to nothing on the way. This is the vanishing gradient problem. The residual is a clean path straight through, so early layers still learn. More on that in part 2.

LayerNorm and Residuals are added after each block to keep the model trainable.

6. The Full Forward Pass

We've built every piece on its own. Now let's wire them together and watch a single token make the whole trip.

One forward pass is just text going in one end and a prediction coming out the other. Tokenize the text, look up each token's meaning and position, then push the stack through the transformer blocks: attention lets the tokens read each other, the MLP reasons over what they gathered, and LayerNorm and residuals keep the whole thing stable. That block repeats over and over (12 times in GPT-2), each pass layering a little more context onto every token.

The full path from text to the next token, every stage of this post in one picture

We've followed a token all the way from tokenization through the transformer blocks. Out the far end comes a refined vector X′, loaded with all the context it gathered along the way. But X′ is still just a list of numbers. We need to turn it back into words and predict the next token, the job we started with. Three steps do it:

A final LayerNorm. One last rescale to tidy up the vector. We're still left with a 768-dimensional embedding vector (in GPT-2), the same shape we started with back in the embedding step.
The language-model head (LM head). To get from a vector back to words, we do the reverse of embedding: embedding turned a token ID into a vector, now we turn a vector into a score for every token in the vocabulary. We don't even need a brand-new matrix, the head is usually the embedding matrix (WTE) transposed, a (768 × vocab) matrix reused in reverse. Multiplying X′ by it gives one raw score per token. Those scores are the logits.
Softmax. Turn the logits into probabilities that sum to 1. That's the probability table from the top of the post.

Back to our sentence, “She called her investment ___”. The LM head turns X′ into one logit per token in the vocabulary, and softmax turns those into probabilities:

Token	Logit	Probability
bank	3.4	0.61
firm	2.4	0.23
advisor	1.6	0.10
account	0.5	0.03
loan	-0.2	0.02
broker	-0.4	0.01

The model is most confident the next word is bank. One logit per token, squashed into one probability per token, and the highest one is the model's best guess. (The names model works the same way, just with vocab = 27, one score for each letter a to z plus the end token.)

Here's the whole forward pass in code:

def gpt(token_ids, positions):
    x = wte[token_ids] + wpe[positions]   # embed: what + where

    for block in blocks:                  # 12 blocks in GPT-2
        x = x + attn(layer_norm(x))       # gather context (+ residual)
        x = x + mlp(layer_norm(x))        # reason over it (+ residual)

    x = layer_norm_final(x)
    return lm_head(x)                     # logits over the vocabulary

That's the forward pass. Embed the tokens, alternate attention and MLP a dozen times, project back to the vocabulary. Every section above is one line here.

Inference

The model scores every position at once, but when generating we only care about the last one: the prediction for the next token.

Generation is a loop. Predict a token, stick it on the end, run again.

"She called her investment" → "bank"
"She called her investment bank" → "about"
...

How you pick from the distribution is a choice:

Greedy: always take the top token. Deterministic, often dull.
Temperature: divide the logits by T before softmax. Below 1 sharpens the odds (safer), above 1 flattens them (wilder).
Top-k / top-p: sample only from the top k tokens, or from the smallest set of tokens covering probability p.

The inference loop: run the forward pass, take the predicted token, stop on EOS, otherwise append it and run again

You stop when the model emits EOS, or when you hit a length cap. The names model samples letters (s, a, m) until it emits the end token, and you've got a fresh name.

What Next?

It took decades of breakthroughs in computer science to finally build AI, and there's still so much left to do. What we covered here is just the foundation, there's a lot more to how LLMs work, at scale and in optimization, that I didn't get into. But hopefully these basics open up the vast ocean of things you can now go learn from.

The best next step is to see all of this in code. Go read microGPT, the whole thing running in a couple hundred lines of Python.

If you found this helpful, reach out on X. I'd love your feedback, and I love making new friends.

This post was inspired by Karpathy's microGPT and blog, Lee Robinson's Understanding AI, and 0xkato's How LLMs Actually Work.

Why You Should Learn How AI Actually Works

2026-07-08T00:00:00+00:00

AI is a fascinating technology. I still remember the first time I tried the Davinci model in 2020 from OpenAI, and I was flabbergasted by how it could artificially write blocks of meaningful text.

Since then a lot has changed. We had ChatGPT in 2022, GPT-4 in 2023, Claude Code in 2025, OpenClaw in 2026, to now whatever is the current track of AI Engineer.

I was born in 2003, by the time I got to university all the biggest technological revolutions of the last decade – the internet, the iPhone, cloud were past their inflection point. But AI is different, it’s the only General-purpose technologies that I’ve witnessed from the launch of ChatGPT to becoming fastest growing consumer technology ever created.

Broken Fundamentals

I am much more AI-pilled than 99% of people out there and been part of this ecosystem since its infancy (building a GPT wrapper in 23, AI gateway 24-26). But I was so busy with the next ‘shiny’ thing in AI that I never took out time to learn the fundamentals until now.

The problem with this approach chasing the next shiny thing is that you are building a skyscraper with a broken foundation.

Why Learn The Basics of AI?

Fair question. For most technology you can simply use it without understanding how it works. I don’t need to understand how airplanes work, or how their microwave heats food, and the same goes for 90% of all the consumer technology out there.

Here’s why I believe understanding AI matters:

There’s so much junk out there with AI; for you to decide what’s useful you need the taste that can be developed once you understand the basics of this technology.
If you are a builder, a good way you develop new ideas is by being at the frontier of technology, understand what it’s capable of doing. Being on the frontier allows you to see the blindspots and build tools to solve them. In return capture value to build wealth. Unless you know how LLMs work, you won’t be able to understand what you can do with them, to mine gold or sell shovels.
AI is a force multiplier. People who fully embrace a new technology aren’t 5x more productive than the old way of working — they’re 100x. But here’s the paradox: AI, like any technology, isn’t perfect. It makes errors. And the only way to reliably spot those errors is to understand the fundamentals. Do you have an intuition for why image models used to mangle human hands, and how frontier labs fixed it? Or why models struggle to count the R’s in “strawberry”? The basics are what let you answer questions like these.
The learning cost is lowest early. It’s easier to grow along with new technology when it’s early. like reading Darwin’s “On the Origin of Species” is better to understand evolution than reading CRISPR. Paying that learning cost while the AI is still young lets you hill climb faster.

What’s Next

This is an exciting time to build. Like every technology wave before it, this is my attempt to document what I’m learning in the open — sharing notes, and along the way meeting new people, running into new ideas, and earning some distribution.

I’ll post once a week: fundamentals, what’s new, and whatever I can’t stop thinking about. Next up: how LLMs work at a fundamental level.

A Brief History of AI Agents

2026-04-25T00:00:00+00:00

Agents

From chat completions API to agent harnesses and everything in between.

I work at @PortkeyAI , where my job is to play with the latest AI tools and figure out how they fit into our gateway. Since ChatGPT launched in late 2022, I’ve watched every abstraction people call an “agent” become outdated within months as models got stronger. Once in a while it’s worth stepping back to see how we actually got here.

Timeline (TL;DR)

Part 1 the cognitive revolution - Chat Completions
Part 2 models get hands - Function Calling
Part 3 models discover autonomy - Agent Loop
Part 4 models get a body - Harness
Part 5 the outer loop - Ralph Loops

every AI product today is a language model wrapped in scaffolding. the model is a line in a config file. obvious now. wasn’t obvious in 2022.

this is five timeline on how we started from chat completions API all the way upto claude code. each part adds one capability the model couldn’t do the part before.

(the pre-LLM history of software orchestration, DAG schedulers, ML pipelines, is covered well in @dexhorthy’s 12-factor agents. this post picks up where language models enter the picture.)

part 1 — the cognitive revolution (2020 – early 2023)

💡the API — chat completions

for most of its history, a language model was a research artifact. you could read the paper, maybe download the weights if you had the right GPU, run it in a notebook. but you couldn’t call it. it lived behind lab doors.

that changed on june 11, 2020, when OpenAI released an API for GPT-3. the announcement was a single paragraph that now reads like a founding stone:

We’re releasing an API for accessing new AI models developed by OpenAI. Unlike most AI systems which are designed for one use-case, the API today provides a general-purpose “text in, text out” interface, allowing users to try it on virtually any English language task.

“text in, text out.” you sent a prompt, you got a completion back. no memory between calls. no ability to act on the world. one forward pass and done.

the only lever was the prompt itself, and an entire discipline, prompt engineering, grew up around this single knob. few-shot examples, chain-of-thought, role-playing instructions — all attempts to squeeze more capability out of a system that could only read text and write text.

this primitive, the single completion call, never goes away. every part that follows wraps more machinery around it.

💡chaining calls

one call wasn’t always enough. ask a model to write a blog post in a single prompt and you’d get something unfocused, or wrong in a way that was hard to fix without starting over.

the fix was obvious: break the task into steps. first call generates a plan. second call implements it. third call verifies the output. wire them in order, branch on failure.

harrison chase packaged this in october 2022 with LangChain. ChatGPT launched a month later, millions of developers wanted to build with language models, and LangChain was the fastest on-ramp.

the developer decides the control flow. the model fills the nodes.

this is all workflow territory. the first real takeaway: if your task breaks down into steps you can define in advance, a workflow beats an agent almost every time.

Part 2 — models get hands (mid 2023)

💡function calling

everything so far had a limitation so fundamental it was easy to miss: the model could only talk. it couldn’t check a database, call an API, run a calculation, or touch the outside world.

on june 13, 2023, OpenAI shipped function calling. Anthropic followed with tool use in early 2024. alongside your prompt, you send a list of tools described as JSON schemas. instead of answering in plain text, the model can reply with a structured call like get_weather(city=”mumbai”). your code runs the function, and the result goes back to the model.

thorsten ball has a good way of explaining what this actually is. imagine telling a friend: “in our conversation, wink if you want me to raise my arm.” weird instruction, but easy to follow. function calling is the same idea, just formalized. you tell the model what tools exist. when it wants to use one, it winks. you execute. you reply with the result.

under the hood, it’s still just text — the model generates a JSON blob instead of a sentence. but conceptually, the model was no longer sealed off. any capability you could wrap in a function became something it could reach for.

most early tool use was single-shot: one function, one result, one response. the model had hands. someone else was still guiding them.

source: OpenAI — function calling and other API updates

Part 3 — models discover autonomy (late 2023 – 2024)

💡the agent loop

up to this point, a language model was a tool. you called it, it answered, you decided what to ask next.

the agentic loop inverted this. instead of you deciding the next step, the model decides. call a tool, get the result, feed it back, ask “what now?” repeat until the model says it’s done.

fifteen lines. ReAct (yao et al., 2022) formalized the pattern. thorsten ball’s how to build an agent proved a working coding agent is about 400 lines once you strip the harness away.

the shift was in who held the steering wheel. before the loop, you drove. after it, the model drove. you built the car, handed over the keys, and told it where to go.

anthropic’s building effective agents guide drew the line: workflows are systems where the developer orchestrates the control flow. agents are systems where the model directs its own process. the loop is where agents begin. and with that shift came a new failure mode: workflows fail like factories, at a known step you can fix. agents fail like people — they get confused, double down, and you have to trace where the reasoning went sideways.

💡the org chart fallacy

once the loop worked, the next idea felt natural: if one agent is good, a team should be better. assign roles — planner, researcher, writer, critic — and wire them like an org chart.

AutoGPT and BabyAGI both launched in march 2023, days after GPT-4. AutoGPT hit the top of GitHub overnight. a whole framework category followed within months: AutoGen, CrewAI, LangGraph, OpenAI Swarm. it felt like the future was a squad of specialized agents collaborating like a high-performing team.

In 1975, brooks’ the Mythical Man-Month observed that adding people to a late project makes it later — not because the people are bad, but because coordination has a cost. multi-agent systems hit the same wall. handoffs lost information. debugging meant figuring out which agent, on which turn, made the wrong call.

teams shipped multi-agent, then quietly went back to one agent with better tools. the frameworks survived for their plumbing: state machines, checkpointing, observability. the idea wasn’t wrong. the shape was. the version that actually works is one boss with disposable interns, not a committee of equals.

💡the wall

even a well-built single agent starts losing coherence after ten to fifteen turns. it forgets what it tried, repeats itself, wanders. each turn is maybe 90% reliable; over fifteen turns, your odds of a clean run are about 20%.

longer context windows helped, but didn’t solve it. more autonomy wasn’t the answer. more structure was.

Part 4 — models get a body (2024 – 2026)

💡the harness

the wall created a design question. the loop worked. tools worked. but pure autonomy degraded fast. what was missing wasn’t intelligence. it was structure.

the answer was the harness: every piece of code, configuration, and execution logic that isn’t the model itself. system prompt, toolset, loop logic, context management, permissions, feedback mechanisms. if you’re not the model, you’re the harness.

source: @Vtrivedy10

the first harnesses that mattered were developer tools. @cursor_ai , @github Copilot Chat, @cognition’s Devin. they shared a pattern: narrow domain, opinionated tools, a UX that made the agent’s actions visible to a human.(code was the natural first domain — it’s verifiable and recursively self-improving, what @swyx calls the pareto principle of AGI.)

Claude Code (Anthropic, 2025) became the canonical example. @trq212 shihipar from the team wrote “seeing like an agent”: you design tools shaped to the model’s abilities, and you learn what those abilities are by watching it work. the tools kept evolving because the assumptions they encoded kept going stale.

sub-agents follow the same logic. the org chart fallacy failed because it used a committee. Claude Code uses a boss with interns: spawn a sub-instance with a clean context and a narrow question, get a summary back, never see the noise. once Claude Code proved the pattern, open-source alternatives followed: OpenCode, Aider, Pi. the harness is a known shape now. what differs is taste.

models today are post-trained with their harness in the loop. @swyx paraphrased McLuhan: “first the model shapes the harness, then the harness shapes the model.” CompileBench and TerminalBench show the same model scoring very differently across harnesses. every harness component encodes an assumption about what the model can’t do, and those assumptions go stale fast.

💡the open edge

Claude Code shipped with a fixed toolset. three things opened the edge.

MCP (Model Context Protocol, november 2024) standardized how agents connect to external tools. client-server over JSON-RPC: implement once on each side and they talk. OpenAI and Google adopted it; by december 2025 Anthropic donated it to the Linux Foundation.
Skills (october 2025) are the other half. MCP gives the agent new tools. a skill gives it new knowledge: a SKILL.md that teaches a workflow at runtime. the agent loads context when relevant instead of stuffing everything into the window at startup.
Computer use (october 2024) redefined what a tool can be. screenshot, coordinates, click, type. any GUI becomes a tool surface. slower than a native API, but universally applicable.

the pattern: a stable harness core and an open edge where capabilities plug in from outside. MCP for tools, skills for workflows, computer use for everything else.

Part 5 — the outer loop (2025 and beyond)

the agent loop let the model decide what to do next. the harness wrapped that loop in structure. the outer loop does the obvious thing: takes the entire harness and puts it back inside a loop.

@GeoffreyHuntley’s ralph wiggum technique is the simplest version. named after the Simpsons character because it is persistently optimistic and surprisingly effective, ralph is a bash loop:

same prompt every iteration. fresh context every iteration. state doesn’t live in the conversation. it lives in the codebase: git commits, test results, a fix_plan.md, an AGENT.md the model updates with what it learns. each iteration reads the current state of the world, picks one thing to do, does it, commits, and exits. the loop restarts with a clean window and a slightly different filesystem.

huntley ran this for three months and built Cursed Lang, a complete programming language with a self-hosting compiler, for a language that didn’t exist in the model’s training data. dex at HumanLayer tested it for refactoring, leaving it overnight against a coding standards doc. his takeaway: code is cheap. the prompt is what matters.

the same pattern works beyond code. @karpathy’s autoresearch puts an agent in a loop against a training run: propose a change, evaluate against validation loss, plan the next experiment, repeat. he ran it for two days against a project he’d already hand-tuned extensively. the agent found twenty real improvements he’d missed.

deep research products from Anthropic, OpenAI, and Google are the polished version. ralph builds code. autoresearch tunes models. deep research writes reports. different domains, same primitive: a loop that calls a harness, evaluates the result, and decides whether to keep going.

every layer compensated for a weakness. as models got better, some of it got absorbed. the prompt hacks from 2022 are unnecessary now. the multi-agent frameworks from 2023 already look ornate. the 2025 harness probably will too by 2028.

if you’re building today: start with a workflow. reach for the agentic loop only when you need runtime decisions. keep scope narrow. don’t add more agents — use the sub-agent pattern. build tools the model can actually use. measure everything. and when the next model ships, remove the outdated patterns.

LLM Pricing Is 100x Harder Than You Think

2026-04-15T00:00:00+00:00

This article was first published on portkey.ai/blog.

LLM pricing is surprisingly difficult. We've tracked $180M in LLM spend across 3,500+ models. Here are the 6 hidden patterns that break cost attribution, the architecture we built to solve them, and we're open-sourcing all of it.

Earlier this month, we open-sourced Portkey’s model pricing database — 3,500+ models across 50+ providers. The same data we use to attribute cost for enterprises processing trillions of tokens through our gateway every day.

Turns out, a lot of teams needed this.

The entire industry is focused on harness design, managed agents, model benchmark scores. Meanwhile, there’s no common ground on something more fundamental: How do you actually attribute cost to model usage?

Think about it. Most projects maintain an in-house pricing database. A JSON file somewhere in your repo with model names and prices. OpenCode has one of these. So does OpenClaw, LibreChat, Pi, Theo’s T3 code. I keep finding new ones.

The pattern is clear: everyone builds their own thing, it’s accurate for a few weeks, then it drifts. There’s no canonical source. No API you can just call. No dataset comprehensive enough to handle the weird edge cases.

Three years post-ChatGPT, there’s still no standard way to calculate what a single request costs across providers.

At Portkey we’ve spent three years building the infrastructure for this. Here’s everything we’ve learned, and we’re releasing the full stack so you don’t have to rebuild it yourself:

Model Data + Free API: Updated daily at portkey.ai/models
Portkey’s Gateway: Open-source AI gateway with built-in pricing engine

The 6 patterns that break pricing

These aren’t edge cases. Every one of them has caused real cost discrepancies for teams using the models.

1. Thinking tokens

Reasoning models like o3 and Claude with extended thinking consume tokens for internal reasoning that never appear in the response. You still get charged for them.

OpenAI’s o1-preview has a 4× output-to-input price ratio ($15/M input, $60/M output). Most of that gap is reasoning overhead. If your system only counts visible output tokens, you’ll undercount agentic workloads by 30–40%.

2. Cache asymmetry

Prompt caching economics are different per provider in ways that matter.

Anthropic charges 25% more for cache writes ($3.75/M vs $3.00/M regular input), with reads at $0.30/M. OpenAI charges nothing for writes. Reads get discounted. If you apply a single “cache discount” multiplier across both, your numbers are wrong for at least one of them.

3. Context thresholds

OpenAI, Anthropic, and Google all have tiered pricing based on context length. Cross 128K tokens and per-token cost can double. $0.075/M becomes $0.15/M. Nothing in the API response tells you which tier you hit. The request just works. Your cost estimate is silently wrong.

4. Same model, different prices

Kimi K2.5 costs $0.5 input / $2.8 output on Together AI, $0.6 input / $3 output on Fireworks. You can’t just track “Kimi K2.5.” You need “Kimi K2.5 on Together AI.”

And it gets worse: Bedrock prepends regional prefixes (us.meta.llama, eu.anthropic.claude-...) that need stripping before you can even look up the price. Azure returns deployment names instead of model identifiers. You need an extra API call to figure out what model you’re running.

5. Non-token billing

DALL·E 3 bills by image quality and resolution. Video generation charges per second. Realtime audio has separate input/output rates. Embeddings are input-only. Fine-tuning is per-token on some models, per-hour on others. Each needs different fields from the request and maps to a completely different pricing structure.

6. New dimensions keep appearing

We started with two billing dimensions: input tokens and output tokens. Now there are over twenty. Web search has per-search pricing. Google’s Grounding with Search has its own rate structure. Tool use, code execution — each ships with its own cost model, and new ones appear faster than providers update their documentation.

Why it matters

Every enterprise wants to adopt AI. Making it actually work is another story. The moment you move past prototypes, cost attribution becomes a dealbreaker:

FinOps goes blind. Teams running hundreds of model variants across departments need per-team, per-user cost breakdowns. When pricing is wrong, the AI budget becomes a single line item nobody can decompose or optimize.
Margins become guesswork. If you’re reselling LLM access — and increasingly everyone is — inaccurate cost data means you’re either leaking money or overcharging customers.
Budgets can’t be enforced. You can set per-team spending limits, but limits only work if cost data is accurate. A model reporting $0 per request will never trip an alert.
Shipping slows down. Teams get blocked on AI features because nobody can answer “what will this cost at scale?”

How Portkey’s gateway handles this

The gateway normalizes every provider response into a single cost structure. Model identifiers get resolved, usage gets normalized, and cost gets tagged per-team and per-user at the routing layer, before it hits your logs.

Under the hood, the architecture separates three things that change at different rates:

Provider Response → Unified Gateway → Pricing Data → Pricing Logic → Cost

When a provider changes their response format, we update the extraction. When rates change, we update config. When new dimensions appear, we extend the schema. Each layer changes independently.

The normalization is where most of the complexity lives. Every provider returns usage data differently. OpenAI gives you prompt_tokens and completion_tokens. Anthropic gives you input_tokens and output_tokens. Bedrock prepends regional prefixes. Azure returns deployment names. We normalize everything into one structure:

The additionalUnits map is what lets us handle new billing dimensions (web search, grounding, tool use) without schema changes. They just become new keys.

Because the gateway sits at the routing layer, it sees every request before it hits your logs. Model identifiers get resolved, usage gets normalized, cost gets tagged. That’s what makes per-team and per-user budget limits possible. Cost attribution happens at the source rather than being reconstructed after the fact from incomplete data.

It’s worth noting that Stripe recently launched their own AI Gateway specifically for LLM token billing, routing requests through a layer that meters usage per customer, per model, per token type. Same core insight: the centralized proxy is the natural place to solve cost attribution.

How we keep 3,500+ models accurate

Building the system was the easier part. Keeping it updated across 3,500+ models is the real challenge. Models launch weekly. Pricing changes without changelog entries. Context thresholds get buried in documentation footnotes. No human team can keep up manually.

We built an agent for this using the Claude Agent SDK, with tools for fetching model lists from provider APIs, web scraping, and GitHub integration.

The key design decision: provider-specific logic lives in skill files, not code. A skill file is a markdown document describing how to handle a specific provider — where to find model lists, how to scrape pricing, what quirks to watch for. When Anthropic changes something, we update the skill file. Not the agent. Not the codebase.

The agent loads skill files, fetches model lists, scrapes pricing sources, formats everything to schema, and opens PRs with citations. It costs about $2–3 per provider run. Novel pricing structures still confuse it. Humans handle judgment calls. But it covers the tedious work that was eating up our time.

Pricing complexity isn’t slowing down

New models, new billing dimensions, new provider quirks. If your cost dashboards don’t match your invoices, this is probably why.

We’re releasing everything so you don’t have to rebuild it from scratch:

Model Data + Free API: Updated daily at portkey.ai/models
Portkey’s Gateway: Open-source AI gateway with built-in pricing engine

The Harness Tax: The Dead Weight Inside Your Coding Agent

2026-04-13T00:00:00+00:00

Agents

Claude Code used 83k tokens to write a Fibonacci script. Pi (OpenClaw) used 8k. Same task. Same output. Where did the rest of the tokens go? Here's the tax no one's talking about.

Harnesses are not going away. Even the best models rely on them. Claude Code alone has ~512k lines of harness code. But nobody talks about what that harness actually costs you at inference time.

I wanted to know: when using coding agents, how much of the payload that hits the model is actually my message? And how much is the harness overhead added?

So I pointed three agents at Portkey’s gateway and captured every request — Pi (the harness behind OpenClaw), OpenAI Codex, and Claude Code. Same request and complete token visibility. Then I gave each one the same two messages:

Message 1: hey

Message 2: write a simple python script to check fibonacci series and save on desktop as agent.py

Pi sent ~2,600 input tokens. Claude Code sent ~27,000. A 10× spread. Same task. Same model capability. The difference was pure harness overhead.

The Harness Tax

The Harness Tax is every token your agent spends on itself before it spends a single token on your task.

You pay this tax before the model does a single unit of useful work. Every agent has one. You never see it unless you look at raw request logs. I routed all three agents through a gateway to get that visibility.

What Goes Into the Harness Tax?

Every request a coding agent makes to the model carries the full harness payload: tool definitions, system prompt, memory instructions, behavioral routing, and conversation history. All of it. On every turn.

Claude Code’s harness costs roughly 27,000 input tokens per request. Codex costs about 15,000. Pi costs about 2,600.

And because the conversation history includes the model’s previous responses — which were themselves inflated by verbose tool-call formatting — the payload grows faster than your actual conversation does.

A real coding session runs 30 to 50 turns. At Claude Code’s rate, a 40-turn session burns through 1.12 million input tokens. Roughly half of those are harness overhead.

You pay the harness tax whether you use the tools or not. The 24 extra tools in Claude Code were defined but never called. Their definitions shipped on every request anyway.

Context Rot

The harness tax isn’t just a cost problem. It’s an attention problem. Every extra token competes with your actual task: your code, your files, your intent.

As the context window fills, the model gets worse at reasoning over the tokens that matter. Every token the harness adds competes for attention against your code, your files, and your actual task. On a complex refactor where the model needs to hold three source files, a test suite, and twenty turns of conversation, 28,000 tokens of framework plumbing aren’t sitting idle. They’re noise.

A 200k context window carrying 28k tokens of harness overhead isn't a 200k window. It's a 172k window with worse attention distribution.

The harness rots in a second way: staleness. Every component encodes an assumption about what the model can’t do on its own. Those assumptions go stale fast. More on that below.

Thin Harness, Fat skills

Pi gives the model four capabilities: read, write, edit a file, and run a shell command. That’s the entire tool surface.

The bet is that a model trained on millions of shell sessions, the internet, and GitHub repos already knows how to compose those primitives into anything else. You don’t need a dedicated list_directory tool when ls -la exists. You don’t need search_files when the model can write grep -r on its own.

“All frontier models have been RL-trained up the wazoo. They inherently understand what a coding agent is.” — Mario Zechner, Pi’s creator

Anthropic’s harness engineering team demonstrated this concretely over three model generations. Their coding agent harness for Sonnet 4.5 required context resets because the model would start wrapping up work prematurely as the window filled. Opus 4.5 shipped — resets became unnecessary. Opus 4.6 shipped — they stripped out sprint decomposition entirely, and it still worked better.

Three model generations. Three layers of harness removed. Load-bearing in January, dead weight by March.

Harnesses encode assumptions that go stale as models improve — Anthropic Engineering

An agent has three layers. Complexity should push up into the model, which gets better at reasoning, planning, and self-correction with every release. It should push down into infrastructure, where routing, governance, observability, and cost controls don’t ride along in the context window. The harness in the middle should carry as little as possible.

What This Means

This was a narrow benchmark. Two messages, one trivial task. Claude Code’s deep tooling may earn back its overhead on complex work that genuinely exercises those 28 tools.

What this benchmark does show: the overhead exists, it’s measurable, and almost nobody is looking at it. For most tasks, the model is carrying 15,000 tokens of framework plumbing it doesn’t need. And that overhead is growing slower than models are improving, which means the tax gets harder to justify.

Route your agent through Portkey to measure your own harness tax.

Further reading:

Mario Zechner’s blog post on building Pi — the design rationale
Armin Ronacher: “Pi: The Minimal Agent Within OpenClaw”
Pi on GitHub · OpenClaw

This article was first published on X.