So this post is me going one level down. For each of the big architectural tricks of the last few years, I want to answer the same two questions: what is it, and which model actually shipped it? That second question matters more than it sounds, because for the closed models (OpenAI, Anthropic) we mostly can’t know — the open-weight releases are where the architecture is legible.

My learning and this writing based on Build a Large Language Model (From Scratch). If my writing lits your interest, I highly recommend reading this book!

GQA — Llama 2/3, Qwen3, Gemma 3

Grouped-Query Attention is the boring default now. If you pick an open-weight model at random today, it’s probably using GQA as a more compute- and parameter-efficient drop-in for Multi-Head Attention (MHA). It’s not new either — it goes back to the 2023 paper GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, and even the larger variants in the good old Llama 2 series used it.

Sharing keys and values reduces the total number of key and value computations, which leads to lower memory usage and improved efficiency.

So, to summarize, the core idea behind GQA is to reduce the number of key and value heads by sharing them across multiple query heads. This (1) lowers the model’s parameter count and (2) reduces the memory bandwidth usage for key and value tensors during inference, since fewer keys and values need to be stored and retrieved from the KV cache.

While GQA is mainly a computational-efficiency workaround for MHA, ablation studies (such as those in the original GQA paper and the Llama 2 paper) show it performs comparably to standard MHA in terms of LLM modeling performance.

However, this assumes the number of key-value groups is chosen carefully. In the extreme case where all attention heads share a single key-value group — known as multi-query attention — memory usage drops even more drastically, but modeling performance can suffer. And on the other extreme, if we set the number of key-value groups equal to the number of query heads, we’re back at standard multi-head attention.

My take: GQA is the one on this list I never notice as a user, which is exactly the point — it’s a free lunch that everyone quietly took.

MLA — DeepSeek V2, V3, R1

Multi-Head Latent Attention, used in DeepSeek V2, V3, and R1, offers a different memory-saving strategy that also pairs particularly well with KV caching. Instead of sharing key and value heads like GQA, MLA compresses the key and value tensors into a lower-dimensional space before storing them in the KV cache.

At inference time, these compressed tensors are projected back to their original size before being used, as shown in the figure below. This adds an extra matrix multiplication but reduces memory usage.

(As a side note, the queries are also compressed, but only during training, not inference.)

As shown in the figure above, GQA appears to perform worse than MHA, whereas MLA offers better modeling performance than MHA — which is likely why the DeepSeek team chose MLA over GQA. (It would have been interesting to see the “KV Cache per Token” savings comparison between MLA and GQA as well.)

So: MLA is a clever trick to reduce KV cache memory use while even slightly outperforming MHA in terms of modeling performance. Not a tradeoff — a win on both axes.

My take: this is the piece that made DeepSeek’s pricing make sense to me. When a provider is an order of magnitude cheaper per token on long inputs..

SWA — Gemma 2, Gemma 3

What is sliding window attention (SWA)? If we think of regular self-attention as a global mechanism, since each sequence element can access every other sequence element, then SWA is local attention: we restrict the context size around the current query position. This is illustrated in the figure below.

Instead of attending to all previous tokens, each token only attends to a fixed-size local window around its position. This localized attention lowers the size of the KV cache substantially.

Sliding window attention was originally introduced in the LongFormer paper in 2020, but the reason to focus on Google’s Gemma models is that they’re very good open-weight models showing that SWA is a genuinely feasible approach in recent, capable models — not just a research curiosity.

Gemma 2 used a hybrid approach that combined local (sliding window) and global attention layers in a 1:1 ratio, with each token able to attend to a context window of 4k tokens. The reason for the 1:1 split is that it strikes a balance between efficiency and global context modeling — an LLM using only local attention can be too restrictive.

Gemma 3 then pushed further toward efficiency: a 5:1 ratio between sliding window and full attention layers, meaning for every five local attention layers there’s one global layer. The window itself also shrank, from 4096 tokens in Gemma 2 to 1024 in Gemma 3.

My take: One global layer holding the line for five local ones suggests most attention work really is local, and that most of the quadratic cost we pay is insurance.

MoE — DeepSeek-V3, Qwen3-MoE

Mixture-of-Experts is the odd one out here: it isn’t an attention variant at all, it’s a feed-forward variant. The core idea is to replace each feed-forward module in a transformer block with multiple expert layers, where each expert is itself a feed-forward module. So one feed-forward block becomes many, as illustrated below.

When DeepSeek rattled the stock market in early 2025, a lot of the media described MoE as a panel of specialist AIs conferring with each other — a coding AI, a medical AI, a math AI, voting on an answer. Look at the figure again: an “expert” is a feed-forward network. That’s it. The router is a small learned gate that picks which of those blocks to run for this one token, and whatever specialization exists is whatever training nudged into them — nobody assigned job titles.

Because only a few experts are active at a time, MoE modules are often called sparse, in contrast to dense modules that always use the full parameter set. The large total parameter count increases the capacity of the LLM — it can absorb more knowledge during training — while the sparsity keeps inference efficient, since we don’t use all the parameters at once.

For example, DeepSeek-V3 has 256 experts per MoE module and 671 billion parameters in total. Yet during inference, only 9 experts are active at a time (1 shared expert plus 8 selected by the router). That’s just 37 billion parameters per token inference step instead of all 671 billion.

One notable feature of DeepSeek-V3’s MoE design is the shared expert: an expert that is always active for every token. The idea isn’t new — it was already introduced in the 2022 DeepSpeed-MoE and 2024 DeepSeek MoE papers.

(An annotated figure from the DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models paper.)

The benefit of a shared expert was first noted in the DeepSpeed-MoE paper, where they found it boosts overall modeling performance compared to having no shared expert. The likely reason: common or repeated patterns don’t have to be relearned by every individual expert, which leaves each of them more room for specialized patterns.

My take: What MoE really breaks is the assumption that capacity and compute are the same quantity. the shared expert is the honest admission that some of it isn’t.

Gated DeltaNet — Qwen3-Next, Kimi Linear

Recently, Qwen3-Next and Kimi Linear proposed hybrid transformers that swap in alternatives to the attention mechanism which scale linearly instead of quadratically with context length.

Both use a 3:1 ratio: for every three transformer blocks employing the linear Gated DeltaNet variant, there’s one block using full attention, as shown in the figure below.

So what is Gated DeltaNet? It’s short for Gated Delta Network, Qwen3-Next’s linear-attention layer, intended as an alternative to standard softmax attention. It was adopted from the Gated Delta Networks: Improving Mamba2 with Delta Rule paper.

Gated DeltaNet was originally proposed as an improved version of Mamba2, combining Mamba2’s gated decay mechanism with a delta rule. Mamba itself is a state-space model — an alternative to transformers, and a big enough topic that it deserves its own post someday.

The “delta rule” part refers to computing the difference (delta, Δ) between new and predicted values to update a hidden state that’s used as a memory state.

My take: I’ve been following Mamba for years as the candidate to replace the transformer, and for most of that time state-space models were fascinating but never quite competitive. So the recent wow moment — these models landing in the same conversation as the state-of-the-art from OpenAI and Anthropic — is genuinely satisfying to watch. What I didn’t expect is how they got there: not by winning the argument, but by refusing to have it. Three linear blocks, one full-attention block, both mechanisms in the same stack. And notice the pattern repeating from Gemma — nobody is willing to drop quadratic attention entirely. Whatever those few full layers are doing, it isn’t replaceable yet.

DSA — DeepSeek V3.2

The DeepSeek-V3.2 model uses Multi-Head Latent Attention alongside DeepSeek Sparse Attention (DSA), with the indexer queries derived from the shared compressed latent representation rather than the raw input. Two techniques from this post, stacked.

What’s DSA? In one line: a cheap learned dot-product scorer that limits each query to the most relevant tokens before the attention softmax.

Standard causal self-attention attends to all previous tokens for each query, yielding O(L²) compute and O(L) KV-cache growth with sequence length L. Sliding Window Attention already showed above that restricting attention to a fixed local window substantially reduces this cost — in SWA, each query token attends only to a local span of nearby previous tokens.

DSA uses the same broad idea of attending to only a subset of previous tokens, but it replaces the fixed window with a learned selection mechanism. For each query token, the model scores candidate past tokens and keeps only the most relevant ones.

DeepSeek Sparse Attention selects a learned subset of past tokens for each query token.

DSA adds two components on top of standard attention.

1. Lightning Indexer

For each query token $t$ and every candidate past token $s$, the indexer computes a scalar relevance score. This implementation makes the scale factors from the reference code explicit:

where:

• $H_I$ is the number of lightweight index heads,
• $q_{t,j}$ is the indexer query vector for token $t$ and head $j$,
• $k_s$ is a shared indexer key vector for past token $s$,
• $w_{t,j}$ is a learned per-head gate scaled by $1 / \sqrt{H_I}$.

The ReLU zeroes out negative dot-product contributions, and the gated sum aggregates across index heads into a single relevance score per past token.

2. Token Selector

After computing all indexer scores, only the top-K highest-scoring positions are kept. All other positions are masked to −∞ before the standard softmax, so the model effectively attends to only $k \ll L$ tokens.

The ReLU in the indexer is not where the final sparsity comes from. Since the scores are summed over multiple index heads, most final scores can still be nonzero. The token selector is what creates the sparse pattern, by keeping only the top-K positions.

In a fused production implementation, this can lower attention compute from O(L²) to O(L·k). The book’s reference implementation keeps the standard dense attention score matrix and applies the DSA-selected top-K mask before softmax — which makes the selection logic easy to inspect, but doesn’t deliver the fused-kernel compute savings.

The figure below summarizes the flow: the lightning indexer scores candidate tokens, the selector keeps top-K positions, and the resulting mask restricts the usual attention softmax.

DSA first scores candidate tokens, then keeps the top-K tokens for the final attention mask.

My take: SWA guesses that relevance is nearby; DSA learns where relevance actually is — the more honest bet for long documents, where the token you need is often thousands of positions back. What struck me most is how much this resembles sparse retrieval in RAG: the lightning indexer is a cheap first-stage scorer over every candidate, top-K is the retrieval cutoff, and the expensive attention only ever sees the shortlist. Same two-stage shape I keep building around models with a retriever and a reranker — except here it lives inside the model and gets trained end to end instead of bolted on outside.

Cross-layer KV sharing — Gemma 4 E2B, E4B

We discussed GQA above, where several query heads share the same key and value heads. Cross-layer KV sharing applies a related idea across transformer layers instead of within one.

Instead of computing a fresh key and value projection in every layer, later layers reuse K/V tensors from an earlier layer. They still compute their own queries, so each layer can form its own attention pattern. The main memory saving comes from storing fewer K/V tensors in the cache.

This idea is also called cross-layer attention, and it’s described in Brandon et al., Reducing Transformer Key-Value Cache Size with Cross-Layer Attention. Gemma 4 E2B and E4B use a related shared KV-cache scheme, which makes it a natural addition to the GQA, MLA, and SWA techniques above.

In Gemma 4, KV sharing is combined with GQA or MQA and sliding window attention — three of the techniques from this post in one model. The book’s simplified GPT example implements only the cross-layer KV-sharing part, so the code stays focused on the main mechanism.

The simplified rule it uses is:

1. Early layers compute and cache their own K/V tensors.
2. Later layers reuse the most recent K/V tensors from an earlier producing layer.
3. All layers still compute their own query projections.

This reduces the number of K/V caches that grow with context length. The tradeoff is reduced model capacity, because some layers no longer get their own K/V projections.

My take: stacking GQA + MQA + SWA + cross-layer sharing in one model tells you these tricks are mostly orthogonal — each attacks a different axis of the same cache. Which also means the next efficiency jump probably won’t come from one clever idea, but from someone stacking a fourth.

In my work, I have been interviewing for a potential teammate. As this position carries technical responsibility (data engineer), it’s required to utilize AI during the technical interview. The live technical assignment I use has both an AI-supported programming task and an AI-prohibited algorithmic task. As I highlighted live, it’s expected to be impossible to cheat. However, it happened today: so-called AI-cheating, in person.

As I wanted to assess how the candidate utilizes AI, I started with the AI-assisted problem-solving case. The candidate had prepared a repository claiming to have helper functions. I consider this a positive sign, as the candidate had prepared something for this interview.

The candidate starts writing a prompt, and I ask what the intention behind it is. The candidate writes a prompt full of comprehensive keywords. Okay, it’s a good sign. It looks a bit unnatural, injecting only keywords, but okay, maybe the candidate is considering how token embedding works by focusing on keywords.

Initially, the candidate couldn’t understand the purpose of the problem; however, in a short time, the candidate provided better context. I regarded this as fast learning from the problem. Okay, it’s a good sign.

The issue starts with the non-AI-assisted case. I offered a short 2-minute restroom break before the next phase. A brief breath of fresh air, then on to the next task. The candidate read through the problem in detail and understood the case very well. I heard a few typing sounds, so my suspicion started, but I let the candidate continue. First, I didn’t want to offend the candidate. Second, if it’s cheating, it’s better to detect it than to give a chance to stop. It’s better because cheating during an interview means any serious cheating could happen once working together.

Anyhow, the candidate completed the interview quite well. A bit of guidance and support from me as an interviewer was provided, but it was well done. Okay, it’s a good sign. Time to work on the next problem. Again, it’s supposed to be a non-AI-assisted problem. This time, I was sitting next to the candidate, discussing how to solve it. The brilliant candidate becomes silent. Nothing progressed, for one reason only: I’m next to the candidate. Suddenly, a Claude Code artifact pops up during the interview. Okay, it’s apparent that another agent session is running in parallel with the non-AI-assisted challenge. Initially, the candidate switches the tab, but when I ask to open it, the candidate is speechless.

I gently ask the candidate to open the running terminal. It has the whole history of cheating. Interesting. The candidate’s brilliant answers in all phases came from this. The session shows /rc, which means remote control of the session. Not 100% sure, but it’s suspicious that the candidate could read the non-AI-assisted problems’ content through the Claude Code app. The excuse for cheating is apparently: “it’s for my personal learning, I don’t cheat.” Okay, it’s expected. A quick acknowledgment and apology would have been ideal at the end of this. However, it’s hard to admit. Denial is easy.

So, what’s my take? People get dumber by delegating thinking to AI. It’s a great channel to explore knowledge and sharpen ideas. However, as soon as we give up thinking for ourselves, it becomes a counterproductive tool. It is easy to make fun of others, but I should observe myself to see if I do the same.

An attention function can be described as mapping a query and a set of key-value pairs to an output.

The sentence is the paper’s own and it is accurate. It also explains nothing. It does not say why there are three projections instead of one, what each of them costs when the model is served, or which claims about attention maps survive scrutiny.

This post covers what I skipped. It also corrects the name. We say Q, K, V. The mechanism has four weight matrices and two circuits, and the one the name omits is tied for the largest.

1. The same input, different weights

In self-attention, Q, K and V are not three pieces of information about a token. They are one vector passed through three learned matrices.

Here $x$ is one row of the residual stream, the token’s current representation at this layer. Nothing else enters. No extra data is fetched and no side channel is read. The three outputs differ only because the weights differ.

That changes where to look when you want to know what a head does. The answer is in the matrices, not in the token.

The mechanism built on them is one line:

The $\sqrt{d_k}$ divisor is a gradient fix. I used to read it as a normalisation of similarity, which it is not. The paper’s reason is that for large $d_k$, “the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients.”

2. Q and K need separate matrices

If Q and K are both $x$ times a weight, why not use one weight and halve the parameters? The roles already differ by position, since $i$ asks and $j$ is asked.

Set $W_Q = W_K = W$ and the score becomes:

Swapping $i$ and $j$ gives the same number. The affinity matrix is forced symmetric.

The easy version of this argument overstates what that means, and I nearly published it. Symmetric scores are not symmetric attention weights. Softmax normalises row by row, and under a causal mask each row sums over a different set of positions, so $\alpha_{ij}$ and $\alpha_{ji}$ differ either way. What a shared matrix removes is the freedom to set the two affinities independently.

It removes more than symmetry. $WW^\top$ is positive semi-definite, so the diagonal is pinned at $\mathrm{score}(i,i) = \lVert x_i W \rVert^2 \ge 0$, and Cauchy-Schwarz gives:

No token can score another above its own self-score unless that other token has a much larger projected norm. A head built on a shared matrix is biased toward attending to itself, which is a poor property for a mechanism whose job is moving information between positions.

Two matrices give a general bilinear form:

$W_Q W_K^\top$ carries no symmetry constraint and no definiteness constraint. The relation it encodes is directed, and a token is free to find itself uninteresting. Q is what a position looks for. K is what a position offers as a match target. Both come from the same $x$, so a token’s search and its findability are separate learned functions of identical input.

Those two results are derivations rather than citations. I am not quoting anyone here, and the algebra is short enough to check.

3. What V adds that K does not

The third matrix asks a third question: what gets handed over once a match happens.

K decides how a token gets found. V decides what it contributes. Separating the two lets a token be a strong match target while contributing something unrelated to its own identity.

I held two wrong ideas here until recently, and I suspect they are common.

The first was that V holds the token’s content and must therefore be wider than K. It is not wider. The paper introduces “queries and keys of dimension $d_k$, and values of dimension $d_v$”, which allows them to differ, then sets $d_k = d_v = d_{model}/h = 64$. Standard implementations follow.

The second was that V is the token’s data. It is a learned projection. Nothing requires $v_j = x_j W_V$ to resemble token $j$ at all. That assumption is what makes “the head copies the token” feel true when it usually is not.

4. The matrix the name leaves out

After the $h$ heads each produce a $d_{head}$-wide result, the paper concatenates them and projects back to model width:

$W_O$ is the fourth matrix. In Llama 2 70B it ties with $W_Q$ as the largest of the four, and the Q, K, V name gives it no billing.

Block matrix multiplication makes concatenate-then-project identical to each head owning its own slice of $W_O$:

with each $W_O^{(i)}$ of shape $d_{head} \times d_{model}$. The concatenation is notation. The structure underneath is per-head and additive, which is why Anthropic’s circuits framework describes heads as “independent operations, each outputting a result which is added into the residual stream”, and why an individual head can be attributed, ablated, or pruned at all.

The ordering matters. $W_V$ maps a source token down into the head’s private $d_{head}$ subspace. $W_O$ maps that payload back up into $d_{model}$ and chooses which directions of the residual stream the head writes into.

V on its own is not in the residual basis. In Llama 2 70B it is 128 numbers in a subspace no other layer reads. $W_O$ is what makes them legible. So what a head writes is $W_O W_V x$, never V alone.

5. Four matrices, two circuits

The four matrices pair up. Anthropic’s circuits framework splits a head into a QK circuit that computes the attention pattern and an OV circuit that computes how each token affects the output if attended to.

                    x  (residual stream)
                    |
        +-----------+-----------+
        |           |           |
      x W_Q       x W_K       x W_V
        |           |           |
        q           k           v
        |           |           |
        +-----+-----+           x W_O
              |                  |
          QK circuit         OV circuit
         W_Q W_K^T            W_O W_V
       "where to look"     "what to write"

Every claim about an attention head is one of two kinds, and the two need different evidence.

Conflating them is a common failure in writing about attention. “This head handles negation” is two claims stacked: the head finds the negation token, which is QK, and what it writes flips something downstream, which is OV. An attention map supports the first claim only.

6. The two circuits get equal budgets

I assumed the value path must be the smaller one, since Q, K, V reads like three peers and $W_V$ is one of them. Then I looked at a real configuration. Llama 2 70B uses hidden_size 8192, num_attention_heads 64, and num_key_value_heads 8, so $d_{head}$ is 128.

$W_V$ is one eighth the size of $W_Q$. That is grouped-query attention, which I come back to below. The paths still add up to the same total:

Each path is one wide matrix paired with one narrow one, mirrored. It holds under plain multi-head attention too, where all four are $d_{model} \times d_{model}$. Per head, both circuits are $d_{model} \times d_{model}$ maps of rank at most $d_{head}$.

The architecture spends the same on where to look and on what to write. Nothing in it treats the content path as needing more capacity.

7. Only two of the four cost anything at inference

The four are symmetric in budget and completely asymmetric in behaviour during generation.

In a decoder-only model the causal mask means position $t$ attends only to positions up to $t$. So $x_j$ at every layer depends only on tokens up to $j$. Append a token and nothing about position 3 changes, at any depth. Its $k$ and $v$ are frozen the moment it is in the past. Its $q$ is dead weight, because position 3 already did its lookup and will never repeat it.

That is the derivation of the KV cache. K and V are stored because they are re-read at every future step. Q is discarded because it is used once. Causality does the work here rather than the naming. In a bidirectional encoder, deeper-layer keys for past positions do change when a token is appended, because the layers below attended to it, and caching would be invalid.

The cost of what is kept:

Q does not appear in it. Neither does $W_O$. Only two of the four matrices produce anything that has to be stored.

Storing it is the bottleneck. Training parallelises across the sequence and incremental decoding cannot. Shazeer’s multi-query attention paper puts it plainly: decoding is “often slow, due to the memory-bandwidth cost of repeatedly loading the large ‘keys’ and ‘values’ tensors.” The constraint is moving the cache rather than arithmetic.

That formula has exactly one term an architect can cut without touching depth, width, or context length, and it is $n_{kv}$. Multi-query attention sets it to 1. Grouped-query attention sets it to some $g$ between 1 and $h$, using “an intermediate (more than one, less than number of query heads) number of key-value heads” to reach “quality close to multi-head attention with comparable speed to MQA.” Llama 2’s 34B and 70B use it, with 8 key-value heads against 64 query heads.

Query heads stay at $h$ in all three variants. That is why $W_V$ is eight times smaller than $W_Q$ in the table above.

Sharing keys and values does not collapse the heads. Eight query heads over two key-value heads still produce eight distinct attention patterns, because $W_Q$ differs per head even where $W_K$ is shared. What grouped-query attention gives up is memory, not the diversity that multi-head exists for.

8. Heads partition the model, they do not add to it

Multi-head attention does not widen the layer.

The original model uses $h = 8$ with $d_k = d_v = d_{model}/h = 64$ and $d_{model} = 512$. Eight heads is 512 divided eight ways, not eight copies of 512. The paper is explicit that “due to the reduced dimension of each head, the total computational cost is similar to that of single-head attention with full dimensionality.”

At fixed $d_{model}$, adding heads is a trade rather than an upgrade. You get more simultaneous lookups, each in a narrower subspace.

What the trade buys is stated in one sentence: multi-head attention “allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.” The operative word is averaging. One softmax produces one distribution per position, and a position usually needs several unrelated things at once, such as its syntactic parent, its antecedent, the previous token, and the matching bracket. One distribution cannot peak in four places without smearing.

Whether all those heads earn their place is a separate question, and the evidence has a boundary I have to mark. Voita et al. pruned 38 of 48 encoder heads on an English to Russian WMT model for a drop of 0.15 BLEU, and found that “the most important and confident heads play consistent and often linguistically-interpretable roles” and prune last. Michel et al. found many heads removable at test time, with some layers reduced to a single head.

Both studies used 2019-era encoder-decoder NMT models and BERT. Neither says anything direct about a modern decoder-only LLM at scale. “Most heads in GPT-class models are redundant” gets repeated constantly and does not follow from these papers. I am not making that claim.

9. What an attention map cannot tell you

The tempting move with all of this is to screenshot a heatmap and point at the bright cell.

The mechanical reason not to comes straight out of the two circuits. An attention map is a QK object. It records where a head looked. It never contains V or $W_O$, so it cannot record what the head wrote. The contribution of a source token is $\alpha_{ij} W_O W_V x_j$, and if that lands near zero, or in directions no downstream layer reads, a large $\alpha_{ij}$ moves nothing. Per head that map has rank at most 128 out of 8192 dimensions, which leaves room to write nothing at all.

The empirical literature agrees and remains unresolved. Jain and Wallace found attention weights “frequently uncorrelated with gradient-based measures of feature importance”, with very different attention distributions producing equivalent predictions. Wiegreffe and Pinter replied that the conclusion depends on the definition of explanation, and proposed four concrete tests. Cite both or neither.

There is also the blunt fact of attention sinks. Models assign strong attention to initial tokens “even if they are not semantically important.” A head putting 0.6 of its mass on token 0 is usually reporting nothing.

The position I would defend is neither that attention maps explain the model nor that they are meaningless. They are QK evidence. They constrain where a head looked, and any claim about what it did needs the other circuit.

What I check now

The first three take the same input. Different weights give them different jobs, different lifetimes, and different bills.

Reading an attention implementation now, I check four things: which of the four matrices are shared across heads, which of them are cached, what rank each circuit has, and whether a claim being made about a head is QK or OV. My 2022 post said that attention maps a query and a set of key-value pairs to an output. That is still true. The difference is that I can now say what each of those words costs.

References

• Vaswani et al., Attention Is All You Need.
• Elhage et al., A Mathematical Framework for Transformer Circuits.
• Shazeer, Fast Transformer Decoding: One Write-Head is All You Need.
• Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.
• Touvron et al., Llama 2: Open Foundation and Fine-Tuned Chat Models.
• Voita et al., Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned.
• Michel et al., Are Sixteen Heads Really Better than One?.
• Jain and Wallace, Attention is not Explanation.
• Wiegreffe and Pinter, Attention is not not Explanation.
• Xiao et al., Efficient Streaming Language Models with Attention Sinks.
• 3Blue1Brown, Attention in transformers, step-by-step.

This post explains what a JWT is, what it carries, and how you can put your own information inside it — like which environment (prod or dev), which customer, and which exact permissions a caller has. No security background needed.

The boarding pass analogy

A JWT is a boarding pass for software.

When you fly, you first go to the check-in counter, prove who you are with your passport, and receive a boarding pass. From then on, nobody at the gate calls the check-in counter to ask “is this person really allowed on this flight?” They just look at the pass. It says who you are, which flight you may board, which seat class you get, and it has security features so nobody can forge it.

The same flow happens in our system:

 You (or an AI agent)          Keycloak                 MCP server
        |                  (check-in counter)         (boarding gate)
        |                          |                        |
        |--- 1. "Here is my  ----->|                        |
        |     username/password"   |                        |
        |                          |                        |
        |<-- 2. Boarding pass -----|                        |
        |     (the JWT)            |                        |
        |                          |                        |
        |--- 3. "I want to read the database. Here is ----->|
        |        my JWT."                                   |
        |                                                   |
        |<-- 4. Gate checks the pass. Valid? Come in. ------|

Keycloak is the check-in counter: it verifies your identity once and hands you a signed pass. The MCP server is the gate: it never needs to see your password. It only reads the pass and checks the security seal.

A JWT has three parts

If you look at a raw JWT it seems like gibberish:

eyJhbGciOi...  .  eyJpc3MiOi...  .  SflKxwRJSM...
   HEADER            PAYLOAD          SIGNATURE

Three chunks of text separated by dots. Each has one job:

+----------------------------------------------------------+
|  HEADER      "What kind of pass is this?"                 |
|              e.g. { "alg": "RS256", "typ": "JWT" }        |
|              (which sealing method was used)              |
+----------------------------------------------------------+
|  PAYLOAD     "The actual facts" (called *claims*)         |
|              who you are, who issued it, what you may do  |
+----------------------------------------------------------+
|  SIGNATURE   "The tamper-proof seal"                      |
|              proves Keycloak wrote it and nobody          |
|              changed a single letter afterwards           |
+----------------------------------------------------------+

One thing surprises everyone: the payload is not encrypted. Anyone who holds the token can read it, just like anyone who picks up your boarding pass can read your name and seat. What they cannot do is change it — if even one letter of the payload is altered, the signature no longer matches and the gate rejects the pass. So the rule is simple:

Put facts in a JWT, never secrets. Names, roles, permissions: yes. Passwords, API keys: never.

The facts inside: claims

Each fact in the payload is called a claim. Some claims are standard — every JWT system in the world understands them:

A minimal payload from Keycloak looks like this:

{
  "iss": "https://auth.example.com/realms/mcp",
  "aud": "mcp-server",
  "sub": "user-42",
  "exp": 1767100000,
  "realm_access": { "roles": ["analyst"] }
}

Adding your own facts: custom claims

Standard claims answer “who are you and who issued this?” But real systems need more context. In our case the MCP server needs to know three extra things before it does anything:

1. Which environment is this token for — production or development?
2. Which customer’s data may this caller touch — customer A or B?
3. Exactly which actions are allowed — read the database? read a sheet? write to it?

All three fit naturally into the payload as custom claims:

{
  "iss": "https://auth.example.com/realms/mcp",
  "aud": "mcp-server",
  "sub": "user-42",
  "exp": 1767100000,
  "realm_access": { "roles": ["analyst"] },

  "env": "prod",
  "customer": "A",
  "permissions": ["db:read", "sheet:read", "sheet:write"]
}

Think of these as extra stamps on the boarding pass: “Lounge access: yes. Priority boarding: no.”

Why the resource:action pattern is worth copying

Notice the shape of the permissions: db:read, sheet:write. That’s thing colon verb. It stays readable as the system grows, and the check on the server side becomes one line of logic:

Request: "write row 5 to the sheet"
Check:   does the token's permissions list contain "sheet:write"?
         yes -> do it
         no  -> refuse (403 Forbidden)

Roles and permissions work together but answer different questions. A role is a job title (“analyst”); a permission is a concrete allowance (“may write to sheets”). In Keycloak you define roles, attach permissions to them, and Keycloak expands everything into the token automatically. Users are managed by role; the MCP server only ever checks permissions.

How to add custom claims in Keycloak

Keycloak calls this feature protocol mappers (in newer versions you’ll find it under client scopes). No code required:

1. Store the fact on the user or client — e.g. a user attribute customer = A (Users → your user → Attributes).
2. Create a mapper that copies the attribute into the token (Clients → your client → Client scopes → Add mapper → User Attribute, set the claim name to customer).
3. That’s it. Every new token for that user now contains "customer": "A".

The env claim is often simplest to handle differently: run one Keycloak realm (or client) per environment, and hard-code the claim with a hardcoded claim mapper — env: prod in the prod realm, env: dev in the dev realm. Then a dev token can never claim to be a prod token, because the prod issuer never signed it.

The full journey of one request

Here is everything together — what happens when an AI agent asks our MCP server to write to a spreadsheet:

 AI agent                                        MCP server
    |                                                |
    |  "Append this row to the sheet."               |
    |  + JWT                                         |
    |----------------------------------------------->|
    |                                                |
    |                     The gatekeeper checklist:  |
    |                                                |
    |          1. Signature valid?  (not forged)     |
    |          2. iss = our Keycloak?  (right issuer)|
    |          3. aud = mcp-server?  (meant for us)  |
    |          4. exp in the future?  (not expired)  |
    |          5. env = prod?  (right environment)   |
    |          6. customer = A?  (right data scope)  |
    |          7. "sheet:write" in permissions?      |
    |                                                |
    |     ALL seven pass -> row is written           |
    |     ANY one fails  -> request refused          |
    |<-----------------------------------------------|

Steps 1–4 are the standard checks every JWT system does. Steps 5–7 are ours — and they only exist because we put env, customer, and permissions into the token as custom claims.

The elegant part: the MCP server made all seven decisions without contacting Keycloak and without storing any user data. Everything it needed was on the pass. That’s why JWTs are the default choice for connecting many small services — each gate can verify passes on its own, as long as it knows the issuer’s public key.

Takeaways

• A JWT is a signed, readable, short-lived boarding pass: header (seal type), payload (facts), signature (the seal).
• Readable ≠ forgeable. Anyone can read it; nobody can change it. So: facts in, secrets out.
• iss says who printed the pass, aud says which gate it’s for, exp says when it dies. Check all three, always.
• Custom claims let you carry your own facts — env, customer, permissions — and Keycloak adds them with mappers, no code.
• The resource:action permission format (db:read, sheet:write) keeps authorization checks one line long, forever.

Too many agents at once

How many agents are you running right now?

One day, I had 3 quite complex linear tickets. I created three worktrees and start running agent sessions. Some data issue was reported, so I added one more agent to analyze and debug the problem. Some model traininig was requested, so I added another agent session. Switching among multiple sessions is not free. You can tell it’s just switching tab, however, your brain can’t switch that fast. It’s not only draining, but also quality of your decision plumet.

Agents scale. Your review capacity does not.

Somewhere past two concurrent diffs, you stop reviewing and start approving. Nothing warns you when you cross that line. The output still looks like work getting done, and the pull request still gets a thumbs up.

I call this the orchestration tax. You pay it in the only currency you cannot mint more of. Opening a sixth agent does not add a sixth unit of throughput. It takes review time away from the other five.

So the ceiling is not a number I can give you. Run as many as you can verify. If you cannot say what would make you reject a diff before you open it, you are not reviewing it.

Part of that ceiling is a tooling problem. A terminal or a VS Code window is built for one session at a time. Once you go past a couple of agents, switching between them costs more than the work does, and you start approving out of navigation fatigue. A tool that shows parallel sessions at a glance genuinely raises the number you can verify. It will not make you a better reviewer, but it gives you back the time you were spending on managing.

There are couple of solutions, but Superset is my favorite. I love their minimal setup.

One session, until auto-compact?

Do you know what auto-compact means?

If you use Claude Code all day you have seen the symptoms:

• It re-reads a file it already read.
• It re-proposes the approach you killed an hour ago.
• It edits against a file state that no longer exists.

If auto-compact fired, you were already too late

Auto-compact is not a feature you use. It is the system telling you it is about to throw away detail on your behalf, and it does not ask which details you cared about. By the time you see the notice, the choice of what survives has gone to a summarizer that never saw you wince at one particular line.

Use the agent at its smartest state

Lossy summarization is only half of the problem. Long contexts are not uniformly useful in the first place.

From Lost in the Middle: How Language Models Use Long Contexts.

Accuracy is highest when the answer sits at the start or the end of the context and sags in the middle. In that experiment the middle positions drop to roughly the closed-book baseline, which means the model did about as well as if you had given it nothing. Retrieving the right document into the window does not guarantee the model uses it.

Why the curve has that shape:

• The end performs well because of recency. Generated answer tokens sit closest to the final prompt tokens, and many attention patterns favour nearby information.
• The beginning performs well because of primacy. Early tokens usually carry the instructions and the framing, they influence many later representations, and they tend to get disproportionate attention.
• The middle performs worse because of competition. Relevant tokens are surrounded by distractors, they receive weaker attention, and models are not trained to use every position in a long context equally well.

A bigger window does not fix this. It just moves the sag further out.

From Evaluating Coding LLMs at 1M Context Windows.

Read this one across the columns rather than down them. Scores do not climb as the window grows. On the code-editing side they sit near the floor almost everywhere, and several models peak well before their maximum context. The advertised window is a capacity limit, not a working range.

So do not use the whole 1M just because you have it. Watch your context continuously. /statusline turns it into something you can see instead of something you find out about later.

My own rule: every time a session passes 200k, I stop and decide. Either I /compact deliberately, or I hand off and /clear. What matters is that I pick what survives instead of letting the trigger pick.

Treat your context like your baby. Feed it only what it needs. And /rewind is not an emergency exit you use once a month. It is how you undo a bad turn before it contaminates everything after it.

The tokens you paid before you typed anything

Everything above is about the session you are already in. Some of your context is gone before that. Every skill you install puts a - name: description line into a listing that goes out with every turn, whether you use the skill or not. Plugins and MCP servers cost you the same way.

That listing sits at the very front of the window, which is the position the graph above says matters most. So if you have a hundred skills installed and use four, your best context is spent describing ninety-six things that will never run. The unused ones also give the agent more wrong answers to pick from.

I built context-saver because I wanted to see the number. It is a local single-file UI that shows what each skill, plugin and MCP server costs you in always-paid tokens, and lets you switch off what you do not use. You click, review the diff, and it writes ~/.claude/settings.json with a backup. You can also set a skill to name-only or user-invocable-only, so it stops advertising itself but still works when you ask for it.

Context management is the lowest hanging fruit in getting more out of an agent. It costs nothing except discipline.

Save your attention

You type “make it better”, the agent works, and then you eyeball the diff. Then you do it again on the next completion, and the one after that. Every completion bills your attention, whether the change was any good or not.

The default setup is on the left. You are inside the loop. Every iteration has to pass through you, so it can only run as fast as you can look at it, and it stops the moment you go to lunch.

Loop-engineering is on the right. You are not in the loop. You define the goal the loop runs against, and an orchestrator agent takes the seat you used to sit in. Iteration continues without you. You spend attention once, on stating the target, rather than once per completion.

That only works if the orchestrator can tell whether the goal was met, which is where the design gets interesting.

Notice the split. The orchestrator does not have to be the strongest model in the room. It dispatches work, reads the report, and checks it against the goal statement, and a cheap fast model handles that fine. The expensive models do the actual work. Checking is the easy half. Writing a goal that can be checked is the part you have to think about.

Which is why I keep the condition strict. The goal has to be measurable quantitatively, not qualitatively. Two real ones from my own work:

• MCP authentication: does it support dual mode for an admin user?
• Shopify intra-market extraction: extract every market without blowing up the row count on the dataframe merge.

Both of those have an answer a command can produce. “Make the auth cleaner” does not.

So when you write the goal, name the command that proves it, and pin down how it runs. Which interpreter, which environment, which tree. If it only passes on your machine, you have not specified a goal yet, you have described a feeling.

Where this leaves me

The three tips are really one tip. Agents are abundant and my attention is not, so everything else follows from that.

Cap concurrency at what I can verify. Curate the context before the system curates it for me. Turn measurable tasks into loops that run without me.

None of this needs a better model. It needs me to decide what done looks like before the work starts, which is the part I still get wrong more often than I would like.

The failure may have started during parsing. Dense search may have missed an error code. Fusion may have dropped a useful candidate, or the reranker may never have received it. Even correct retrieval can fail when context packing removes a prerequisite.

I now debug RAG by following the evidence. At each stage I ask where the required source text disappeared.

The example below is a simplified composite. The product and error code are fictional, but the failure pattern is common in operational documentation.

A small case with several retrieval problems

Imagine a gateway runbook containing these facts:

Event E-4317 means that majority quorum is unavailable. A replacement gateway joins as a non-voter and becomes a voter only after synchronization. The standby takes over after three missed heartbeats, but automatic takeover is blocked until quorum returns.

Users may ask:

• “What does E-4317 mean?”
• “How does the backup gateway take over?”
• “How do I diagnose E-4317 after replacing a gateway?”

Dense search handles the second query well because it can connect “backup” with “standby” and “take over” with “assumes active duty.” BM25 has the clearer advantage on the exact error code. The mixed query needs both signals and more than one source span.

The dangerous result is an answer that explains failover but omits that the replacement is still a non-voter. The text sounds relevant, yet it is incomplete. Looking only at the final LLM call hides the real failure.

The pipeline is a series of cutoffs:

parse and version, create citable spans, build dense and sparse representations, filter eligible points, retrieve candidates, fuse, rerank, pack context, then generate.

Once evidence falls outside a cutoff, a later stage cannot recover it:

1. Ingestion sets the limits

I keep the source text separate from the text used for retrieval.

A sentence such as “the replacement initially joins as a non-voter” is a clean citation but a weak standalone search document. Prefixing its title and heading can improve retrieval without pretending that the prefix appeared in the source.

Qdrant can store named dense and sparse vectors on the same point. I use the point ID as the alignment boundary:

client.create_collection(
    collection_name="runbook_chunks",
    vectors_config={
        "dense": models.VectorParams(
            size=DENSE_DIMENSION,
            distance=models.Distance.COSINE,
        )
    },
    sparse_vectors_config={
        "bm25": models.SparseVectorParams(modifier=models.Modifier.IDF)
    },
)

The payload carries tenant_id, document_id, source and pipeline versions, offsets, exact source text, index text, authorization groups, and an is_current flag. The dense and sparse branches therefore refer to the same span and permissions.

FastEmbed’s Qdrant/bm25 model expects Qdrant to apply inverse document frequency, which is why the sparse vector uses Modifier.IDF.

I derive deterministic point IDs from tenant, document, source version, pipeline version, and source offsets. Retrying ingestion then overwrites the same points. This makes the final state idempotent, but it does not make repeated writes free.

Deterministic IDs also leave stale points behind when a document changes from ten chunks to eight. An ingestion manifest must reconcile old IDs after a successful run. Qdrant’s point version is an internal operation version, so the application still needs its own source_version.

Rechunking requires a full rebuild of the affected dense and sparse indexes. Boundaries change the embeddings, BM25 document lengths, average length, document frequency, and candidate population. In production I build a new collection, validate it, and switch an alias.

2. Dense and BM25 scores mean different things

For query vector $q$ and document vector $d$, cosine similarity is:

This score says that an encoder placed two texts in similar directions. It does not prove that the chunk contains an exact identifier, supports a complete answer, or belongs to the current authorized corpus.

Some models use different prompts for queries and passages. FastEmbed exposes query_embed(...) and passage_embed(...) for that reason. Using one generic embedding path can bypass the setup an asymmetric model was trained to use.

BM25 keeps lexical evidence. Its score is:

Term frequency $f(t,d)$ rewards matches but saturates through $k_1$. IDF rewards rare terms. The parameter $b$ adjusts for document length. FastEmbed’s Qdrant BM25 defaults are k1=1.2, b=0.75, and avg_len=256, but tokenization and identifier handling often matter more than small parameter changes.

I choose the first baseline from the query distribution:

Hybrid search earns its extra cost only when the union improves candidate recall.

3. Fusion can only reorder candidates

Let the dense and sparse top-$n$ sets be $D_n(q)$ and $S_n(q)$. Their candidate union is:

Before tuning fusion, I check whether the required evidence exists in $U_n(q)$. If it is missing, I inspect chunk boundaries, index text, filters, approximate search, thresholds, and branch limits.

Query rewriting can damage this stage. Replacing E-4317 with “quorum problem” removes the strongest sparse feature, so I preserve the literal query for BM25.

Dense cosine scores and BM25 scores have no shared scale. Reciprocal Rank Fusion avoids adding them directly:

RRF uses rank positions. A point near the top of both branches receives a contribution from each. The original RRF paper used $k=60$. Qdrant uses zero-based ranks and defaults to $k=2$. Those settings produce different behavior, so experiment logs should include the engine, rank convention, $k$, branch limits, and weights.

Weighted RRF can prefer one branch, but the weights map to branch positions rather than raw scores. I leave them equal until a labeled evaluation set justifies changing them.

Qdrant also supports Distribution-Based Score Fusion. DBSF normalizes scores using the mean and standard deviation of each returned branch, then adds them. It is worth testing when score magnitude within a retriever is stable. RRF is a safer starting point when score scales move between queries.

The compact Qdrant query below shows the important part:

eligible = models.Filter(
    must=[tenant_condition, acl_condition, current_version_condition]
)

points = client.query_points(
    collection_name="runbook_chunks",
    prefetch=[
        models.Prefetch(
            query=dense_query, using="dense",
            filter=eligible, limit=candidate_limit,
        ),
        models.Prefetch(
            query=sparse_query, using="bm25",
            filter=eligible, limit=candidate_limit,
        ),
    ],
    query=models.RrfQuery(rrf=models.Rrf()),
    query_filter=eligible,
    limit=rerank_pool,
    with_payload=True,
).points

candidate_limit is a recall ceiling. A point outside both prefetch lists is invisible to fusion and reranking. I plot candidate-union recall against latency instead of copying a default such as 30.

The same eligibility filter belongs on every branch. Filtering a global top-k afterward is unsafe and wastes candidate capacity on points the user cannot access. Mandatory tenant and ACL filters should come from authenticated server-side identity. Frequently filtered payload fields also need indexes.

Authorization checks continue during context expansion. An authorized anchor does not make an unauthorized neighbour safe to attach.

4. Reranking and context construction solve different problems

A cross-encoder reads the query and candidate together. This allows deeper token interaction than a dual encoder, but it is too expensive to run over the full corpus.

The reranker is limited by its input:

• It cannot recover evidence missing from the candidate union.
• Long candidate text may be truncated at the useful passage.
• A model trained on web passages may fit code or internal terminology poorly.

Model licenses matter too. A strong benchmark result does not grant commercial usage.

I test several rerank pool sizes. For each one I record union recall, final nDCG or evidence recall, latency, and reranked tokens. If evidence often appears around rank 35, a pool of 20 has already lost it. If recall saturates at 20, scoring 100 wastes time.

After reranking, the problem changes from ranking chunks to assembling evidence. The context builder may add a prerequisite, include the next warning, merge overlapping offsets, or replace several children with a parent section. Expansion should happen after reranking and must respect document, version, tenant, and ACL boundaries.

For the gateway query, the packed context needs three facts: what E-4317 means, why the replacement is a non-voter, and why failover remains blocked without quorum. Returning only the heartbeat sentence produces an unsafe answer even though that sentence is relevant.

I send exact source text to the generator and keep retrieval prefixes in index_text. Each source block carries an ID and offsets. The generation prompt requires citations, forbids invented procedures, and asks the model to abstain when the sources are incomplete or contradictory. The application can at least verify that every cited ID exists in the supplied context.

Longer context is not automatically better. Lost in the Middle showed that evidence position can affect model performance. I log packed order, unique source coverage, redundancy, and token count.

5. Evaluation should locate the failure

An end-to-end answer score is useful for release decisions. Diagnosis needs a frozen corpus, source-level evidence labels, and traces from every stage.

The query set should include exact identifiers, paraphrases, mixed queries, rules with exceptions, procedures, stale versions, authorization boundaries, and unanswerable questions. A single average can hide a retriever that improves natural-language queries while breaking error codes.

ANN recall and retrieval relevance are separate checks. I compare Qdrant’s approximate dense results with exact=True on a representative sample. If exact search misses the evidence, I inspect the embedding and index text. If exact search finds it but HNSW does not, I inspect hnsw_ef and index settings.

Recall@k measures candidate coverage. MRR is useful when the first correct hit dominates, while nDCG handles graded relevance across several positions. The measured $k$ should match the number of candidates consumed by the next stage.

Fusion weights, thresholds, candidate limits, and rerank pool size are tuned on one split and reported on another. Otherwise the evaluation mostly measures how well the system fit its test questions.

A short failure-analysis loop

I change one layer at a time. Adding BM25 can fix exact-code recall. Fusion can improve the mixed candidate set. Reranking can improve order inside that set. Context expansion can restore a prerequisite. None of them can repair a source span that ingestion never created.

What I would build first

My first production version would use structure-aware parsing, stable source offsets, heading-prefixed index text, and named dense and BM25 vectors on the same Qdrant point. Every retrieval branch would receive server-derived authorization and version filters.

I would start with equal-weight RRF, measure the candidate limit, rerank only the surviving union, then expand neighbours or parents under the same eligibility rules. The generator would receive exact source blocks with IDs and would cite or abstain.

The trace would record branch IDs and scores, union overlap, fusion settings, reranker scores, expanded offsets, packed order, citations, and per-stage latency. Without that trace, “retrieval failed” is only a description.

The useful design question is:

For this query distribution, which evidence does each stage preserve, where can it disappear, and what trace would prove it?

That question has been more useful to me than debating dense, sparse, or hybrid retrieval in isolation.

References

• Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering.
• Robertson and Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond.
• Cormack et al., Reciprocal Rank Fusion.
• Nogueira and Cho, Passage Re-ranking with BERT.
• Liu et al., Lost in the Middle.
• Ru et al., RAGChecker.
• Qdrant, Hybrid and Multi-Stage Queries, Filtering, and Reranking with FastEmbed.

What is a good compact strategy for a long coding session?

I answered quickly. Keep the objective, decisions, changed files, test state, unresolved problems, and next action. Do not spend summary tokens copying a file that the agent can read again.

Afterwards, I wondered how close that answer was to what an actual coding agent does. I had seen a reconstructed Claude Code repository, so I went through its compaction code.

The repository was widely called a source leak. More precisely, people reconstructed it from a sourcemap published in an npm package. I do not treat it as an official API or as documentation of the current release. Some paths depend on feature flags, and parts have already changed. For example, the snapshot disabled thinking in one fallback summarization path. The current Claude Code context documentation says that v2.1.198 inherits the session’s extended thinking setting during compaction.

Still, the snapshot answered my question. Claude Code does not rely on one perfect summary. It removes context that is cheap to recover, summarizes the remaining conversation, and then reloads selected state.

context after compaction =
    persistent instructions
  + summary of decisions and progress
  + state reloaded from source
  + sometimes a recent verbatim tail

The summary is only one part of the new context. That was the part I had missed.

What the source snapshot shows

The traditional path looks like this:

Structured conversation
        ↓
Keep messages after the latest compact boundary
        ↓
Optionally clear old tool results
        ↓
Append a summarization request
        ↓
Run one model turn with tools blocked
        ↓
Keep the summary and reload selected context

Claude Code starts after the latest compact boundary. It does not summarize raw history that was already compacted. Otherwise each new compact would process both the old conversation and the summary made from it.

Text after /compact becomes a custom focus:

/compact focus on database decisions and exact test failures

This tells the summarizer which details matter most. Current documentation also supports persistent compact instructions in CLAUDE.md.

Old tool results go first

Before full summarization, Claude Code can perform micro-compaction. It looks for large, old results from file reads, shell commands, search tools, web tools, and file edits. Those results can be replaced with a short cleared-content marker.

Parts of this behavior are feature-gated in the snapshot, so installations may differ. The general order is also documented officially: Claude Code clears older tool output first and summarizes the conversation when it needs more room.

That order makes sense. A file can be read again. A test can be rerun. A decision made after reading the file and discussing the failed test may exist only in the conversation.

I classify context by how I would recover it:

The third row needs most of the summary budget. Copying the first two rows in full usually wastes it.

The summarizer sees structured messages

I had assumed that compacting meant rendering the session as one long transcript and asking for a shorter string. The snapshot normally keeps the Claude API message structure. User messages, assistant messages, tool_use blocks, and tool_result blocks remain distinct. Claude Code appends a new user message that asks for the summary.

This distinction helps. A failed test returned by a tool is evidence. A user message that changes the requirement is an instruction. An assistant suggestion that the user rejected should not become the final decision.

The preferred implementation forks the existing conversation so it can reuse the prompt cache. If that path fails, Claude Code sends the active messages explicitly. The summarizer gets one turn and cannot call tools. In the inspected snapshot, compact output is capped at 20,000 tokens.

The compact prompt asks for a structured handoff. It covers the user’s intent, technical concepts, files, errors, corrections, pending tasks, current work, and the next step. It also asks for filenames, signatures, relevant code, and direct quotes when they constrain what should happen next.

The model temporarily produces an analysis and a summary. Claude Code discards the analysis and keeps the summary. It then wraps that summary in a synthetic user message explaining that the session is continuing from an earlier conversation.

Claude Code reloads state afterwards

The next context contains more than that synthetic summary. Claude Code can restore recent files, project instructions, memory, invoked skills, attachments, and context produced by hooks. The current documentation confirms that the system prompt remains loaded and that project-root CLAUDE.md and auto memory are re-injected.

Suppose Claude turns 150,000 tokens of conversation into a 6,000-token summary. The next request may still contain 40,000 or 60,000 tokens after the system prompt, project instructions, skills, and restored files return. These numbers are only illustrative. The point is that post-compact context is not measured by summary length alone.

A summary may record that src/auth/session.ts changed and explain why. The file system provides its exact current contents. It may record that one test failed on a race-condition assertion. The next turn can rerun the test if it needs the complete trace.

The full transcript also remains on disk as JSONL under ~/.claude/projects/. It is available for inspection and recovery without occupying the active context on every request. The session documentation warns that this file format is internal and may change.

An experimental memory path

The snapshot contains another feature-gated path. If /compact has no custom instruction, Claude Code may reuse a structured session-memory file instead of requesting a fresh summary at compact time.

That memory tracks current state, task details, files, errors, decisions, results, and a worklog. The snapshot aims to keep it around 12,000 tokens, then combines it with a recent verbatim tail. This spreads summarization across the session rather than making every retention decision when the context is already full.

In this version, adding custom compact instructions skips that initial shortcut and favors a fresh focused summary. The exact branch may change, but it gives me another reason to add a focus when continuity matters.

How I compact now

Before reading the implementation, I concentrated on writing a better summary prompt. Now I pay more attention to what enters the session and where durable information lives.

Large research reads can happen in a separate agent context. A command that produces 20,000 lines can save the complete output to a file and return only the error or statistic I care about. Test commands, repository conventions, and architecture constraints belong in project instructions, skills, or hooks.

For important decisions, I want four pieces of information:

Decision: what we chose
Reason: why we chose it
Evidence: the relevant file, command, test, issue, or URL
Status: accepted, provisional, or rejected

Failed approaches belong in the summary too. Otherwise the next context can repeat an attractive dead end. I keep user corrections, newly discovered constraints, and unresolved contradictions.

I also prefer to compact between phases. Once exploration has produced a decision and evidence, I can compact before implementation starts. Compacting in the middle of debugging is riskier because observations are still changing and a recent tool result may contain a clue I have not understood yet.

After compaction, I read the generated summary once. I check that modified files are named, rejected ideas are not presented as decisions, removed evidence still has a path or reproduction command, and the next action is concrete.

The instruction I would use

For a long implementation task, I would run:

/compact preserve:
- the exact objective and acceptance criteria
- user corrections and non-negotiable constraints
- decisions with their rationale
- modified files and important symbols
- reproduction commands and exact failing tests
- failed approaches and why they failed
- unresolved risks and the immediate next action

Prefer paths and commands over copied files or long logs that can
be recovered from the repository.

If this applies to every session, I would put a shorter version under # Compact instructions in CLAUDE.md.

Before reading the source, I thought a good compact strategy mostly meant asking for a better summary. My practice now starts earlier. I keep durable rules in files, avoid filling the main session with complete logs, and compact between phases when I can.

The summary still matters. I ask it to preserve why the work reached its current state and what should happen next. Files and commands can supply the exact details they already store better.

References

• Anthropic, How Claude Code works.
• Anthropic, Explore the context window.
• Anthropic, Manage sessions.
• Reconstructed source mirror, src/services/compact. I used it as a historical snapshot, not as an official interface.

That worked often enough that I did not question it. Then I started looking more closely at retrieval failures. Some answers were split across chunks. Other chunks contained the answer but were too vague to match the query. Sometimes retrieval was fine and the LLM still got the wrong context.

“Chunk size” is too small an idea for all of those problems. I now separate chunking into four decisions:

• B, boundary: which source spans can be retrieved?
• R, representation: what text or vectors does search use for each span?
• P, payload: what does the answer-writing LLM receive after retrieval?
• E, evaluation: does the test resemble the work the system will do?

I write this as:

B × R × P, judged by E.

It is a mnemonic, not an equation. An ambiguous representation can ruin a sensible boundary, and a bad final prompt can waste a perfectly good retrieval. That sounds obvious written down. It was less obvious when all four choices hid behind one chunk_size setting.

Part I: one warranty question, four decisions

Consider these lines from a vehicle manual:

Traction-battery warranty

The traction battery is covered for eight years.
The coverage ends earlier when the vehicle reaches 160,000 km.

The user asks:

When does the traction-battery warranty end?

The answer needs the subject, the eight-year limit, and the distance exception. Following those pieces through the pipeline makes B, R, P, and E much easier to distinguish.

B: which spans exist?

Suppose ingestion creates these chunks:

Chunk 1: Traction-battery warranty

Chunk 2: The traction battery is covered for eight years.

Chunk 3: The coverage ends earlier when the vehicle reaches 160,000 km.

Chunk 1 matches the subject but contains no answer. Chunk 2 has the time limit. Chunk 3 has the exception, though “the coverage” is unclear on its own.

A retriever may return all three. Still, the candidate set is awkward because none of its members states the complete rule. This failure happened before ranking. The splitter broke apart evidence that a reader needs together.

For this document, I would probably keep the rule and its exception in one chunk:

Chunk 1: Traction-battery warranty

Chunk 2:
The traction battery is covered for eight years.
The coverage ends earlier when the vehicle reaches 160,000 km.

I would not merge every adjacent sentence. I only want to preserve relationships that a reader actually needs. In manuals, that often means keeping table headers with rows, rules with exceptions, and procedure steps with their warnings or prerequisites. Headings and captions need similar care. Token limits still matter, but they are constraints on a boundary policy, not a definition of meaning.

The first check is simple: does the index contain a complete, citable span? If it does not, a better embedding model will not invent one. I need to change the parser or splitter and rebuild the affected indexes.

R: what does search see?

Now suppose I deliberately keep this short source span:

The coverage ends earlier when the vehicle reaches 160,000 km.

It is a good citation. It is also hard to retrieve by itself because “the coverage” could refer to almost anything. I can leave the source untouched and index a more descriptive version:

Document: 2026 Vehicle Warranty
Section: Traction-battery warranty
Context: This section defines the time and distance limits of battery coverage.

The coverage ends earlier when the vehicle reaches 160,000 km.

The title and heading make the sentence easier to match with a query about the traction battery. They do not need to appear in the citation as though they came from the manual.

At this point there are three different objects, which is where I kept getting confused:

The simplest representation is the chunk text with its existing title and heading path. Contextual Retrieval goes further: an LLM writes a short, chunk-specific prefix before ingestion, and that prefix becomes part of the embedding and BM25 index. Late chunking takes another route. It encodes a larger context first, then pools the token vectors that belong to the smaller span. A multivector index can also keep several local representations instead of forcing the whole chunk into one vector.

What matters is when and why the extra text is used. An LLM-generated prefix belongs to R because search indexes it. The label does not depend on whether an LLM wrote the text. And metadata sitting unused next to a vector will not improve similarity; search has to embed it, index it for sparse retrieval, filter on it, or otherwise include it in scoring.

So if a good source span exists but will not match the query, I keep its offsets fixed and experiment with its representation. That tells me whether R was the problem without quietly changing B at the same time.

P: what reaches the answer-writing LLM?

Assume retrieval ranks this child span first:

The coverage ends earlier when the vehicle reaches 160,000 km.

Search has done its job. The final LLM still needs the preceding sentence to state the warranty correctly. At query time, the context builder can fetch the neighbour:

The traction battery is covered for eight years.
The coverage ends earlier when the vehicle reaches 160,000 km.

That is P. Neither the stored boundary nor the similarity score changed. The system added context after it had chosen the anchor.

INGESTION
document
  → B: create source spans
  → R: create searchable representations
  → build indexes

QUERY TIME
question
  → retrieve and rerank using R
  → P: expand, merge, deduplicate, and pack context
  → answer-writing LLM

In practice, P might add a neighbouring span, replace several children with their parent section, or merge overlapping offsets. Procedures usually need their original order. Expansion must also stop at version and permission boundaries.

More context is not automatically safer. If every 100-token child expands into a 2,000-token parent, the prompt fills with the same noise that small retrieval units were meant to avoid. I prefer to rerank first, expand only the surviving anchors, remove overlap by source offset, and then pack the result into the token budget.

When a correct anchor ranks well but the answer is still incomplete, I leave the index alone at first. Replaying the same ranked anchors through another packing policy is a cleaner test.

E: what did the experiment prove?

Suppose a fixed-window splitter and a structure-aware splitter both score 95% on a set of single-sentence fact questions. They are tied on that set. That is all I can safely say.

Production might contain tables, exceptions, comparisons, and multi-step procedures. The test has told me nothing about those cases. Structure-aware chunks may work better there. Fixed windows may be just as good and cheaper. Both may fail for unrelated reasons.

The papers do not give us one winner either. Dense X Retrieval reported strong proposition-level retrieval on its open-domain QA setup. Is Semantic Chunking Worth the Computational Cost? did not find consistent gains that paid for semantic chunking across its tasks. HiChunk argues that evidence-sparse RAG benchmarks can hide differences between chunking policies.

Those findings apply to the policies and workloads that the papers tested. My own evaluation set needs local facts, rules with exceptions, tables, procedures, and comparisons. It also needs questions that draw on several evidence spans, plus section-level and document-level questions. Some questions should have no answer in the corpus so that abstention is testable.

Source annotations matter as much as reference answers. A correct answer can hide bad retrieval if the model already knows the fact. When an answer is wrong, source offsets help locate the failure in B, R, P, or generation.

While debugging, I use this table:

This sequence has saved me from treating every retrieval problem as a request for a different token count.

Part II: one equation for each decision

I keep one equation for each decision. Any more than that belongs in the papers.

B: the candidate set

Let a document be a token sequence $d=(x_1,\ldots,x_n)$. A boundary policy with configuration $\theta$ produces:

Changing $\theta$ changes the candidates themselves. Candidate count, duplicated tokens, dense embeddings, BM25 length statistics, ingestion cost, and update cost change with it. A fair boundary experiment rebuilds every affected index.

For plain text, LangChain’s RecursiveCharacterTextSplitter is a reasonable baseline. Docling’s HybridChunker is more interesting for PDFs and manuals because it uses document hierarchy, tokenizer limits, headings, captions, and repeated table headers. RAGLite adds semantic boundaries over Markdown-oriented input. HiChunk explores learned hierarchical boundaries and auto-merge retrieval.

Recursive or structure-aware splitting is my baseline. Semantic or learned splitting has to fix a measured boundary failure and earn back its extra ingestion cost.

R: the indexed representation

Let $f$ be the embedding model. The representation can include context that is absent from the cited source span:

The source span $c_i$ stays put while the searchable form changes. This is why I test boundaries and representations separately.

Anthropic’s Contextual Retrieval is a concrete recipe for adding chunk-specific prefixes to dense and BM25 indexes. RAGLite implements segmented late chunking for supported local embedders and keeps multiple contextualized vectors for a chunk. Jina AI publishes a late-chunking reference implementation for models that expose token-level outputs. If one pooled vector still loses too much local detail, ColBERTv2 shows the late-interaction end of the design space, at a higher storage cost.

With a pooled embedding API, deterministic title and heading prefixes come first; I do not pay an LLM to write context for every chunk until that baseline fails. For a local embedder that exposes token vectors, the clean comparison is ordinary embeddings versus late chunking over identical source spans.

P: the context budget

Let $A$ be the reranked anchors and $p(c)$ the payload built around anchor $c$:

$B_{\mathrm{ctx}}$ is the generator’s context budget. The objective favors useful evidence and broader coverage while charging for repetition.

LlamaIndex’s HierarchicalNodeParser and AutoMergingRetriever implement the child-to-parent pattern directly. Leaf nodes are precise search anchors; related leaves can be replaced by a broader parent before synthesis. Its sentence-window and node-postprocessor patterns suit cases where a full parent is too much.

Whichever framework I use, I keep source offsets. They are far more dependable than string similarity for removing overlap, finding neighbours, preserving provenance, and enforcing version or permission boundaries.

E: evidence recall

Let $E_q$ be the source positions required to answer query $q$, and $U_k(q)$ the unique source positions covered by the top $k$ results:

Evidence recall tells me whether retrieval covered the answer. It needs a companion metric for density or redundancy. Retrieving the whole document can achieve perfect recall while burying the answer in irrelevant text.

Ragas includes context precision, context recall, faithfulness, and other RAG metrics, though source-offset metrics may need custom code. LangSmith can store datasets, trace retrieval and generation, and compare experiments even when the application itself does not use LangChain. HiCBench shows one way to build evidence-dense questions with hierarchical boundary annotations.

No single score settles a chunking experiment. I track evidence coverage and context precision, then inspect answer and citation correctness. I also log index size, ingestion cost, retrieved tokens, and latency. A quality gain that doubles the index or makes updates painfully slow may still be the wrong trade.

My starting point

For a new documentation or manual corpus, my first version would look like this:

Source documents
      ↓
Structure-aware parsing
      ↓
Small child spans with stable source offsets
      ↓
Title + heading path in the indexed representation
      ↓
Dense + BM25 candidate retrieval
      ↓
Reranking
      ↓
Conditional neighbour or parent expansion
      ↓
Source-offset deduplication and budgeted packing
      ↓
Answer-writing LLM with explicit citations

I preserve tables, procedures, code units, headings, lists, and clauses before enforcing the token cap. I store the exact source span separately from its indexed representation. Metadata prefixes come first; generated context or late chunking only enter the experiment if that baseline has a representation problem.

At query time, I retrieve small anchors, rerank them, and expand the survivors. Expansion cannot cross a document version or permission boundary. Any change to boundaries or indexed text triggers a rebuild of both dense and sparse indexes.

I split evaluation results by query type and keep source-level evidence labels. An advanced chunker earns its place by improving the quality-cost trade-off on that workload, not by sounding more sophisticated.

When the system fails, I now work from left to right: B, R, P, then E. That usually tells me what to test next. It is slower than changing 512 to 768 and hoping, but at least I can explain what changed.

References

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.
• Chen et al., Dense X Retrieval: What Retrieval Granularity Should We Use?.
• Günther et al., Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models.
• Superlinear AI, RAGLite: an open-source RAG toolkit combining semantic boundaries, segmented late chunking, contextual headings, and multivector chunk retrieval.
• Pierse et al., Late Chunking: Balancing Precision and Cost in Long Context Retrieval.
• Anthropic, Contextual Retrieval.
• Sarthi et al., RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval.
• Jiang et al., LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs.
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts.
• Zhong et al., Mix-of-Granularity: Optimize the Chunking Granularity for Retrieval-Augmented Generation.
• Wang et al., Document Segmentation Matters for Retrieval-Augmented Generation.
• Lu et al., HiChunk: Evaluating and Enhancing Retrieval Augmented Generation with Hierarchical Chunking.
• de Moura Júnior et al., Adaptive Chunking: Optimizing Chunking-Method Selection for RAG.
• Santhanam et al., ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction.

(Primary motivation: Rezende & Mohamed, 2015. :contentReference[oaicite:0]{index=0})

1. What KL Divergence Actually Is

KL divergence measures how far one probability distribution is from another:

A few properties I need to keep front-of-mind:

• Not symmetric: $D_{KL}(P|Q) \neq D_{KL}(Q|P)$.
• Not a “distance” metric (no triangle inequality).
• It is an expectation under $P$, meaning the direction strongly affects behavior.

The direction is not cosmetic: it changes optimization behavior (mode-covering vs mode-seeking).

2. Where My Confusion Came From: KL vs Cross-Entropy

I previously thought:

“KL term is basically cross-entropy.”

This is only conditionally true.

The key identity is:

where

• cross-entropy: $H(P,Q) = -\mathbb{E}_{x\sim P}[\log Q(x)]$
• entropy: $H(P) = -\mathbb{E}_{x\sim P}[\log P(x)]$

So if $P$ is fixed, then $H(P)$ is constant, and minimizing $D_{KL}(P|Q)$ is equivalent to minimizing cross-entropy $H(P,Q)$.

That is why in standard supervised classification with fixed labels, it feels like “cross-entropy = KL”.

The correction to my belief

The “KL term” is not intrinsically cross-entropy. Cross-entropy is what KL reduces to when the target distribution $P$ is fixed and we only optimize $Q$.

That “fixed target” assumption is exactly what breaks in many important ML objectives.

3. The VAE / Variational Inference Setting: KL is Not a “Label Loss”

Rezende & Mohamed frame variational inference as maximizing a lower bound on $\log p(x)$ (the evidence), because the true marginal likelihood is typically intractable. They write the ELBO as:

This is the core equation in the paper. :contentReference[oaicite:1]{index=1}

Here the KL is:

This is the conceptual point I had wrong: this KL is not comparing “predictions vs labels.”
It is comparing:

• $q_\phi(z \mid x)$: an approximate posterior produced by the inference network (encoder)
• $p(z)$: the prior

So the KL term is a regularizer / information constraint: it prevents the inference network from encoding arbitrary information in $z$ and pushes the posterior toward the prior.

Why my cross-entropy intuition fails here

In classification, $P$ is typically fixed (true labels).
In the ELBO, $q_\phi(z \mid x)$ is learned and changes during training.

So even though the identity $D_{KL} = H(P,Q) - H(P)$ always holds mathematically, the “$H(P)$ is constant” trick is not something I can rely on in intuition. The moving part $q_\phi$ is exactly what I’m optimizing.

4. Another Important Subtlety: KL Direction Matters

The ELBO uses:

This is the “reverse” direction relative to what I mentally associate with supervised learning (which usually resembles $D_{KL}(P|Q)$ with fixed $P$).

This direction has practical consequences:

• $D_{KL}(P|Q)$ (“forward KL”) strongly penalizes putting low probability mass where $P$ has mass → tends to be mode-covering.
• $D_{KL}(Q|P)$ (“reverse KL”) strongly penalizes putting mass where $P$ has low mass → tends to be mode-seeking.

In variational inference, this asymmetry is part of why approximate posteriors can miss modes (the “mode-seeking” behavior is a known limitation). Normalizing flows in Rezende & Mohamed are partly motivated by making $q_\phi(z \mid x)$ flexible enough to reduce that approximation gap. :contentReference[oaicite:2]{index=2}

5. Why Normalizing Flows Make the KL Term Even More Central

Rezende & Mohamed propose constructing the approximate posterior by transforming a simple base distribution through a sequence of invertible mappings (“normalizing flow”):

• start: $z_0 \sim q_0(z_0 \mid x)$ (often Gaussian)
• transform: $z_k = f_k(z_{k-1})$ for invertible $f_k$
• end: $z_K$ has a more complex distribution $q_K(z_K \mid x)$

Because the transform is invertible, the density changes via change-of-variables:

This flexibility makes the approximate posterior richer, which tightens the variational bound. This is one of the main contributions of the paper. :contentReference[oaicite:3]{index=3}

My updated intuition

The KL term is not an “annoying extra penalty.” It is the explicit mismatch measure between what my inference model can represent ($q_\phi$) and what the generative model assumes ($p$).
Normalizing flows are an upgrade to $q_\phi$ so that this mismatch can be reduced without sacrificing scalability.

6. When the “Target Distribution is Not Fixed” Happens in Practice

This directly answers my earlier confusion: when is $P$ not fixed?

In variational inference and flows, the “target” distribution in a KL can involve learnable distributions such as \(q_\phi(z \mid x)\) or teacher/student distributions that change over time. Concrete examples:

• Variational inference / VAE: \(q_\phi(z \mid x)\) is learned.
• Normalizing flows in VI: the whole posterior family is learned through transformations.
• Online distillation / EMA teachers: teacher distributions evolve during training.
• RL: policies and visitation distributions shift.

This is the world where “KL term = cross-entropy” stops being a reliable mental shortcut.

7. The Corrected Takeaway I Want to Keep

My previous belief:

• “KL is basically cross-entropy.”

What I now believe:

• KL is a distribution mismatch measure.
• Cross-entropy is one special case view of KL when the target distribution is fixed.
• In variational inference (including normalizing flows), the KL term is the core regularizer that shapes the posterior approximation, and its direction matters.

If I keep that in mind, the ELBO decomposition in Rezende & Mohamed stops looking like “reconstruction loss + random KL penalty” and starts looking like what it really is:

a likelihood-fitting term plus an explicit constraint on how much the posterior is allowed to deviate from the prior, with flows making that posterior expressive enough to be useful. :contentReference[oaicite:4]{index=4}

This post is my corrected summary, written as a set of “mis-beliefs → corrections,” grounded in the variational inference framing emphasized in the paper.

1) My first mis-belief: “Decoder is the posterior”

What I believed

The decoder is the posterior.

What’s actually true

In VAE terminology:

• The posterior is $p_\theta(z \mid x)$.
• But it is usually intractable in deep generative models.
• So VAEs introduce a tractable approximation $q_\phi(z \mid x)$.

That means:

• Encoder $\approx$ approximate posterior (inference network):
  $q_\phi(z \mid x) \approx p_\theta(z \mid x)$

• Decoder $\approx$ likelihood / generative model:
  $p_\theta(x \mid z)$

So the decoder is not “the posterior.” The decoder is one of the ingredients used to define the posterior (via Bayes’ rule), but the direction is the opposite:

• Decoder: $z \rightarrow x$ (generate)
• Encoder: $x \rightarrow z$ (infer)

This mapping is consistent with the paper’s emphasis: variational inference replaces posterior inference with optimization over a variational family, where $q_\phi(z\mid x)$ is the variational distribution and is learned to match $p_\theta(z\mid x)$.

2) My second mis-belief: “Encoder compresses, decoder expands”

What I believed

Encoder compresses a pipeline; decoder expands a pipeline. Therefore, VAE is a fancy compress–decompress trick.

What’s actually true (and why this confusion happens)

This “compress / expand” idea comes from image VAEs, where:

• $z$ is low-dimensional,
• $x$ is a high-dimensional image,
• and the decoder visually looks like an upsampling network.

But for forecasting (e.g., predicting yield as a scalar), “expand” is not a meaningful concept:

• The target $y$ is often 1-dimensional.
• The decoder does not necessarily “expand”; it often maps $(x, z)$ to a scalar distribution.

So the right mental model is:

The encoder and decoder are not defined by dimensionality changes.
They are defined by probabilistic roles in variational inference.

• Encoder: a recognition model for approximate posterior inference.
• Decoder: a generative model defining the likelihood.

3) What “VAE inference” really means (and why “generative inference” confused me)

My confusion

The phrase “generative inference” didn’t make sense to me, because inference should mean “estimate hidden state,” not “generate outputs.”

The correction

In probabilistic modeling, “inference” usually means:
compute or approximate the posterior over latent variables.

In a VAE, the true posterior is:

But VAEs do not compute this directly. Instead, they learn:

So:

• VAE inference = run the encoder to obtain $q_\phi(z \mid x)$ (or its parameters).
• Generation / sampling = sample $z \sim p(z)$ (or a conditional prior) and decode $x \sim p_\theta(x\mid z)$.

These are different operations. Inference is “backward reasoning.” Generation is “forward simulation.”

The paper’s message is basically:

we make inference scalable by turning it into an optimization problem, and then amortizing it with a neural network.

4) Strawberry yield forecasting: how I map “state vs observation” properly

Now I apply this to a practical problem:

• I observe temperature and fruit count.
• I want to forecast strawberry yield.

Observed variables (measurements)

Let:

• $T_{1:t}$ = temperature history up to time $t$
• $C_{1:t}$ = fruit_count history up to time $t$
• $Y_t$ = yield (today, or at harvest)

I’ll group the observed covariates as $x$:

Latent state (unobserved but important)

A forecasting model often benefits from a hidden state representing things I don’t measure well:

• plant vigor
• stress (heat / water / disease)
• phenological stage
• cultivar or management effects
• microclimate differences

Call that latent crop condition $Z_t$.

So, in Bayesian terms, what I actually want is:

i.e., “given observed data, what crop states are plausible?”

That is exactly a posterior.

5) What the decoder becomes in this forecasting case

In a forecasting-oriented VAE (more precisely, a conditional VAE), the decoder is a probabilistic forecast model:

This is the “forward” story:

• if the crop state is $z$
• and covariates are $x$
• then yield $Y$ follows some distribution.

For example, the decoder might output $(\mu_\theta(x,z), \sigma_\theta(x,z))$ for a Gaussian yield distribution.

So yes: the decoder is the forecasting component.
But it is not “expanding” by default—it is defining a likelihood.

6) Then what is the encoder in this forecasting case?

During training, I have both covariates and yield, so I can infer what latent crop state best explains the outcome:

Interpretation:

“Given sensors and realized yield, what hidden crop condition must have been present?”

That’s posterior inference (approximate), and it is the core meaning of “VAE inference.”

This is aligned with Rezende & Mohamed’s viewpoint:
the inference model $q_\phi$ is trained to approximate the true posterior while keeping optimization tractable.

7) The forecasting-time detail I initially missed: I need a conditional prior

A key practical point:

At prediction time, I do not know $y$, so I cannot directly use $q_\phi(z \mid x, y)$.

To forecast, I need a distribution over latent states given only covariates:

sometimes called a conditional prior.

Then forecasting is:

1) Sample latent crop states: $z^{(k)} \sim p_\psi(z \mid x)$
2) Decode yields: $y^{(k)} \sim p_\theta(y \mid x, z^{(k)})$
3) Aggregate samples to get a predictive distribution.

This is how the model produces:

• a mean forecast
• prediction intervals
• multi-modal outcomes (if relevant)

8) Comparing this to LightGBM (the baseline that keeps me honest)

LightGBM is typically:

a direct mapping from engineered covariates to yield.

A VAE-style forecast is instead:

This difference matters when:

• there are hidden factors not captured by $x$,
• the same $x$ maps to multiple plausible yields,
• or uncertainty calibration is important (operations planning, labor scheduling, contracts).

If none of those are true, LGBM is usually the better engineering choice.

9) My corrected takeaways (the short list)

What I used to think

• Encoder compresses
• Decoder expands
• Decoder is the posterior
• “Generative inference” means predicting

What I now think

• Encoder is the approximate posterior: $q_\phi(z \mid \cdot)$
• Decoder is the likelihood / generative model: $p_\theta(\cdot \mid z)$
• In forecasting, decoder is best seen as a probabilistic forecaster, not an “expander”
• “VAE inference” means latent state inference via the encoder
• For forecasting, I often need a conditional prior $p_\psi(z \mid x)$ to sample latent states when $y$ is unknown

10) One sentence that finally fixed my mental model

A VAE-style forecaster for strawberry yield is:

a model that learns a latent crop-condition variable $z$ and uses variational inference (via an encoder) to approximate the posterior over $z$, while a decoder defines a probabilistic forecast $p_\theta(y \mid x, z)$ that can be sampled for uncertainty-aware predictions.

This framing made the Rezende & Mohamed (2015) motivation click:
variational inference is the mathematical reason VAEs exist, and “encoder/decoder” are just neural parameterizations of the variational posterior and the likelihood.