It all started innocently enough. You just wanted to read a novel.
Then the cast kept growing, and you started forgetting the clues planted earlier.
So you thought: let's build a language model! (Well, no.)
Classifying text, deciding whether two sentences are related, and finding an answer in a passage can share the same learning starting point. BERT, released by Google in October 2018, was an important text-understanding model along this path, using a Transformer encoder to turn text into representations that different tasks can use. BERT paper
The process first pretrains on large amounts of unlabeled text and then fine-tunes for a specific task. For classification, attach a classification output layer; for reading comprehension, predict where the answer appears in the passage. During fine-tuning, both the task layer and BERT's weights are updated. Different tasks can start from the same pretrained weights and train their own models separately. BERT §3, Figure 1
GPT had already used Transformer pretraining and fine-tuning before BERT. BERT's key contribution was bidirectional pretraining, letting text representations use context on both sides. Its paper reported state-of-the-art results at the time on eleven natural language processing tasks. GPT paper §3, BERT abstract and §1
With that foundation ready, give standard BERT a reasonably substantial document, and an engineer soon finds that the 512 tokens are gone before the document is finished.
Later, model specifications began listing 32K, 128K, a million; research reports even reached ten million. The numbers kept growing, as though the original problem was just someone forgetting to add a few zeros to a configuration file.
If it were that easy, GPU vendors would probably have sold a lot fewer graphics cards.
From 512 to a million, sequence length grows by roughly 1,953 times. If we simply raise the limit, how do the positional parameters, attention computation, and training data keep up?
Before adding those zeros, we need to find out where the original 512-token door is locked.
Where exactly is that 512-token door locked?
Do 512 tokens mean 512 characters? That depends on how the tokenizer splits the text.
A token is a unit produced by a tokenizer: it might be a character, part of a word, or punctuation. Different tokenizers can split the same text differently, so 512 tokens cannot be converted into a fixed number of Chinese characters.
The range of tokens a model can refer to in a given computation is usually called its context window. For a generative model, the prompt, attached documents, and text already generated may all occupy this window. Products may also impose separate input and output limits.
After text is split into token IDs, the model maps each ID to a sequence of numbers called a token embedding. Those numbers describe content; positional information needs to be supplied separately.
BERT's input needs position as well as content. Its standard implementation prepares a learned positional embedding table, in which position 0 has one vector, position 1 has another, and so on. The original configuration provides 512 positions, ending at index 511. BERT §3, Official modeling.py
Then token number 513 arrives. It needs a vector for position 512, but the original table has no such row.
Token 513 needs index 512. The original table ends at index 511.
Special symbols such as [CLS] and [SEP] take up positions too. The sign at the door says “capacity: 512,” but the staff have already occupied a few seats.
How will those extra positions learn their jobs?
Engineers can indeed enlarge the table, initialize new positions, and retrain or fine-tune. Those new parameter rows have yet to learn how to work; we are changing both the available positions and the range of data the model must handle. The length limit has changed; the model still needs to adapt to the new positions.
The BERT paper also notes that 90% of its pretraining steps used sequences of length 128 and only the final 10% used 512. One reason was the high attention cost of long sequences. BERT Appendix A.2
So the 512 limit comes from positions and training, and also from computation and memory. Let's sort out the addresses first, then return to the graphics cards.
BERT reads text and produces representations with a bidirectional encoder; common generative LLMs use a decoder with a causal mask to generate from preceding text. Both face the positional and computational problems of long sequences. The long-context models ahead have their own architectures and training origins.
Without a position table, how do we keep the addresses?
Consider two sentences: “The box contains the key” and “The key contains the box.” They use the same words, but what contains what has reversed. That second key must be impressively large.
For self-attention without positional signals or a causal mask, permuting the inputs permutes the corresponding outputs. This is called permutation equivariance, in which content-based computation follows the content while the model still lacks an extra clue telling it “where this word originally sat.” A generative model's causal mask supplies a direction of visibility; we set that aside here to examine positional signals on their own. Transformer §3.2–3.5
| p = 0 | p = 1 | p = 2 | |
|---|---|---|---|
| S1 | BoxA — | holdsB — | KeyC — |
| S2 | KeyC — | holdsB — | BoxA — |
Input illustration: box holds key, versus key holds box. Content vectors A, B, C follow the same words. Adding position gives the box A+P0 in the first sentence and A+P2 in the second. Each phrase is one illustrative position; actual tokenization may be finer.
The diagram treats the three phrases as three cards; actual tokenization may be finer. The box's content vector is A. Move it to the other end, and it is still A. Once combined with a position vector, however, it becomes A + P₀ in the first sentence and A + P₂ in the second. The same content now has a different location.
That is the basic idea of additive absolute positional encoding. The token embedding and position vector have the same dimension; we add them before feeding them into the model:
Once position vectors are mixed into the input, the model learns to use those signals through training. We have not designated a particular dimension as the page-number drawer.
BERT stores the vectors for individual positions in a learned table. To avoid learning positional parameters row by row, we need another way to generate them.
The 2017 Transformer, which predates BERT, already generated position vectors directly using sines and cosines at different frequencies. Transformer §3.5
Why turn one position into several waves?
Positions are simply 0, 1, 2. The most obvious idea is to feed in that integer.
A single index supplies one number. We also want signals that make it easy for the model to use changes between neighboring positions and distances at different scales. Sinusoidal encoding feeds the same position into a set of functions that change at different speeds.
Start with one pair, . Here p is the position, and ω determines how far the phase advances for one step. Think of phase as where we are along the waveform; an advance of 2π radians completes a cycle. A larger ω changes the curve faster; a smaller one changes it more slowly. The same p yields different values at different frequencies.
- [0]
- 0.00
- [1]
- 1.00
- [2]
- 0.00
- [3]
- 1.00
Eight-dimensional sinusoidal example, showing only dimensions 0–3. Frequencies 1 and 0.1 produce sine and cosine pairs at positions 0–7; the remaining four dimensions are omitted. Values are calculated and rounded to two decimals.
By the time the cursor crosses this range, the fast wave has completed a full cycle, while the slow wave has moved only a little. The four numbers below are the sin and cos values sampled from two frequency pairs and arranged into a vector. This small example uses and displays only the first four dimensions; the other four still belong to the full encoding.
The original paper groups dimensions in pairs:
Here i indexes the pairs starting at 0, and is the total dimension of the input position vector. The eight-dimensional example has four pairs, with frequencies 1, 0.1, 0.01, and 0.001; the diagram displays the first two. There is still one position p; we now describe it at several scales. Transformer §3.5
Why give sin a cos companion? A value on a single waveform can occur at many phases; a pair can describe phase together. More usefully, the angle-addition identities mean that shifting all positions by a fixed number of steps updates each sin/cos pair through the same two-dimensional linear transformation. This gives the model a structure it can use for relative displacement. Transformer §3.5
Once position is added to content and projected, which positional relationships remain in the attention scores?
Knowing the page number versus knowing the distance
The model is looking for the key in the novel's last chapter. “The answer is on page 3” gives an absolute position; “the answer is 20 pages before me” gives a relative position. Insert a preface before the story and all page numbers change, while the distance between two clues can stay the same.
First, meet attention's three characters. Three learned projections turn each position's hidden vector into a Query, Key, and Value:
- Query (Q): the vector this position uses to compare with other positions.
- Key (K): the vector another position offers for that comparison.
- Value (V): the content vector combined using the resulting weights.
For now, imagine the final question as Q, the clue markers in each passage as K, and the related content as V. In practice these are numerical vectors learned by the model, with no hand-filled “key location” field. Q and K produce a score through a dot product, which multiplies corresponding entries and sums them. Multi-head attention uses several sets of projections and comparisons; we will follow just one head. Transformer §3.2
If we add position to content first, what gets mixed into the Q–K dot product? Set the projections to the identity for now and use just one number, with content 2 plus position 1 on one side and content 3 plus position 4 on the other.
| xn = 3 | Pn = 4 | |
|---|---|---|
| xm = 2 | 2 × 36 | 2 × 48 |
| Pm = 1 | 1 × 33 | 1 × 44 |
After mixing, the dot product is . Expanding it gives four pieces: 6 for content–content, 8 for content–position, 3 for position–content, and 4 for position–position.
A real model also has projection matrices. Writing their composition as gives the same four types of interaction:
So the properties of the position vectors alone tell only part of the story. Actual Q–K scores also include interactions between content and position. This expansion helps later research ask which terms should carry positional information, and how to represent it. RoFormer §2.2–2.3, Equation 6
Su Jianlin's “Transformer Positional Encoding: A Headache for Researchers” (2021) surveys these design perspectives and makes useful further reading. For more positional designs, follow the original works it cites.
Could we arrange things so that the positional change in a comparison enters its score directly through relative displacement?
RoFormer introduced Rotary Position Embedding, or RoPE, in 2021. It first obtains Q and K, groups their dimensions into pairs, then rotates each two-dimensional vector according to position. These pairs belong to the dimensions actually rotated within each head; keep them separate from the input position-vector dimension discussed earlier. RoFormer §3.1–3.2
Let denote a two-dimensional rotation. For a pair with frequency ω, position m rotates by mω and position n by nω. Fix the projected content vectors q and k and examine only this rotation layer. Rotating both vectors back by mω preserves their dot product: q returns to its original direction, while k retains an angle of (n−m)ω:
The right-hand side contains . Shifting both positions together preserves this difference and the relative relationship introduced by rotation. Content q and k still participate in the score. RoFormer §3.2, Equations 13–16
Fixed q=k=(0,1). Positions move together with gap 2. Frequencies 1 and 0.1 keep dot products cos(2)≈−0.416 and cos(0.2)≈0.980. This isolates rotary position effects on fixed content vectors.
The diagram fixes , so both hands have equal length. The positions move together from 0/2 to 2/4 and 4/6, always two steps apart. For frequency 1, the rotated dot product stays at ; for frequency 0.1, it stays at . Absolute positions change, but each pair's comparison value stays the same. Here we isolate rotation with fixed content; the full model also updates content representations.
The earlier additive encoding adds position vectors to the input. RoPE turns positions into rotation angles for Q and K. Both provide positional signals, with different mathematical forms and entry points into the computation.
Position 100,000 can also be inserted into the rotation formula. The model then meets combinations of positions it did not become familiar with during training.
The hands keep turning into unfamiliar territory
Suppose a model was originally trained only on sequences of length 2,048. It has seen positional combinations, relative distances, and the attention scores those positions form together with text content within that range.
Now feed it 32,768 tokens directly. The rotation formula still runs, but the combinations of distances and angles go beyond its original training range.
This is length extrapolation. Directly extending an existing RoPE model can noticeably hurt quality. RoPE's relative-position form provides a useful structure; the quality of extrapolation still needs methods and experiments to establish it.
In 2023, Position Interpolation (PI) chose an approach:
If the model knows the old coordinate range, squeeze the new positions back into it.
For original position and extension factor , the position sent to RoPE becomes:
For example, start with four positions, 0, 1, 2, 3, and make room for eight tokens. Direct extension uses 0 through 7. Interpolating by a factor of two instead uses 0, 0.5, 1, 1.5, all the way to 3.5.
Wait, 3.5 is still larger than the old last position, 3. The factor here is based on window lengths, so 8 divided by 4 gives 2 and the scaled coordinates lie in . Original training sampled only the integers 0, 1, 2, 3. PI adds intermediate ticks and puts the final point between 3 and 4. The model still needs to adapt to these denser positions.
All eight tokens remain. Scaling by 2 halves adjacent distances; the last coordinate is 3.5, beyond the last training sample at 3.
Does squeezing the coordinates merge two tokens into one?
All eight tokens remain in the diagram. What changes is the coordinate sent to the positional encoding. Positions become denser, but the novel's text remains intact, and attention still processes eight positions.
Now look at Figure 1 from the paper. The upper left shows the original 2,048 range; the red points on the right extend straight into unfamiliar positions. The lower plot compresses positional samples from the new sequence back into the old range. This illustrates positional scaling, with a schematic RoPE waveform on the vertical axis. PI Figure 1
This arrangement makes adapting to longer sequences easier. PI §3.1 uses 1,000 fine-tuning steps by default. In Tables 1–2, LLaMA 7B/13B extend from 2,048 to 32,768; the reported results reach 16,384 for 33B and 8,192 for 65B. PI §3.1–3.2, Tables 1–2
Around the same time, the llama.cpp community was testing RoPE positional scaling, sharing code changes and perplexity comparisons. The PI paper cites this discussion. llama.cpp discussion, June 2023
But fitting eight positions onto a four-position scale also reduces the distance between neighbors.
If we squeeze a million or more positions into a range of 2,048, what happens to the ticks that used to distinguish local differences?
A compressed map still needs local detail
PI applies the same scaling factor to every rotary frequency.
But each RoPE frequency originally handles positional changes at a different scale. Slow down the fast second hand as well, and the short-distance differences familiar to the model change too.
How can we extend the long-distance range while preserving local resolution?
Writing out the RoPE frequencies helps clarify the difference. A common form is , where b is the base, d the rotary-vector dimension, and i the two-dimensional pair index. PI divides every pair's angle by s; changing the base b instead changes frequencies by different amounts for different i.
For example, at i = 0 the frequency is 1, unaffected by the base. Later frequencies do change with it. That leaves room to disturb the fastest changes less and assign more of the extension to slower components. The NTK-aware approach discussed in YaRN follows this nonuniform scaling idea. YaRN §3.1, Appendix A.2
In 2023, YaRN broke the problem down further. It examines how uniform interpolation affects high-frequency information and local relative distances, then treats frequency regions differently: largely preserve local variation at high frequencies, interpolate low frequencies, and connect them with a smooth transition. It also adjusts attention-score scaling. YaRN §3
YaRN also traces community explorations including NTK-aware and NTK-by-parts. These scaling approaches all try to extend the long-distance range while disturbing useful local positional information as little as possible.
In 2024, LongRoPE went further and asked whether, beyond differences in frequency, the positions at the start of a sequence should follow exactly the same scaling rule.
Its experiments found two kinds of nonuniformity, with suitable scaling varying across RoPE dimensions and some initial sequence positions benefiting from separate treatment. The method searches both scaling factors and the initial range whose positions are retained, then combines this with progressive extension. LongRoPE §2–3
LongRoPE seeds its search with scaling schemes corresponding to PI, NTK, and YaRN, then repeatedly compares configurations by perplexity through mutation, crossover, and selection. Perplexity measures the model's next-token prediction probabilities. This search adjusts positional configurations; weight fine-tuning takes place in the progressive extension procedure. LongRoPE §3.2, Algorithm 1
- PIUniform position scaling
- LLaMA · 1,000 steps
7B/13B → 32K
33B → 16K · 65B → 8K - YaRNScale by frequency range
- Llama 2 7B/13B · 64K training
400 + 200 steps → 128K - LongRoPESearch, then extend
- LLaMA2-7B · 400 + 600 steps
128K → 256K → 2048K
PI: Tables 1–2 · YaRN: §4 · LongRoPE: §4
One LLaMA2-7B procedure in the paper trains at 128K for 400 steps, then at 256K for 600 steps, and subsequently searches positions on the fine-tuned model to extend to 2048K. That final stage does not fine-tune weights on two-million-token sequences. The paper uses , so 2048K means 2,097,152 tokens. LongRoPE §3.3–4.1, Table 6
Read the step counts alongside their costs. Appendix A.2 records eight A100s for one week and sixteen A100s for two weeks for those two fine-tuning stages, respectively. Positional search incurs additional computation. LongRoPE Appendix A.2–A.3
The longest sequence used in training can differ from the longest sequence tested after extension. On seeing “supports two million,” look next for the extension settings, the tasks evaluated, and results at each length.
Positional methods have other branches too. ALiBi adds a linear distance bias before softmax, where each head uses a fixed, predefined slope and more distant positions are penalized more heavily. PI, YaRN, and LongRoPE here address existing models that use RoPE. ALiBi §3, Figure 3
The coordinates are arranged, but the novel still contains a million tokens. PI has deleted no text, and RoPE has eliminated none of the full pairwise comparisons.
The coordinates work. Why is the GPU still out of memory?
Return to the question at the end of the novel. This position takes its Q and compares it with the K at each earlier position to produce scores. Softmax turns the scores into weights, which combine the values V.
A head usually computes Q–K dot products and divides by . Here d is the Key-vector dimension, used to scale the scores. Transformer §3.2.1
With just four candidate positions, write this row of scaled scores as . Each position's weight is:
Suppose the relative weights after exponentiation are 1, 1, 2, 4 and the content values are 10, 20, 30, 40. First sum the weighted content into u, then divide by the total weight to obtain the output:
| K | s | exp(s) | α | V | αV |
|---|---|---|---|---|---|
| K1 | 0 | 1 | 0.125 | 10 | 1.25 |
| K2 | 0 | 1 | 0.125 | 20 | 2.5 |
| K3 | ln 2 | 2 | 0.25 | 30 | 7.5 |
| K4 | ln 4 | 4 | 0.5 | 40 | 20 |
| Σ | 8 | 1 | — | ||
- s
- Compare scores
- α
- Normalize weights
- αV
- Weight the values
α = exp(s) / Σ exp(s)
The table keeps every step visible from left to right. Exponentiating scores s gives 1, 1, 2, 4; dividing by their sum, 8, gives normalized weights α; multiplying by each V gives 1.25, 2.5, 7.5, 20. Sum the final column and the answer is 31.25.
These are teaching numbers for this article. Actual values V are vectors; we use one component first to make the arithmetic clear. We will reuse these numbers when we discuss blocks.
One Q against four Ks is easy. When processing the whole input, though, every position has its own Q to compare with other positions.
For sequence length , one head's full score matrix contains entries. Q and K have shape ; their dot products form . Multiplying by V of shape then produces the combined result for each position. Transformer §3.2, Table 1
| Tokens | FP16 |
|---|---|
| 512 | |
| 32,768 | |
| 131,072 | |
| 1,000,000 |
Memory table: one layer, one head, FP16, full score matrix.
In the small case, 4 positions give 4 × 4 comparisons, or 16 cells. Increase the length to 8 and both axes double; the old 16 cells occupy only a quarter of the new matrix. The table on the right converts real lengths into memory usage, rather than trying to cram a million positions into a tiny diagram.
At a million tokens, one layer, one attention head, and one full FP16 score matrix alone need bytes: 2 TB, about 1.82 TiB.
That excludes model weights, Q/K/V, intermediate results in other layers, training gradients, and optimizer states. Just one of those large intermediate tables already needs 2 TB.
From 512 to a million, length grows roughly 1,953 times, but pair counts grow about 3.81 million times. Extending a position table and growing an attention matrix therefore incur costs on very different scales.
Generative models use a causal mask to block future positions, making visible pairs triangular. Pair counts still grow quadratically with N. Kernels can skip work in masked regions, while the quadratic growth remains.
What do fewer connections save?
This is a reasonable direction. Most sentences in a novel first depend on nearby text; comparing every token with landscape descriptions hundreds of thousands of words away may be a poor use of resources.
In 2020, Longformer connected positions mainly to a nearby local window, giving a small number of designated positions global connections. BigBird combined local, global, and random connections to provide additional paths for information across regions. Longformer §3, BigBird §2
With full comparisons, the Query in the orange row can reach every Key. With local plus global connections, it mainly sees nearby positions and global positions. Extra sparse connections bring in a few distant points too. For the same Query, the diagram's connection count changes from 16 to 6, then to 8.
This is a teaching illustration of connection rules. Actual windows, global positions, random blocks, and masks must follow each paper's settings.
Global positions read the entire sequence and can be read by other positions. When local-window width, the number of global positions, and the number of extra random connections per nonglobal position stay fixed, total pair counts grow linearly with sequence length. Each global position still has N connections; since their number stays fixed, their combined cost is also linear.
In exchange, the final question now depends on retained direct links or paths through several layers to reach the drawer in chapter one. The model needs training for this connection structure, and tasks must check whether it can obtain the information it needs.
Another direction is a relay between segments. Transformer-XL, from 2019, retains hidden states from the previous segment so that the next can refer back to it, using relative positional designs to handle positions across segments. Transformer-XL §3
This creates continuity between segments, within the capacity of the retained states and available memory. For our novel, we need to check whether chapter one's clue remains directly available in the current state or must pass through several steps to affect the final answer.
Long-sequence models can therefore advance by changing connections, retaining states, and other approaches. Each has its own information paths; evaluation must examine the actual mechanism.
Suppose, though, that we want to keep full attention, letting the novel's ending score every preceding position directly. Is there somewhere else to save?
Researchers then shifted their attention from “how often do we compare pairs?” to “how often do we move data?”
See the whole input without storing the whole table
A straightforward attention implementation computes , stores the score matrix, applies softmax, stores the weights, and multiplies by V. The steps map neatly onto the textbook formula.
But does a formula containing an intermediate matrix require the implementation to hold that entire matrix in memory at once?
In 2022, FlashAttention tackled precisely this point. It considers the GPU memory hierarchy, made up of larger-capacity HBM and smaller, faster on-chip SRAM. Where data is read, computed on, and written back all affect actual speed. FlashAttention §2–3, NeurIPS 2022 version
Imagine packing the novel into boxes and bringing in only the Q, K, and V blocks needed for each calculation. If each completed block leaves enough statistics to assemble the full result later, we can avoid materializing the entire table.
But the softmax denominator sums over all candidate positions. Halfway through the book, we do not yet know how important the second half will be.
Return to our earlier numbers, with weights 1, 1, 2, 4 and content 10, 20, 30, 40.
The first block reads the first two positions and retains two quantities:
The second block retains:
Finally, add the numerators and add the denominators:
The result matches processing all four positions at once.
Both blocks read: (30 + 220) / (2 + 6) = 31.25, matching the complete calculation.
The diagram fixes one row of Q, calling the numerator u and denominator ℓ. Blue KV₁ moves from HBM into SRAM and produces 30 and 2. The accumulators remain, the working block is released, and orange KV₂ arrives. It produces 220 and 6, combining into 250/8 = 31.25. We track only one Q row here; the full layer also schedules reads and writes for other Q blocks and statistics.
Could each block calculate its own average first, and then we average those two answers?
Try it and a different result appears:
That answer is wrong. The second block has total weight 6, while the first has only 2. Each block cannot get half the vote.
Merging must retain each block's total weight to assign its contribution to the output correctly.
Real FlashAttention must also handle numerical stability. Direct exponentiation can overflow for large scores, so it maintains the largest score seen so far. When a new block brings a larger maximum, the old numerator and denominator are rescaled before accumulation continues. FlashAttention Algorithm 1
For each Q row, what must remain is this set of updatable statistics and the output accumulator vector. Once computed, the full score matrix can be discarded; recomputation during backpropagation further reduces intermediate storage.
This preserves the mathematical result of full attention, subject to floating-point error in actual arithmetic. It improves memory use, HBM traffic, and kernel efficiency; full pairwise computation remains .
FlashAttention-2 later improved GPU work partitioning and parallelism, reducing some non-matrix-multiplication work to use the hardware more efficiently. FlashAttention-2 §3
Even after removing the score table, a million tokens' Q, K, V, outputs, and other layers' intermediates remain large, and quadratic pairwise work remains substantial. Next, the work must be divided across a larger system.
Open the technical notes & sources
To check how merging works when the maximum changes, write a block's statistics as , , and .
To merge two blocks, first take , then calculate:
The final output is . If the second block in our example brings a maximum score of , merging gives numerator 62.5 and denominator 2, still yielding 31.25. Actual kernels process multiple rows and vector dimensions together; here we isolate one row and one output component.
When one GPU cannot keep up, share the work
First, look at what else can be saved within a single device.
A Transformer has a feed-forward network (FFN) after attention. The FFN can process each position separately, but usually expands the feature dimension internally. Retaining intermediate features for an entire long sequence at once still puts considerable pressure on memory.
Blockwise Parallel Transformer (BPT) joins blockwise attention and the feed-forward network into a more complete blockwise pipeline. Once certain attention-output blocks are ready, their corresponding FFN computation follows. Together with recomputation and related scheduling, this reduces the need to retain intermediate features for the entire sequence. BPT §3, Figure 2
FlashAttention addressed memory use within an attention layer; BPT extends the view to the layer formed by attention and the FFN together.
Next, a single device still has limited capacity and speed. So divide the sequence across several devices.
Suppose the novel is split into three large parts, A, B, C, each held by one GPU. GPU A has Q for its own segment, but retaining full attention still requires comparing it with K and V from the other segments.
The obvious solution is to collect all the other segments. But then every device must again store a large copy of K and V. After all that division of labor, the same novel ends up piled on everyone's desk.
Ring Attention extends BPT's blockwise approach. Each device keeps its own Q, while K and V move station by station around a ring. Computation on the current block can overlap transfer of the next. When one block finishes, the next is processed and its attention results accumulated. Ring Attention §3
Three-device illustration. Each device keeps its Q block and visits every KV block across three rounds; the real algorithm also overlaps transfers and computation.
Q stays on its device while the green K/V blocks circulate around the ring. Over three rounds, every device encounters all blocks in turn. This illustrates the transfer schedule; generative models still apply a causal mask to process only the positions each Q may actually reference.
Why is FlashAttention's blockwise accumulation useful here? When a device receives a new KV block, it can update the same statistics and merge the results without collecting every KV block simultaneously.
The speedup from several GPUs depends on whether a block's computation can hide its transfer time. If computation is too short or the network too slow, devices spend time waiting for data. Ring Attention's communication overlap therefore depends on block size, device count, bandwidth, and hardware compute performance together.
Distributing sequence storage and overlapping communication with computation lets devices cooperate on longer inputs. They still need to perform all the computation for allowed pairs between them.
Once the system can handle sequences this long, what should the training data teach the model?
The machine can run it. Has the model practiced?
Start with an obvious training-data recipe, concatenating ten thousand unrelated short articles into one very long sequence.
The length is there, and the positions reach far enough. Next ask what clues the data actually offers for predicting later tokens. If most answers come from a few hundred nearby characters, the model has plenty of opportunities to solve the task with local information.
Looking back from the last chapter to the first and tracking characters across chapters require the data and training objective to provide that kind of practice. Document masks and concatenation schemes also affect which positions can exchange information.
Long-context research therefore also needs training lengths, long-document data, and adaptation procedures. Positional interpolation helps the model connect to new coordinates; long-sequence training lets it actually learn at those positions.
We have already seen LongRoPE search for longer positions after fine-tuning weights on shorter sequences. Different methods make different arrangements for extrapolation and data; experiments must establish their extended performance.
In 2024, Large World Model (LWM) applied these system techniques to long-text and audiovisual models. Let's focus on how its text model, LWM-Text, extends context.
Initialized from LLaMA-2 7B, it uses Ring Attention for distributed long-sequence processing and Pallas, a tool for writing accelerator kernels, to fuse RingAttention with FlashAttention. It increases the RoPE base as length grows. This training setup actually used TPU v4-512. LWM §3.1, Table 1
The authors split LWM-Text pretraining into five stages of 32K, 128K, 256K, 512K, and 1M. Each uses Books3 books filtered by length and initializes from the preceding stage's model. Lengths in Table 1 are powers of two, so the final 1M is 1,048,576 tokens. Later memory examples in this article use a round one million instead. LWM §3.2, Table 1
LLaMA-2 7B → LWM-Text
- 32K01→
- 128K02→
- 256K03→
- 512K04→
- 1M05
Each arrow reuses the previous stage’s model weights.
TPU v4-512 · 1M = 1,048,576 Tokens · Table 1
Across the five stages, each arrow carries forward the preceding model's weights. The lower row lists the actual total training tokens at each stage. The 32K stage reads 4.8B, and the 1M stage reads another 1.8B. The length of each sequence and the total tokens read during a stage are two different numbers.
Why progress in stages? Each step at the longest length is expensive. If every update used that length from the start, fixed compute and time budgets would constrain the amount of training data and the update schedule. Starting shorter and extending gradually is this paper's way of managing training cost.
Long-context performance can also regress later.
The Llama 3 report notes that following long-context pretraining with supervised fine-tuning using only short-context data hurts long-document performance. The authors therefore added long-document question answering, summarization, and code-reasoning data to maintain long-context ability through later training. Llama 3 §4.3.4
The same report describes extending the 405B model from 8K to 128K in six stages, checking retrieval ability and short-context performance at each stage. Llama 3 §3.4.2
We want models to read farther while still handling their old short questions well. LongRoPE also searches positional settings for shorter ranges to recover short-context performance. Extension needs evaluation on long and short tasks together. LongRoPE §3.3
Now suppose the novel has been read, and the model is about to answer “the blue drawer.”
Once answering begins, which computations can we keep?
We can separate a generative model's work on long input into two phases.
The first phase, prefill, processes the prompt and documents, computes each layer's representations, and establishes the state needed for the answer. Full score matrices, FlashAttention, and distributing long sequences are closely related to this phase's costs; these tools also apply in settings such as training.
The second phase, decode, generates the answer token by token. Each new position has its own Q, K, and V and needs to reference preceding content.
Under causal attention with a fixed model and fixed positional configuration, earlier positions cannot see the future. When the answer gains a token, the K and V already computed for earlier positions can therefore be reused. That retained data is the KV cache.
Why KV cache, without Q in the name? Because a new position asks with its own Q. Earlier Qs have already produced their outputs, and new positions only need to reference those Ks and Vs.
The new Q compares with existing Ks and uses the resulting weights to combine V. For that single new position, attention is one row against N positions. It avoids recomputing earlier positions but still needs access to their available K and V.
So the longer the novel, the more KV each layer retains.
Assume every layer retains the full sequence and uses the same KV-head count and dimension. For one request, estimate KV-cache size as:
The leading 2 accounts for K and V. With 32 layers, 8 KV heads, 128 dimensions per head, and 2 bytes per value, a million tokens require about 122.1 GiB. That counts KV only; model weights are extra.
32 layers · 128 dimensions · 2 bytes/value
One request, 32 layers, 128 dimensions per KV head, 2 bytes per value. Compare 8 and 32 KV heads. KV cache only; weights are separate.
For the same million tokens, increasing KV heads from 8 to 32 while holding everything else fixed raises the total to about 488.3 GiB. Both memory curves grow with length; the 32-head curve is always four times the 8-head curve. These are teaching configurations chosen for this article. Check a model's actual architecture for layer count, KV-head count, and data type.
Grouped-Query Attention (GQA) lets a group of Q heads share one K head and one V head, reducing the amount of KV to store and read. Query-head and KV-head counts can therefore differ. This requires a matching architecture and training. To convert an existing multi-head model, the paper averages the K and V projection matrices within each group separately, then performs additional training. GQA §2.2–3.1
When many requests arrive together, the service also needs memory allocation. Some conversations have just begun; others have filled pages. Reserving one large contiguous region for each can waste considerable space.
PagedAttention divides the KV cache into blocks, managing logical-to-physical mappings to improve reservation, fragmentation, and copying for sharing. This improves cache management; the model architecture and numerical storage format still determine KV data per token. PagedAttention §3–4
When examining a long-context system, separately measure at least how long the first answer takes to appear after the whole novel is submitted, and how many tokens per second it can produce once answering has begun.
Prefill with full causal attention has quadratic pairwise work. A decode step using KV cache refers to preceding K and V, with longer sequences increasing the associated reads and computation. The two phases have different work shapes and potentially different bottlenecks.
For the length limit, the wait before answering, answer speed, and the number of concurrent users, the earlier sections have each shown where the bottleneck lies.
This cache is intermediate state for the current generation. The novel becomes context for the present computation; model parameters change through training. The two store different things and have different lifetimes.
The last hurdle, the one most like a reading test, checks what exactly it answered correctly.
A million-token context: what kind of reading is tested?
Back to the key in chapter one.
A common language-model report card is perplexity, which measures how much probability the model assigns to the next token in real text. LongRoPE also uses long-text perplexity to evaluate language modeling after extension and compare context lengths. LongRoPE §4.2, Tables 5–7
Yet many next words can be predicted from nearby text. To ask specifically whether the model can retrieve chapter one's key at the end, we need to design that task directly, controlling where the answer is hidden, how far away it is, and how much distractor text surrounds it.
The most direct test places a unique piece of information inside a long text and asks the model to retrieve it. LongRoPE includes five-digit passkey retrieval, which inserts a code amid a large amount of distractor text and asks for it at the end.
The paper's LongRoPE-LLaMA2-7B (ft=256K) maintains at least 90% accuracy in this test from 4K to 2048K. Each length has ten trials with random passkey placement: nine correct out of ten is 90% under this setup. In the same figure, LongRoPE-Mistral-7B (ft=128K) falls to 60% at 2048K. Here ft denotes the context window configured for fine-tuning; the Mistral run actually used 16K training sequences, unlike the LLaMA2 training lengths discussed earlier. LongRoPE §4.1–4.2, Figure 4
This result shows that the model really can recover a particular kind of information from very distant input positions.
If the key stays in the drawer, things are fairly straightforward. Now change where things go.
Chapter one says only: “The key goes into the box.” Chapter eight says: “The box goes into the coat.” Chapter fifteen says: “The coat is handed to Xiaomei.” Finally: “Who should we ask for the key now?”
The last sentence alone tells us Xiaomei has the coat, but leaves its relationship to the key unknown. We must follow key to box, box to coat, and finally coat to Xiaomei. Remove any link and the path breaks.
Three independent facts: the key is put in a box; the box is put in a coat; the coat is given to Xiaomei. Direct retrieval finds the box. Connecting all three relations locates the key with Xiaomei; the final sentence alone mentions only the coat. Aggregation counts all three placement or handover actions: 1 + 1 + 1 = 3. Teaching example.
Single-point retrieval asks “Which container directly holds the key?” Chapter one is enough. Linking across chapters requires retaining relations to form “key → box → coat → Xiaomei.” Asking instead “How many placements or handovers occur across these three chapters?” means adding one action per chapter: 1 + 1 + 1 = 3, including the first placement. This is this article's teaching example.
Even putting the same clue in a different position can change the result.
Lost in the Middle makes this positional comparison. Below is Figure 1 from the published TACL version: fix a multi-document question-answering setup and vary the position of the answer-containing document among twenty documents. Original Figure 1
From left to right on the horizontal axis, the answer document moves from the beginning to the end. The vertical axis shows question-answering accuracy. The purple curve is this GPT-3.5-Turbo-0613 experiment; the dashed red line is the closed-book baseline, with no documents supplied.
Start with the middle of the curve, where the same answer-containing document leads to worse answers when placed in the middle. This figure's total length is about 4K, illustrating positional sensitivity for this model and task. Million-token models need fresh checks over their actual operating range.
In 2024, RULER divided long-context testing more finely into retrieval settings, multi-step variable tracking, information aggregation, and question answering. The first three use synthetic tasks. Question answering uses SQuAD and HotpotQA questions with distractor documents added to control length. Across ten models and thirteen tasks in v1, it observed that models doing well on simple needle retrieval could still degrade substantially with longer inputs or more complex tasks. RULER §3–4
Compare the claimed and effective lengths with the scores at each tested length.
This is why “one million” deserves a full follow-up question:
A million tokens usable for which tasks, with information in which positions, at what accuracy and cost?
The 2024 Gemini 1.5 Pro report demonstrated text needle-retrieval and code next-token-prediction experiments at scales up to ten million tokens. Other long-document question-answering and code tasks have their own lengths and metrics. The report does not disclose the complete long-context implementation recipe. Gemini 1.5 §2, §4.2
Now open that novel again
Moving from 512 to a million requires positional encoding, attention computation, memory, and training to keep up. Input length, processing cost, and which clues the model can use in an answer each need their own examination.
Next time a specification says 1M or 2M, first find the training sequence length, then the actual tasks' lengths, answer positions, accuracy, and cost. Comparing those conditions with your own documents gives you a basis for judging whether the model can do the job.
Then turn back to chapter one: the key is in the box, the box is in the coat, and the coat was handed to Xiaomei.
Whew. That was quite a journey. Next time, I'm just reading the novel.
