Position Encoding in Transformers: From Sinusoidal to RoPE, ALiBi, and Long-Context Scaling

Introduction

Every modern large language model must solve a deceptively simple problem: how does the network know the order of words in a sentence? The self-attention mechanism at the heart of the transformer architecture computes pairwise interactions between all tokens in a sequence, but it does so in a way that is completely agnostic to position. If you randomly shuffle the tokens, the raw attention scores — before any positional information is injected — remain identical. The model cannot distinguish “the cat sat on the mat” from “mat the on sat cat the” without some explicit signal about where each token lives in the sequence.

This is not a minor detail. Position is fundamental to language. “Dog bites man” and “man bites dog” contain the same tokens but carry opposite meanings. Without positional encoding, a transformer is a bag-of-words model with a very expensive attention mechanism. Every major design decision in position encoding — from the original sinusoidal approach to modern techniques like RoPE and ALiBi — flows from this single fact: self-attention is permutation-invariant, and we must break that symmetry.

Over the past several years, the field has iterated through multiple generations of positional encoding. Each generation solved problems introduced by the previous one, while introducing new trade-offs of its own. This article traces that evolution in detail, covering:

1. Why positional encoding is necessary (the permutation-invariance problem)
2. Learned absolute embeddings (GPT-2 and its limitations)
3. Sinusoidal embeddings (the original transformer)
4. ALiBi (zero-parameter linear biases in attention)
5. RoPE (rotary embeddings via complex-valued rotations)
6. Context extension (linear scaling, NTK-aware scaling, YaRN)
7. Attention sinks and StreamingLLM (infinite-length inference)
8. A decision framework for choosing the right approach

By the end, you should have both the mathematical intuition and the practical engineering knowledge to make informed decisions about positional encoding in your own systems.

---

1. Why Position Encoding Exists

The Permutation-Invariance Problem

Consider the standard scaled dot-product attention formula. Given a sequence of $n$ tokens, we compute queries $Q$, keys $K$, and values $V$ by projecting the input embeddings $X$ through learned weight matrices:

$Q = XW_Q, \quad K = XW_K, \quad V = XW_V$

The attention output is then:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right) V$

Now suppose we apply an arbitrary permutation $\pi$ to the token positions, producing a reordered input $X_\pi$. The attention scores become:

$\frac{(X_\pi W_Q)(X_\pi W_K)^\top}{\sqrt{d_k}}$

Because the dot product operates over all pairs of tokens, permuting the input simply permutes the rows and columns of the attention matrix. The set of pairwise interactions remains identical. The softmax normalizes each row independently, so the resulting attention weights are just a permuted version of the original. In other words, attention treats its input as a set, not a sequence.

This is by design — set-level reasoning is powerful for many tasks. But for language modeling, where token order is essential, we need to inject positional information explicitly.

What Position Encoding Must Achieve

A good positional encoding scheme should satisfy several properties:

1. Uniqueness: Each position gets a distinct encoding, so the model can tell positions apart.
2. Consistency: The encoding for a given position should be deterministic — the same position always gets the same signal.
3. Bounded: The magnitude of the encoding should not grow unboundedly with sequence length, or it will dominate the token content signal.
4. Generalization: Ideally, the model should handle sequences longer than those seen during training (length extrapolation).
5. Relative sensitivity: Many linguistic phenomena depend on the distance between tokens rather than their absolute position. “The” at position 5 and “the” at position 500 should behave similarly relative to their local context.

No single approach satisfies all of these perfectly. The history of positional encoding is the story of trading off among these desiderata.

---

2. Learned Absolute Positional Embeddings (GPT-2)

How They Work

The simplest approach to positional encoding, used in GPT-2 (Radford et al., 2019) and BERT (Devlin et al., 2018), is to maintain a learnable embedding table $E_{\text{pos}} \in \mathbb{R}^{L \times d}$, where $L$ is the maximum sequence length and $d$ is the model dimension. At each position $i$, the positional embedding $E_{\text{pos}}[i]$ is added element-wise to the token embedding:

$h_i^{(0)} = E_{\text{tok}}[x_i] + E_{\text{pos}}[i]$

This combined vector is then fed through the transformer layers. The model learns what each position “means” during training, jointly optimized with all other parameters via backpropagation.

Strengths

Learned absolute embeddings are simple to implement and require no mathematical design choices beyond choosing the maximum length $L$. The model can learn arbitrary positional patterns — if certain positions have special roles (beginning-of-sequence, end-of-sentence markers), the embeddings can capture that. In practice, for sequences within the training length, learned absolute embeddings work well.

Critical Limitations

The fundamental problem is the hard context length ceiling. If the model is trained with $L = 1024$ (as in GPT-2), there is no embedding vector for position 1025. You cannot extrapolate. The model has never seen a gradient for that position, and the embedding table simply does not have an entry for it.

The second major issue is lack of relative position awareness. There is nothing in the design that encourages the model to learn that the relationship between positions 3 and 7 should be similar to the relationship between positions 103 and 107. The model may learn such patterns implicitly, but it is not guaranteed, and it wastes capacity that could be used for other purposes.

Finally, learned absolute embeddings are wasteful at inference time. If you allocate a 2048-position embedding table but typically run inference on 128-token prompts, you are storing 1920 unused embedding vectors in memory. This is not catastrophic, but it reflects an inelegant coupling between the maximum possible length and the resources consumed.

Historical Context

Despite these limitations, learned absolute embeddings dominated the early transformer era. GPT-2 used them with $L = 1024$. The original BERT used them with $L = 512$. For the tasks and sequence lengths of 2018-2019, these limits were acceptable. It was the push toward longer contexts — 4K, 8K, 32K, and eventually 128K+ tokens — that made the limitations untenable and drove the search for better alternatives.

---

3. Sinusoidal Positional Embeddings (Original Transformer)

The Vaswani et al. Design

The original “Attention Is All You Need” paper (Vaswani et al., 2017) introduced a non-learned approach using sinusoidal functions. Rather than training a position embedding table, the authors defined positional encodings analytically:

$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d}}\right)$ $PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d}}\right)$

Here, $pos$ is the position index, $i$ is the dimension index, and $d$ is the model dimension. Each dimension pair $(2i, 2i+1)$ oscillates at a different frequency, creating a unique “fingerprint” for each position.

Intuition: A Binary Clock Analogy

The sinusoidal encoding can be understood by analogy with binary counting. In a binary number, the least significant bit toggles every step, the next bit every two steps, the next every four steps, and so on. The sinusoidal encoding works similarly, but with smooth sinusoidal waves instead of discrete bit flips. Low-frequency dimensions (large $i$) change slowly across positions, encoding coarse positional information. High-frequency dimensions (small $i$) change rapidly, encoding fine-grained position.

The base frequency of $10000$ means that the lowest-frequency dimension completes one full cycle over approximately 10000 positions. This creates a rich spectrum of frequencies that can, in principle, distinguish any two positions within a 10000-token window.

The Relative Position Property

Vaswani et al. noted a key mathematical property: for any fixed offset $k$, the encoding at position $pos + k$ can be expressed as a linear transformation of the encoding at position $pos$. Specifically:

$PE_{pos+k} = M_k \cdot PE_{pos}$

where $M_k$ is a rotation matrix that depends only on $k$, not on $pos$. This means the encoding carries information about relative position — a constant offset always produces the same linear transformation, regardless of where you are in the sequence.

In theory, this allows the model to learn relative-position-dependent attention patterns. In practice, the additive injection into the input (rather than directly into the attention computation) somewhat dilutes this property by the time information reaches deeper layers.

Limitations

Despite being more principled than learned embeddings, sinusoidal encodings share a key problem: they are injected at the input layer and then transformed by many layers of nonlinear computation. By the time the signal reaches deep attention layers, the positional information may be attenuated or distorted. Additionally, while sinusoidal encodings theoretically support arbitrary lengths, models trained with them still struggle with extrapolation in practice, because the model’s learned attention patterns were never trained on the longer positions.

---

4. ALiBi: Attention with Linear Biases

Motivation

Press, Smith, and Lewis (2022) proposed ALiBi (Attention with Linear Biases) as a radically simple alternative. Their key insight: instead of encoding position into the token representations at all, inject positional information directly into the attention scores as a fixed bias. No learned parameters. No sinusoidal functions. Just a linear penalty proportional to the distance between the query and key positions.

The ALiBi Formula

In standard attention, the score between a query at position $m$ and a key at position $n$ is:

$\text{score}(m, n) = \frac{q_m \cdot k_n}{\sqrt{d_k}}$

ALiBi modifies this to:

$\text{score}(m, n) = \frac{q_m \cdot k_n}{\sqrt{d_k}} - s_h \cdot |m - n|$

where $s_h$ is a fixed slope assigned to attention head $h$. The slopes are set geometrically:

$s_h = \frac{1}{2^{8h/H}}$

for a model with $H$ attention heads. This means head 1 might have a slope of $\frac{1}{2}$ (strong distance penalty, focuses on nearby tokens), while the last head might have a slope of $\frac{1}{256}$ (weak penalty, attends broadly across the sequence).

Why It Works: Multi-Resolution Attention

The geometric distribution of slopes creates a multi-resolution view of the sequence. Some heads are essentially local attention mechanisms, strongly penalizing any token more than a few positions away. Other heads act as near-global attention, barely penalizing distance at all. This division of labor means the model can simultaneously capture local syntactic patterns and long-range semantic dependencies without any positional parameters.

Extrapolation Capability

Because ALiBi operates purely on relative distance, and the penalty function is a simple linear ramp, the model never encounters a “new” position at inference time. Whether the distance is 10 or 10000, the formula is the same: multiply the slope by the distance. This gives ALiBi excellent length extrapolation — models trained on short sequences can process much longer sequences at inference time with relatively little degradation.

Press et al. demonstrated that a model trained on 1024 tokens could extrapolate to 2048 tokens with minimal perplexity increase, and even to 6144 tokens with graceful degradation. This was a significant improvement over absolute embeddings, which simply fail at positions beyond the training range.

Implementation

ALiBi is trivially implemented. You precompute a matrix of shape $(n, n)$ containing the values $-s_h \cdot |i - j|$ for all position pairs, then add it to the attention logits before the softmax. In a FlashAttention kernel, this translates to a simple addition during the score computation.

import torch
import math

def build_alibi_bias(num_heads: int, seq_len: int) -> torch.Tensor:
    """
    Build the ALiBi bias matrix for all heads.
    Returns: [num_heads, seq_len, seq_len]
    """
    # Geometric slopes: 1/2, 1/4, 1/8, ..., 1/2^(8H/H) = 1/256
    slopes = torch.tensor([
        1.0 / (2 ** (8 * (h + 1) / num_heads))
        for h in range(num_heads)
    ])

    # Distance matrix: |i - j| for all position pairs
    positions = torch.arange(seq_len)
    distances = torch.abs(positions.unsqueeze(0) - positions.unsqueeze(1))

    # [num_heads, seq_len, seq_len]
    bias = -slopes.unsqueeze(-1).unsqueeze(-1) * distances.unsqueeze(0)
    return bias

Trade-offs

The simplicity of ALiBi is both its strength and its limitation. The linear penalty function makes a strong assumption: the importance of a token decays linearly with distance. For many language tasks this is a reasonable prior, but it may not capture more complex positional relationships. Additionally, the bias matrix has shape $(n \times n)$ per head, which means it grows quadratically with sequence length. At very long contexts (32K+), this bias matrix itself becomes a memory bottleneck.

---

5. RoPE: Rotary Position Embeddings — A Deep Dive

Motivation and Core Idea

Rotary Position Embeddings (Su et al., 2021) take a fundamentally different approach from both absolute embeddings and ALiBi. Rather than adding position information to the input or biasing the attention scores, RoPE rotates the query and key vectors in a way that makes their dot product naturally depend on relative position.

The key mathematical insight is that multiplication by a rotation matrix is an isometry — it preserves vector norms and angles. This means the rotation does not distort the content information in the query and key; it only changes the phase, encoding position without destroying semantic content.

Mathematical Foundation: Rotation in Complex Space

To understand RoPE, it helps to think of each pair of dimensions in a vector as coordinates in a 2D plane — or equivalently, as the real and imaginary parts of a complex number.

Consider a head dimension $d_h$. We group the dimensions into $d_h / 2$ pairs: $(x_0, x_1)$, $(x_2, x_3)$, …, $(x_{d_h-2}, x_{d_h-1})$. Each pair is treated as a complex number:

$z_i = x_{2i} + j \cdot x_{2i+1}$

For a token at position $m$, RoPE multiplies each complex number by a position-dependent rotation:

$z_i' = z_i \cdot e^{j m \theta_i}$

where $\theta_i$ is the frequency for dimension pair $i$:

$\theta_i = \frac{1}{10000^{2i/d_h}}$

Expanding the complex multiplication:

$x_{2i}' = x_{2i} \cos(m\theta_i) - x_{2i+1} \sin(m\theta_i)$ $x_{2i+1}' = x_{2i} \sin(m\theta_i) + x_{2i+1} \cos(m\theta_i)$

This is simply a 2D rotation of the pair $(x_{2i}, x_{2i+1})$ by angle $m\theta_i$.

The Relative Position Property: A Proof

Now comes the critical property. Let $q_m$ be a query vector at position $m$ and $k_n$ be a key vector at position $n$. After applying RoPE, the dot product contribution from dimension pair $i$ is:

$\text{Re}\left[(R_m q)_i \cdot \overline{(R_n k)_i}\right]$

where $R_m$ denotes the rotation by angle $m\theta_i$ and $\overline{\cdot}$ denotes complex conjugation. Expanding:

$(R_m q)_i \cdot \overline{(R_n k)_i} = (q_i \cdot e^{jm\theta_i}) \cdot \overline{(k_i \cdot e^{jn\theta_i})}$

$= q_i \cdot \overline{k_i} \cdot e^{jm\theta_i} \cdot e^{-jn\theta_i}$

$= q_i \cdot \overline{k_i} \cdot e^{j(m-n)\theta_i}$

Taking the real part:

$\text{Re}\left[q_i \cdot \overline{k_i} \cdot e^{j(m-n)\theta_i}\right]$

This depends on $m - n$ only, not on $m$ or $n$ individually. The absolute positions cancel out, leaving only the relative offset. This is the formal proof that RoPE is a relative position encoding.

Frequency Bands and the Base 10000

The choice of frequencies $\theta_i = 1/10000^{2i/d_h}$ creates a geometric progression from high frequency ($\theta_0 \approx 1$, completing a full rotation every ~6 positions) to very low frequency ($\theta_{d_h/2-1} \approx 1/10000$, completing a full rotation over ~63000 positions).

This multi-scale frequency design serves a critical purpose:

• High-frequency bands ($\theta$ near 1): These dimensions rotate rapidly with position, encoding fine-grained positional differences. They allow the model to distinguish adjacent tokens, which is important for local syntactic patterns (e.g., verb agreement, determiner-noun proximity).

• Low-frequency bands ($\theta$ near $1/10000$): These dimensions rotate slowly, changing only slightly between adjacent positions but accumulating meaningful phase differences over long distances. They encode coarse positional information, allowing the model to distinguish tokens that are hundreds or thousands of positions apart.

• Mid-frequency bands: These provide intermediate resolution, capturing sentence-level and paragraph-level structure.

The base value of 10000 was chosen empirically by Su et al. It determines the ratio between the fastest and slowest frequencies. A larger base spreads the frequencies more, giving more resolution to long-range positions at the expense of short-range precision. A smaller base compresses the frequency range, favoring short-range precision. The value 10000 provides a good balance for typical NLP tasks.

import torch

def compute_rope_frequencies(dim: int, max_seq_len: int, base: float = 10000.0):
    """
    Compute rotation frequencies for each dimension pair.

    theta_i = 1 / (base^(2i/d)) for i = 0, 1, ..., d/2 - 1

    Returns cos and sin tables for all positions.
    """
    inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2).float() / dim))
    positions = torch.arange(max_seq_len)

    # [seq_len, dim/2] -- outer product gives all position-frequency combinations
    freqs = torch.outer(positions, inv_freq)

    # [seq_len, dim] -- duplicate for pairing with both sin and cos
    emb = torch.cat([freqs, freqs], dim=-1)

    return torch.cos(emb), torch.sin(emb)

Applying the Rotation

In implementation, we do not actually work with complex numbers (most deep learning frameworks optimize better with real-valued tensors). Instead, we apply the rotation as a real-valued transformation:

def apply_rope(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor):
    """
    Apply rotary embedding to input tensor.

    x: [batch, seq_len, num_heads, head_dim]
    cos, sin: [seq_len, head_dim]

    For each dimension pair (x_2i, x_{2i+1}):
      x_2i'     = x_2i * cos - x_{2i+1} * sin
      x_{2i+1}' = x_2i * sin + x_{2i+1} * cos
    """
    x1 = x[..., ::2]   # Even indices
    x2 = x[..., 1::2]  # Odd indices

    cos_half = cos[..., ::2]
    sin_half = sin[..., ::2]

    rotated = torch.stack([
        x1 * cos_half - x2 * sin_half,
        x1 * sin_half + x2 * cos_half
    ], dim=-1).flatten(-2)

    return rotated

Efficient Caching for Inference

During autoregressive inference, the model generates one token at a time. Recomputing sin/cos tables at every step is wasteful. A practical implementation caches these values:

class RoPECache:
    """Cache RoPE sin/cos for efficient inference."""

    def __init__(self, dim: int, max_seq_len: int, base: float = 10000.0):
        self.dim = dim
        self.max_seq_len = max_seq_len
        self.base = base

        # Precompute inverse frequencies
        inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2).float() / dim))
        self.inv_freq = inv_freq

        # Cache will be populated on first forward pass
        self._cos_cached = None
        self._sin_cached = None
        self._seq_len_cached = 0

    def _update_cache(self, seq_len: int, device, dtype):
        if seq_len <= self._seq_len_cached:
            return

        self._seq_len_cached = max(seq_len, self.max_seq_len)

        positions = torch.arange(
            self._seq_len_cached, device=device, dtype=dtype
        )
        freqs = torch.outer(positions, self.inv_freq.to(device))

        emb = torch.cat([freqs, freqs], dim=-1)
        self._cos_cached = emb.cos()
        self._sin_cached = emb.sin()

    def forward(self, x: torch.Tensor, position_ids: torch.Tensor):
        """
        x: [batch, seq_len, num_heads, head_dim]
        position_ids: [batch, seq_len]
        """
        self._update_cache(
            position_ids.max().item() + 1, x.device, x.dtype
        )

        cos = self._cos_cached[position_ids]
        sin = self._sin_cached[position_ids]

        return apply_rope(x, cos.unsqueeze(2), sin.unsqueeze(2))

Why RoPE Won

RoPE has become the dominant positional encoding in modern LLMs. LLaMA, LLaMA 2, LLaMA 3, Mistral, Mixtral, Qwen, Qwen 2, and Yi all use RoPE. Several properties made it the winner:

1. Minimal memory overhead: Unlike ALiBi’s $O(n^2 H)$ bias matrix, RoPE only needs a frequency table of size $O(n \cdot d_h)$, which is linear in sequence length.

2. Strong relative position signal: The rotation is applied at every attention layer, so positional information is fresh and undiluted at every depth of the network.

3. FlashAttention compatibility: The element-wise rotation can be fused into the FlashAttention kernel with minimal overhead, unlike ALiBi’s bias matrix which requires a separate memory access pattern.

4. Extensibility: As we will see in the next section, RoPE’s frequency-based design admits elegant context extension techniques that allow models trained on short contexts to generalize to much longer ones.

---

6. Performance Comparison: ALiBi vs. RoPE

Before diving into context extension, it is worth comparing ALiBi and RoPE head-to-head on key engineering metrics.

Throughput at Different Sequence Lengths

As sequence length grows, the encoding mechanism’s efficiency increasingly determines achievable throughput. At short sequences, all methods perform nearly identically because positional encoding is a negligible fraction of total compute. At longer sequences, the differences become pronounced.

At 512 tokens, the difference is roughly 4%. At 2048, it widens to about 10%. At 8192, RoPE holds a 27% throughput advantage, driven almost entirely by the difference in memory access patterns. ALiBi’s quadratic bias matrix becomes a bandwidth bottleneck, while RoPE’s element-wise rotations remain cache-friendly.

Hardware Considerations

For those working on custom kernels or specialized hardware:

ALiBi is straightforward to implement — it is a simple addition to the attention logits before softmax. Any standard softmax kernel can incorporate it with minimal modification. However, the bias matrix grows quadratically with sequence length. At 32K tokens with 32 heads, the bias matrix alone consumes $32 \times 32768 \times 32768 \times 2$ bytes $\approx$ 128 GB in FP16, which exceeds the memory of any single GPU. In practice, you must recompute the bias on-the-fly from position indices, which adds compute.

RoPE requires trigonometric operations (sin/cos precomputation), but these are done once and cached. The per-token rotation is just two multiplications and an addition per dimension pair — operations that map perfectly to fused multiply-add (FMA) instructions. On NVIDIA Ampere and Hopper GPUs, the trig overhead is negligible compared to the memory savings. The element-wise nature of the rotation also means it has excellent L1/L2 cache locality.

---

7. Context Extension Techniques

One of the most active areas of research in 2023-2024 was context length extension — taking a model trained on a short context (e.g., 4K tokens) and enabling it to handle much longer sequences (16K, 32K, 128K+) without full retraining. RoPE’s frequency-based design makes it particularly amenable to such extensions.

The Problem: Out-of-Distribution Positions

When a RoPE model trained on 4K tokens encounters position 8000, the rotation angles for that position are values the model has never seen during training. The high-frequency dimensions will have wrapped around multiple times (which is fine — the model has seen all phases of high-frequency rotations), but the low-frequency dimensions will be at angles the model has never encountered. This causes attention patterns to break down, and perplexity spikes.

Linear Scaling (Position Interpolation)

The simplest fix is to rescale all position indices so that longer sequences map back into the trained range. If the model was trained on positions $[0, L)$ and you want to handle sequences of length $L' = sL$, divide all position indices by $s$:

$m' = m / s$

This compresses the position space so that position $L'$ maps to the original position $L$. The model now sees familiar rotation angles, at the cost of reduced positional resolution — positions that were one apart now appear $1/s$ apart.

def linear_scaling(position_ids: torch.Tensor, scale: float):
    """
    Scale positions to fit longer sequences into the trained range.

    Example: trained on 4K, want 16K. scale = 4.
    Position 8000 becomes 2000 (within training range).
    """
    return position_ids.float() / scale

NTK-Aware Scaling

NTK-aware scaling (bloc97, 2023) takes a more nuanced approach. Instead of uniformly compressing all frequencies, it adjusts the base of the frequency computation. This has the effect of stretching the low-frequency dimensions (which need more room to avoid out-of-distribution angles) while leaving the high-frequency dimensions largely unchanged (since they wrap around anyway).

The modified base is:

$b' = b \cdot s^{d/(d-2)}$

where $b = 10000$ is the original base, $s$ is the scaling factor, and $d$ is the head dimension. This exponential scaling of the base shifts the entire frequency spectrum downward, but the high-frequency components are already wrapping around so fast that the shift barely matters to them.

def dynamic_ntk_scaling(
    dim: int,
    max_seq_len: int,
    original_max_len: int,
    base: float = 10000.0
) -> torch.Tensor:
    """
    NTK-aware scaling adjusts the base frequency.
    Preserves high-frequency components while scaling low-frequency ones.
    """
    scale = max_seq_len / original_max_len

    # Scale the base exponentially
    scaled_base = base * (scale ** (dim / (dim - 2)))

    inv_freq = 1.0 / (
        scaled_base ** (torch.arange(0, dim, 2).float() / dim)
    )
    return inv_freq

The name “NTK-aware” comes from Neural Tangent Kernel theory. The intuition is that the positional encoding functions as a kernel, and the NTK framework tells us how to scale the kernel’s bandwidth to handle a wider input range without losing resolution in the critical high-frequency region.

YaRN (Yet another RoPE extensioN)

YaRN (Peng et al., 2023) combines the best ideas from linear scaling and NTK-aware scaling with an additional insight: different frequency bands should be treated differently. Low-frequency dimensions benefit from linear interpolation (they have smooth, slowly varying angles). High-frequency dimensions benefit from NTK-style base scaling (they wrap around quickly anyway). YaRN introduces a smooth ramp function that blends between these two strategies:

import math

class YaRNRoPE:
    """
    YaRN combines NTK scaling with per-dimension interpolation
    and attention scaling. Achieves the best quality at extended
    context lengths.
    """

    def __init__(
        self,
        dim: int,
        original_max_len: int,
        target_max_len: int,
        base: float = 10000.0,
        beta_fast: float = 32.0,
        beta_slow: float = 1.0,
    ):
        self.scale = target_max_len / original_max_len
        self.dim = dim
        self.base = base

        # Base inverse frequencies
        inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2).float() / dim))

        # Compute wavelength of each frequency in units of positions
        # wavelength_i = 2 * pi / theta_i
        wavelengths = 2 * math.pi * base ** (
            torch.arange(0, dim, 2).float() / dim
        )

        # Ramp function: 0 for low-freq dims, 1 for high-freq dims
        # Controlled by beta_slow and beta_fast thresholds
        low = original_max_len / beta_fast
        high = original_max_len / beta_slow
        ramp = torch.clamp(
            (wavelengths - low) / (high - low), 0.0, 1.0
        )

        # Blend: low-freq dims get linear scaling (1/scale),
        # high-freq dims keep original frequencies (factor 1.0)
        scaling_factors = (1.0 - ramp) * (1.0 / self.scale) + ramp * 1.0

        self.inv_freq = inv_freq * scaling_factors

        # Attention temperature scaling to compensate for
        # the distribution shift in attention logits
        self.attention_scale = 0.1 * math.log(self.scale) + 1.0

    def compute_frequencies(self, positions: torch.Tensor):
        """
        positions: [seq_len]
        Returns: [seq_len, dim/2] frequency values
        """
        return torch.outer(positions.float(), self.inv_freq)

The $\beta_{\text{fast}}$ and $\beta_{\text{slow}}$ parameters control the transition region between linear and NTK scaling. Dimensions whose wavelength is shorter than $L / \beta_{\text{fast}}$ are treated as high-frequency (kept unchanged). Dimensions whose wavelength is longer than $L / \beta_{\text{slow}}$ are treated as low-frequency (linearly scaled). Dimensions in between are smoothly blended.

YaRN also introduces an attention temperature scaling factor. When you extend context length, the distribution of attention logits shifts (more tokens means more potential attention targets), which can cause the softmax to become too peaked or too flat. The temperature correction $t = 0.1 \ln(s) + 1$ compensates for this shift.

The data tells a clear story. At modest extensions (4K to 8K, a 2x increase), all methods perform acceptably. At aggressive extensions (4K to 32K, an 8x increase), the gap is dramatic: linear scaling retains only 67% of baseline quality, while YaRN retains 92%. For production systems that need reliable long-context performance without full retraining, YaRN is the current state of the art.

Practical Guidelines for Context Extension

Based on extensive benchmarking across model sizes and tasks:

• Up to 2x extension: Linear scaling is sufficient and simplest to implement. Quality loss is minimal.
• 2x to 4x extension: NTK-aware scaling is recommended. It preserves high-frequency positional patterns that linear scaling destroys.
• 4x to 8x extension: YaRN is strongly recommended. The per-dimension blending and attention temperature correction provide meaningful quality improvements.
• Beyond 8x extension: You likely need fine-tuning on longer data. Even YaRN begins to degrade at very large extension factors, because the model’s attention patterns were simply never trained on such long-range dependencies.

---

8. Attention Sinks and StreamingLLM

The Problem: Truly Infinite Context

All the methods discussed so far operate within a fixed (if extended) context window. But what about applications that need to process truly unbounded streams of text — chatbots that run for hours, real-time transcription systems, or continuous document monitoring? Even a 128K context window eventually fills up.

The naive approach is a sliding window: drop the oldest tokens when the window is full. But this fails catastrophically in practice. Xiao et al. (2023) discovered why: the first few tokens in a sequence receive disproportionately high attention, regardless of their content. They called this phenomenon attention sinks.

What Are Attention Sinks?

During training, the softmax in attention must allocate its probability mass somewhere. Even when no token in the context is particularly relevant to the current query, the model cannot output a uniform zero attention pattern — the softmax always sums to 1. The model learns to “dump” excess attention onto the first token as a no-op: attending to token 0 acts as a way to avoid attending to anything in particular.

This is not a bug but an emergent behavior. The first token serves as a “sink” for unused attention probability. It emerges consistently across model families, sizes, and training procedures.

Why Sliding Windows Break

When you use a sliding window and evict the first token, you destroy the attention sink. The model’s attention distribution becomes unanchored — the probability mass that was going to token 0 now gets redistributed across other tokens in unpredictable ways, causing quality to collapse immediately and irreversibly.

StreamingLLM: The Solution

StreamingLLM (Xiao et al., 2023) proposes an elegant fix: maintain a two-part KV cache consisting of:

1. Sink tokens: The first $k$ tokens (typically $k = 4$), permanently pinned in the cache.
2. Rolling window: The most recent $w$ tokens, managed as a sliding window.

When the cache is full, you evict the oldest token from the rolling window but never touch the sink tokens. The total cache size is fixed at $k + w$, enabling infinite-length inference with constant memory.

class StreamingKVCache:
    """
    StreamingLLM KV cache: sink tokens + rolling window.
    Enables infinite-length inference with fixed memory.
    """

    def __init__(
        self,
        num_sink_tokens: int = 4,
        window_size: int = 1020,
        num_heads: int = 32,
        head_dim: int = 128,
    ):
        self.num_sink = num_sink_tokens
        self.window_size = window_size
        self.max_cache = num_sink_tokens + window_size

        # Pre-allocate cache
        self.k_cache = torch.zeros(
            1, self.max_cache, num_heads, head_dim
        )
        self.v_cache = torch.zeros(
            1, self.max_cache, num_heads, head_dim
        )
        self.cache_len = 0

    def update(self, new_k: torch.Tensor, new_v: torch.Tensor):
        """
        Add new key-value pairs, evicting old window tokens
        if cache is full. Sink tokens are never evicted.
        """
        n_new = new_k.shape[1]

        if self.cache_len + n_new <= self.max_cache:
            # Cache not full yet: just append
            self.k_cache[:, self.cache_len:self.cache_len + n_new] = new_k
            self.v_cache[:, self.cache_len:self.cache_len + n_new] = new_v
            self.cache_len += n_new
        else:
            # Cache full: keep sinks, slide window
            # Shift window left by n_new positions
            window_start = self.num_sink
            self.k_cache[:, window_start:-n_new] = (
                self.k_cache[:, window_start + n_new:].clone()
            )
            self.v_cache[:, window_start:-n_new] = (
                self.v_cache[:, window_start + n_new:].clone()
            )
            # Insert new tokens at the end
            self.k_cache[:, -n_new:] = new_k
            self.v_cache[:, -n_new:] = new_v

Position Handling in StreamingLLM

A subtle but critical detail: when using StreamingLLM with RoPE, the position IDs for the rolling window must be adjusted. After evicting tokens, the remaining window tokens have non-contiguous original positions (e.g., the sink tokens are at positions 0-3, and the window might contain tokens originally at positions 5000-6020). If you naively use these original positions, the RoPE rotations create large jumps in the frequency space, confusing the model.

The solution is to re-index the rolling window with contiguous positions starting from $k$ (the number of sink tokens). The sink tokens keep positions $0, 1, ..., k-1$, and the window tokens get positions $k, k+1, ..., k+w-1$, regardless of their original positions. This introduces a small positional inaccuracy (the model thinks the window tokens are close together even if they were originally far apart), but in practice this works well because the model’s attention patterns are primarily local anyway.

Limitations

StreamingLLM is not a free lunch. It discards information from the evicted tokens permanently. If the model needs to refer back to something said 10000 tokens ago and it has fallen out of the window, that information is simply gone. For applications that require reliable long-range recall, StreamingLLM should be combined with an external retrieval mechanism (RAG) rather than used as the sole solution.

---

9. Decision Framework

Given the landscape of options, how should you choose a positional encoding for your system? The answer depends on your constraints.

When to Use Each Method

Use learned absolute embeddings only if you are maintaining a legacy system and cannot modify the architecture. There is no scenario where this is the best choice for a new model.

Use sinusoidal embeddings only for educational purposes or very small models where simplicity matters more than performance. The original transformer formulation is elegant but has been superseded.

Use ALiBi if you need the simplest possible implementation with no learnable positional parameters, your sequences are moderate length (under 8K), and you want built-in length extrapolation without any scaling tricks. ALiBi is also a good choice if you are memory-constrained at short sequence lengths (under 2K), since it requires no parameter storage. However, be aware that very few modern pretrained models use ALiBi, so your ecosystem options are limited.

Use RoPE for any new decoder-only model. It is the industry standard, supported by all major inference frameworks (vLLM, TGI, TensorRT-LLM), compatible with FlashAttention, and extensible via the scaling techniques described above. The overwhelming majority of pretrained model weights available today (LLaMA family, Mistral family, Qwen family) use RoPE.

Use RoPE + YaRN when you need to extend an existing RoPE model to longer contexts without full retraining. For extensions up to 4x, NTK-aware scaling is simpler and almost as good. For extensions beyond 4x, YaRN is the best available option.

Use StreamingLLM when you need infinite-length inference with fixed memory. Combine with retrieval-augmented generation for applications that need long-range recall.

The Big Picture

The trajectory of positional encoding research tells a clear story: the field has converged on RoPE as the standard for new models, with active research focused on extending its range (YaRN, scaling techniques) and handling edge cases (streaming, attention sinks). ALiBi remains a valid alternative for specific use cases, but the ecosystem has moved decisively toward rotation-based embeddings.

For practitioners, the most important takeaway is that positional encoding is no longer a “set it and forget it” design decision. It interacts with context length, inference efficiency, kernel implementation, and even the streaming architecture of your serving system. Understanding the trade-offs covered in this article will help you make informed choices as context lengths continue to grow and new techniques emerge.

---

10. Summary and Practical Recommendations

Let us consolidate the key points from each section.

The fundamental problem is that self-attention is permutation-invariant. Without positional encoding, a transformer cannot distinguish token order, making it essentially a bag-of-words model.

Learned absolute embeddings (GPT-2 era) solve this by adding a trainable vector per position, but they impose a hard context length limit equal to the number of trained positions. They cannot extrapolate.

Sinusoidal embeddings (original transformer) use analytical functions instead of learned parameters, providing a theoretically unbounded position space. In practice, models still fail to extrapolate because the attention patterns were never trained on longer positions.

ALiBi injects position directly into attention scores as a linear penalty: $\text{score}(m,n) = q_m \cdot k_n - s_h |m-n|$. It uses zero learned parameters, extrapolates well up to ~6x training length, but has quadratic memory overhead from the bias matrix at long sequences.

RoPE rotates query and key vectors in complex space so that their dot product depends only on relative position: $\text{Re}[(R_m q) \cdot \overline{(R_n k)}]$ depends on $m - n$ only. It uses frequencies $\theta_i = 1/10000^{2i/d}$ spanning from fast local encoding to slow global encoding. RoPE has minimal memory overhead, excellent kernel compatibility, and is the dominant choice in modern LLMs.

Context extension via linear scaling, NTK-aware scaling, and YaRN allows RoPE models to handle sequences far beyond their training length. YaRN achieves the best quality by blending per-dimension scaling strategies and adjusting attention temperature.

Attention sinks and StreamingLLM enable infinite-length inference by preserving the first few tokens (which act as attention sinks) alongside a rolling window of recent tokens.

The field has clearly converged on RoPE as the standard. If you are building or fine-tuning a model today, use RoPE. If you need longer context, apply YaRN. If you need streaming, add StreamingLLM. This combination covers the vast majority of practical deployment scenarios.

# Quick reference: testing your context extension setup
def evaluate_context_extension(model, test_data, context_lengths):
    """
    Always verify perplexity at your target context length.
    Training-length performance does NOT predict extended performance.
    """
    results = {}
    for ctx_len in context_lengths:
        samples = [s[:ctx_len] for s in test_data if len(s) >= ctx_len]
        ppl = compute_perplexity(model, samples)
        results[ctx_len] = ppl
        print(f"Context {ctx_len:>6d}: PPL = {ppl:.2f}")
    return results

# Example usage:
# evaluate_context_extension(
#     model,
#     pg19_test,
#     [2048, 4096, 8192, 16384, 32768]
# )

The choice of positional encoding is one of the most consequential architectural decisions in a transformer system. It determines your maximum context length, your inference memory footprint, your kernel compatibility, and your ability to extend to longer contexts post-training. Understanding the trade-offs between these approaches — and knowing when to apply each one — is essential knowledge for anyone building or deploying transformer-based systems at scale.