The Hidden Memory Architecture of LLMs

The Hidden Memory Architecture of LLMs

Your LLM is not running out of intelligence. It is often hitting context and runtime memory limits.

This article is written from the lens of translating "unexpected LLM behavior" into engineering controls you can measure, verify, and enforce. Where latency is percentiles, not averages. Where concurrency is real. Where cost has a curve. Where one bad assumption turns into an incident.

What Evolved, and What Did Not

The modern LLM wave rides on the Transformer architecture introduced in Attention Is All You Need. What changed since then is not the core idea of attention — what changed is the engineering around it:

• Kernels got smarter about memory movement
• Inference got separated into phases and pipelines
• KV cache went from a tensor to an allocator problem
• Serving systems started looking like OS schedulers

LLM performance is now strongly shaped by memory behavior, not just FLOPs. That is not a vibe — it is why whole research lines exist around IO-aware attention and KV cache management.

A Story from CognitionX 2025

This happened live at the CognitionX Dubai Conference 2025.

Most CognitionX events are community-focused on engineering-first learning, turning modern AI and cloud capabilities — including Microsoft technologies — into practical systems people can build, measure, and operate. The event brought together Microsoft MVPs and practitioners to share proven patterns and hands-on best practices.

The goal was to land a point in a way engineers can't unsee:

A live demo was run on an NVIDIA A100 80GB instance. Before anything, the runtime was intentionally warmed — the very first request on a fresh process or fresh GPU context can include one-time overhead (model weight loading, CUDA context creation, kernel initialization, allocator warm-up) that is not representative of steady-state inference.

The demo started with a clean run: a short input, fast output, stable behavior. This is what most demos show — a model that looks powerful and responsive when prompt length is small, concurrency is low, and runtime state is minimal.

Then, one variable was changed on purpose: constraints and context kept being added exactly the way real users do — more requirements, more follow-ups, more iterations back to back. Same model, same serving stack, same GPU. The only thing that changed was the amount of context being processed and retained by the runtime.

As context grew and request patterns became less predictable, end-to-end latency increased and sustained throughput dropped, and the available memory headroom tightened. Nothing "mystical" happened to the model. The serving system was simply pushed into a regime where it was more constrained by memory footprint, memory bandwidth, batching efficiency, and scheduler behavior than by raw compute.

The Mental Model That Fixes Most Confusion

LLM inference is the runtime forward pass where the model turns input tokens into a probability distribution for the next token. It runs in two phases:

1. Prefill — process the whole prompt once and build KV cache
2. Decode — generate tokens one-by-one while reusing KV cache

Performance and stability are dominated by context limits + KV cache memory/bandwidth, not just compute.

The Two Phases

• Prefill processes the full prompt tokens in parallel and creates the KV cache.
• Decode generates tokens autoregressively, one token at a time, reusing the KV cache.

The first real punchline: Prefill is compute heavy. Decode is memory hungry.

Decode reuses prior keys and values, which means the system is constantly reading KV cache from GPU memory. That is why decode often becomes memory-bandwidth bound and tends to underutilize GPU compute.

When people ask why the GPU looks bored while tokens are slowly streaming, the answer is usually: because decode is waiting on memory.

KV Cache Is Not an Optimization — It Is the Runtime State

In a Transformer decoder, each layer produces keys and values per token. If those had to be recomputed for every new token, latency would explode. So we cache K and V. That cache grows with sequence length.

The KV cache is one of the largest pieces of mutable state in LLM inference, and it is dynamic — it grows per request, per turn, per decoding strategy.

This is exactly the problem that the vLLM PagedAttention paper attacks (arXiv): high-throughput serving needs batching, but KV cache memory becomes huge and changes shape dynamically, and naïve management wastes memory through fragmentation and duplication.

Paging: The KV Cache Allocator Is the Hidden Bottleneck

If you allocate KV cache as big contiguous tensors per request, two things happen: you over-allocate to plan for worst-case length, and you fragment memory as requests come and go.

PagedAttention addresses this by storing KV cache in non-contiguous blocks allocated on demand, eliminating external fragmentation by making blocks uniform, and reducing internal fragmentation by using smaller blocks. The vLLM paper claims near-zero waste in KV cache memory with this approach, and reports 2–4x throughput improvements compared to prior systems.

# Conceptual: PagedAttention block allocation
# Instead of one contiguous tensor per sequence:
#   kv_cache = torch.zeros(max_seq_len, num_heads, head_dim)  # wasteful
#
# PagedAttention uses fixed-size blocks allocated on demand:
class KVBlockAllocator:
    def __init__(self, block_size: int, num_blocks: int, device: str):
        self.block_size = block_size
        self.free_blocks = list(range(num_blocks))
        self.pool = torch.zeros(
            num_blocks, block_size, 2, num_heads, head_dim, device=device
        )  # 2 for K and V

    def allocate(self) -> int:
        """Allocate a single block — no contiguous requirement."""
        if not self.free_blocks:
            raise RuntimeError("OOM: no free KV blocks — reduce batch or sequence length")
        return self.free_blocks.pop()

    def free(self, block_id: int):
        """Return block to pool — zero fragmentation."""
        self.free_blocks.append(block_id)

Attention Budgets: The Real Meaning of Context Limits

Context window is often marketed like a feature. In production it behaves like a budget that you spend.

The FlashAttention paper (arXiv) opens with the key fact: Transformers get slow and memory-hungry on long sequences because self-attention has quadratic time and memory complexity in sequence length.

When you choose longer contexts, you are not choosing more text. You are choosing:

• More KV cache to store
• More memory bandwidth pressure during decode
• More IO pressure inside attention kernels
• More tail latency risk under concurrency

Context length is not a free upgrade. It is an architectural trade.

Multi-Tenancy: The Memory Security Problem Nobody Wants to Own

Memory is not only a performance layer. It is also a security surface.

The same Zero-Trust logic that applies to agent architectures applies one layer deeper — at the memory level. Once you batch users, cache prefixes, and reuse state, you are operating a multi-tenant platform whether you admit it or not. Isolation and scope become first-class design constraints.

KV cache can become a leakage channel if scoping is neglected:

• Cross-tenant prefix caching without strict scoping and cache key namespaces
• Shared batch scheduling that can leak metadata through timing and resource signals
• Debug endpoints that expose tokenization details or cache keys
• Logs that accidentally store prompts, prefixes, or identifiers

Determinism Under Load

In well-controlled setups, an LLM can be highly repeatable. But under certain serving conditions — especially high concurrency and dynamic batching — the same model with the same request and same parameters may produce different output.

Reproducibility is a systems property. The model is only one part of the computation. What actually runs is a serving runtime: batching and scheduling decisions, kernel selection, numeric precision paths, and memory pressure.

Temperature=0 makes the decoding rule deterministic — but it does not make the runtime deterministic:

import numpy as np

# Near-tie: tiny perturbation can flip argmax
z = np.array([0.5012, 0.5008, 0.1, -0.2])  # top-2 candidates are close
a = int(np.argmax(z))
b = int(np.argsort(z)[-2])
margin = z[a] - z[b]
eps = 3e-4  # tiny perturbation scale

print(f"Top: {a}, Second: {b}, Margin: {margin:.6f}")

# Worst-case-style delta: push top down, runner-up up
delta = np.zeros_like(z)
delta[a] -= eps
delta[b] += eps
z2 = z + delta

print(f"Argmax before: {int(np.argmax(z))}, after tiny delta: {int(np.argmax(z2))}")

# Autoregressive divergence (toy transition model)
rng = np.random.default_rng(0)
V, T = 8, 30
W = rng.normal(size=(V, V))

def next_token(prev: int, tweak: bool = False) -> int:
    logits = W[prev].copy()
    if tweak:
        top = int(np.argmax(logits))
        second = int(np.argsort(logits)[-2])
        logits[top] -= 1e-3
        logits[second] += 1e-3
    return int(np.argmax(logits))

yA, yB = [0], [0]
inject_step = 3

for t in range(1, T):
    yA.append(next_token(yA[-1], tweak=False))
    yB.append(next_token(yB[-1], tweak=(t == inject_step)))

first_div = next((i for i, (x, y) in enumerate(zip(yA, yB)) if x != y), None)
print(f"First divergence step: {first_div}")
print(f"Run A: {yA}")
print(f"Run B: {yB}")

A tiny runtime delta can flip one token selection in a near-tie. After that, the prefixes diverge and every subsequent step is conditioned on a different history. This is not "model mood" — it is a direct consequence of the autoregressive feedback loop.

Architectural Design: AI as Distributed Memory

The goal is to keep control plane and data plane clean, and treat memory as a first-class layer. If you do that, scaling becomes a deliberate engineering exercise instead of a firefight.

Memory Layer Decisions

• Long prompts + repeated prefixes: enable prefix caching, and scope it properly per tenant / per model config
• OOM or low batch size: treat KV cache as an allocator problem — adopt paging strategies (PagedAttention-style thinking)
• Tail latency spikes: consider separating prefill and decode where it fits, but accept KV becomes a distributed object with transfer + consistency overhead
• Decode feels slow / GPU looks bored: consider speculative decoding, but benchmark it honestly under your workload and acceptance rate

Closing: What You Should Take Away

If you remember one thing, make it this: LLM inference can behave like a stateful memory system first, and a model endpoint second.

The serving layer — KV cache growth, memory bandwidth during decode, allocator/paging behavior, and batching/scheduling — is what decides whether your system is stable under real traffic, or only impressive in demos.

What This Means, Depending on Your Role

Senior Engineer — Stop debugging by folklore. When behavior is "weird," ask first: did the effective input change, did the runtime state change, or did the execution path change? Then prove it with telemetry.

Principal Engineer — Design the serving invariants: cache scoping rules, allocator strategy, admission control, and a determinism stance. PyTorch gives you switches for deterministic enforcement — use them deliberately.

SRE — Treat inference like an OS workload: queues, memory headroom, allocator efficiency, and p95/p99 under concurrency.

CTO / Platform Owner — The win isn't buying bigger GPUs. It's building control points: governance boundaries, isolation for shared state, determinism expectations, and operational discipline.

Originally published on Microsoft Tech Community.