Skip to content
ginsberg.si

Transformers from the Weights Up/Part 1 of 20

Transformers from the Weights Up, Part 1: The 30,000-Foot View

Text in, token out, and what a 50 GB weights file really is

1 Nov 2026

p(tn+1∣t1..n)=softmax⁡ ⁣(WU⋅Norm⁡(fL∘⋯∘f1(E[t1..n])))np(t_{n+1} \mid t_{1..n}) = \operatorname{softmax}\!\left(W_U \cdot \operatorname{Norm}(f_L \circ \cdots \circ f_1(E[t_{1..n}]))\right)_n

E is the embedding table, each fℓ is a block (image 7), WU is the unembedding (lm_head), and the subscript n means we read only the last position's distribution at inference time.

The pipeline in one picture

(a) One forward pass, then the loop
TextToken IDsEmbeddingN × blocksLogitsSoftmaxSampleAppendthecatsatonthep(next token | tokens so far)

Toy numbers for illustration: logits for the four candidates are made up; a real model scores all ~100k–150k vocabulary entries every step.

One forward pass through a transformer, then the loop that appends the next token.
  1. 1

    Tokenize

    A BPE tokenizer maps text to integer IDs. "the cat sat on the" is 5 tokens here; real tokenizers often merge a leading space into the token. (IDs shown are placeholders.)

  2. 2

    Embed

    Row lookup in a [V, d] matrix. Shape goes [n] → [n, d]. No arithmetic, just indexing.

  3. 3

    N blocks

    Each block: x ← x + Attn(Norm(x)); x ← x + MLP(Norm(x)). Shape stays [n, d] the whole way.

  4. 4

    Unembed

    Final norm, then lm_head: [n, d] → [n, V] logits, one score per vocabulary entry.

  5. 5

    Softmax and sample

    Turn the last row's logits into probabilities; pick one (greedy, temperature, top-k/top-p).

  6. 6

    Append and repeat

    The new token becomes input. With a KV cache, only that one token is pushed through the stack.

Images 1–3: the neuron

Image 1 is a single multiply. Image 2 is a dot product. Image 3 adds a bias and a nonlinearity. That is the entire atomic unit; everything in a 25B-parameter file is either one of these w's, one of these b's (modern LLMs mostly drop biases), or a norm gain.

y=σ(∑i=1nwixi+b),σ(z)=11+e−zy = \sigma\Bigl(\sum_{i=1}^{n} w_i x_i + b\Bigr), \quad \sigma(z) = \frac{1}{1+e^{-z}}
Image 1: a single weight
Image 1: One input times one weight: x · w = 0.8 × 0.5 = 0.4.
Image 2: a weighted sum
Image 2: Five inputs (0.3, -0.6, 0.8, 0.5, 0.9), each multiplied by its own weight and added up: Σ wᵢxᵢ.
Image 3: a neuron
Image 3: Add a bias and squash the result through a sigmoid: σ(Σ wᵢxᵢ + b).
(b) A neuron: images 1 → 2 → 3
0.3w1=0.50-0.6w2=-0.400.8w3=0.700.5w4=0.200.9w5=-0.30sigmoid0.707z = 0.880

Image 1 check: 0.8 × 0.5 = 0.4

Inputs fixed to the values in image 2. Edge thickness = |w|, color = sign. One neuron is one row of a weight matrix plus one bias entry.

A neuron: five inputs, weighted edges, bias, and a sigmoid.

Images 4–5: layers and MLPs

Stack 16 neurons that share the same 5 inputs and you get image 4: one matrix W of shape [16, 5], one bias vector [16]. Stack layers and you get image 5. In a transformer the MLP is two or three such matrices per block, and it is where most of the parameters live (about 81% in the config in panel e).

hℓ+1=σ(Wℓhℓ+bℓ)→MLP⁡(x)=Wdown⋅(SiLU⁡(Wgatex)⊙Wupx)h_{\ell+1} = \sigma(W_\ell h_\ell + b_\ell) \rightarrow \operatorname{MLP}(x) = W_{\mathrm{down}} \cdot \bigl(\operatorname{SiLU}(W_{\mathrm{gate}} x) \odot W_{\mathrm{up}} x\bigr)
Image 4: a layer
Image 4: Sixteen neurons sharing the same five inputs: σ(Wx + b).
Image 5: a multi-layer network (MLP)
Image 5: Layers stacked on layers: hₗ₊₁ = σ(Wₗhₗ + bₗ), three hidden layers down to three outputs.

The right-hand form is SwiGLU, used by Llama, Qwen, Mistral and most current open models. It is why the weights file has three FFN tensors per layer (gate_proj, up_proj, down_proj) instead of two. Forward cost: about 2 FLOPs per weight per token, so a dense model costs ≈ 2 × params FLOPs per generated token.

Image 6: attention

The MLP processes each position independently. Attention is the only place positions talk to each other. Each position emits a query, a key, and a value (three matmuls with q_proj, k_proj, v_proj); query–key dot products decide how much of each value to pull in.

Attn⁡(Q,K,V)=softmax⁡ ⁣(QKTdk+M)V\operatorname{Attn}(Q, K, V) = \operatorname{softmax}\!\left(\frac{QK^{T}}{\sqrt{d_k}} + M\right) V

M is the causal mask (0 on and below the diagonal, −∞ above). Divide by √dk so the dot products' variance doesn't grow with head dimension and saturate the softmax.

Image 6: attention
Image 6: A heatmap of softmax(QKᵀ/√d)·V over the cat sat on the mat: which words look at which.
(c) Attention weights softmax(QKᵀ/√d) — image 6
thecatsatonthematthe1.00−∞−∞−∞−∞−∞cat0.270.73−∞−∞−∞−∞sat0.080.670.25−∞−∞−∞on0.080.130.570.23−∞−∞the0.190.090.100.410.21−∞mat0.050.130.160.300.200.15

Query row "the" (position 0) mixes value vectors:

the1.00
cat0.00
sat0.00
on0.00
the0.00
mat0.00

Output at this position = Σⱼ aⱼ·vⱼ, then W_O projects it back to d and it is added to the residual stream.

Toy scores for illustration. Each row is a query position and sums to 1. With the causal mask, position i can only read positions ≤ i, which is what lets one training pass learn every next-token prediction at once.

Attention heatmap over the cat sat on the mat, with and without a causal mask.

Image 7: the block and the residual stream

h=x+Attn⁡(Norm⁡(x)),y=h+MLP⁡(Norm⁡(h))h = x + \operatorname{Attn}(\operatorname{Norm}(x)), \quad y = h + \operatorname{MLP}(\operatorname{Norm}(h))

Your image 7 omits the norms. Production models normalize the input of each branch (pre-norm) with RMSNorm, whose only weights are a gain vector of length d, so norms are a rounding error in the parameter count. The residual additions are why 50–100+ layer stacks train at all: gradients flow straight down the identity path.

RMSNorm⁡(x)=xmean⁡(x2)+ε⊙g\operatorname{RMSNorm}(x) = \frac{x}{\sqrt{\operatorname{mean}(x^{2}) + \varepsilon}} \odot g
Image 7: the transformer block
Image 7: h = x + Attn(x), then y = h + MLP(h). The + is the residual connection.
(d) The residual stream through one block — image 7
xhyRMSNorm → Attention+RMSNorm → MLP (SwiGLU)+1/6 — x enters the block (the residual stream, shape [seq, d])

Pre-norm layout used by Llama/Qwen-style models. Each sub-layer reads from the stream and writes an additive update; the stream itself is never overwritten.

Residual stream through one transformer block: attention and MLP each add an update.

Image 8: next token and the generation loop

pi=ezi/T∑jezj/Tp_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}

z = logits from lm_head, T = temperature. T → 0 is greedy (argmax), T = 1 is the model's raw distribution. "mat" winning in image 8 is just the highest z.

Image 8: next-token prediction
Image 8: p(next word | words so far) for the cat sat on the blank: mat scores highest, then floor, idea, moon.
KV bytes per token=2⋅L⋅nkv⋅dhead⋅bytes=2⋅52⋅8⋅128⋅2=212,992 B≈0.2 MB\text{KV bytes per token} = 2 \cdot L \cdot n_{kv} \cdot d_{\mathrm{head}} \cdot \text{bytes} = 2 \cdot 52 \cdot 8 \cdot 128 \cdot 2 = 212{,}992 \text{ B} \approx 0.2 \text{ MB}

For the panel (e) config, a 32k-token context therefore holds about 7 GB of KV cache on top of the weights. GQA (8 KV heads instead of 40) is what keeps that number 5× smaller.

Training vs inference

Training

Run the forward pass on every position of real text at once (causal mask makes each position predict its successor), compute cross-entropy against the actual next tokens, backpropagate, and update every weight with AdamW. Memory: weights + gradients + two Adam moments (typically fp32) + activations, a multiple of the bf16 weights.

L=−1n∑tlog⁡pθ(ti+1∣t≤i)\mathcal{L} = -\frac{1}{n} \sum_t \log p_{\theta}(t_{i+1} \mid t_{\le i})
Inference

Weights are frozen. Only the forward pass runs, the last position is read, a token is sampled, and the loop repeats. Memory: weights + KV cache + small activations. Nothing you do at inference changes the file.

What a 50 GB weights file is

A checkpoint is a dictionary: tensor name → (dtype, shape, raw bytes). In safetensors that is literally an 8-byte header length, a JSON header listing each tensor's name, dtype, shape and byte offsets, then one flat blob. There is no code and no graph in it; the architecture lives in config.json and in the inference engine. 25.3B parameters × 2 bytes (bf16) ≈ 50.6 GB, and that's the whole story of the file size.

(e) What's inside ~50 GB of bf16 — an illustrative 25.3B dense config
ffn gate_proj6.82B · 27.0%ffn up_proj6.82B · 27.0%ffn down_proj6.82B · 27.0%FFN (gate/up/down) 80.9% · attention 12.9% · embed + lm_head 6.2% · norms 0.002%Per layer ≈ 456.1M params ≈ 912 MB in bf16. The k/v slivers are thin because of GQA (8 KV heads vs 40 query heads).

Illustrative config (not a specific released model): d_model=5120, layers=52, 40 query heads / 8 KV heads × head_dim 128, SwiGLU ffn=25600, vocab=152064, untied embeddings. Total = 25,276,462,080 params × 2 bytes = 50.55 GB.

Share of parameters in an illustrative 25.3B model: FFN, attention, embeddings.
(f) One weight in bf16: 1 sign · 8 exponent · 7 mantissa bits
0
0
1
1
1
1
0
0
0
1
0
0
1
0
1
0
Hex
0x3c4a
Sign
0 → +
Exponent
120 − 127 = -7
Mantissa
1 + 74/128 = 1.578125
Stored value
0.012329102 (you asked for 0.0123; error 2.91e-5)

bf16 is the top half of an fp32, rounded to nearest-even. Same 8-bit exponent as fp32 (same range, no overflow surprises in training), but only 7 stored mantissa bits ≈ 2–3 significant decimal digits.

Sixteen bf16 bits of a weight: one sign, eight exponent, seven mantissa.

Preview: Q4_K_M

Q4_K_M is a llama.cpp GGUF recipe, not one format. Most tensors become Q4_K: weights are grouped in super-blocks of 256, split into 8 sub-blocks of 32; each weight is stored as a 4-bit code, each sub-block gets a 6-bit scale and 6-bit min, and each super-block one fp16 scale and one fp16 min-scale. That is 144 bytes per 256 weights = 4.5 bits per weight. The "_M" (medium) mix keeps some sensitive tensors at Q6_K (6.5625 bpw), notably lm_head and attn_v / ffn_down in a subset of layers, so the file averages a bit higher, roughly 4.8–4.9 bpw. Our 50.6 GB model would land around 15 GB.

w^=d⋅sj⋅q−dmin⁡⋅mj,q∈{0…15},sj,mj∈{0…63}\hat{w} = d \cdot s_j \cdot q - d_{\min} \cdot m_j, \quad q \in \{0 \ldots 15\}, \quad s_j, m_j \in \{0 \ldots 63\}
(g) Teaser: 256 weights → one Q4_K super-block

1/4 — 256 bf16 weights = 512 bytes (one Q4_K super-block's worth)

Synthetic weights. This toy uses plain per-sub-block min/max; real Q4_K also quantizes the 8 scales and mins themselves to 6 bits against fp16 super-scales and searches for error-minimizing scales. Lesson 17 does it properly.

A Q4_K super-block: 256 weights split into eight sub-blocks of 32.

Key numbers to remember

2 B
bytes/param in bf16fp32 = 4, Q8_0 ≈ 1.06, Q4_K = 0.5625
≈ 2N
FLOPs per token, forwardN = parameter count (dense)
≈ 2/3
of block params in the FFNSwiGLU with ffn ≈ 3.5–5× d; 81% in panel e
4.5
bits/weight, Q4_K block144 B per 256 weights
1·8·7
bf16 sign·exp·mantissa bitssame exponent range as fp32
256 / 32
k-quant super / sub-block8 sub-blocks per super-block
2·L·kv·hd·B
KV bytes per token≈ 0.2 MB in the panel e config
BW ÷ bytes
decode tokens/s ceilingweights are re-read every step

Self-check

1. In the panel (e) config, why are k_proj and v_proj 5× smaller than q_proj?

2. Does the length of your prompt change the size of the weights file or the per-token weight reads during decode?

3. Why does a Q4_K_M model generate faster than its bf16 parent on the same hardware, even though it does extra dequantization math?

4. Where do positions exchange information inside a block, and where don't they?

5. bf16 and fp16 are both 16 bits. What does bf16 trade, and why is that the right trade for training?

6. Using ≈ 2N FLOPs/token, what does generating one token cost for the 25.3B model?

Resources