Transformers from the Weights Up, Part 2: Dot products, the neuron, and shapes

Image 2 in numbers: with x = (0.3, −0.6, 0.8, 0.5, 0.9) and panel (b)'s starting weights (0.5, −0.4, 0.7, 0.2, −0.3), the products are 0.15, 0.24, 0.56, 0.10 and −0.27, so x·w = 0.78. Add b = 0.1 and z = 0.88; σ(0.88) ≈ 0.707. That is the whole neuron.

Three readings of one dot product

Pause

‖w‖ = 1.50

w

(1.41, 0.51)

Σ xᵢwᵢ

2.4·1.41 + 1.2·0.51 = 3.999

‖x‖‖w‖cos θ

2.683 · 1.50 · 0.994 = 3.999

Projection of x on ŵ

2.666

Cosine similarity

0.994

Scale w and the dot product scales with it, but the cosine does not. Attention scores and FFN "key" matches (Lesson 5) are raw dot products, so the norms of the vectors matter, which is one reason every branch reads a normalized copy of the stream.

x is fixed at (2.4, 1.2); w sweeps around the circle. The yellow segment is how far x reaches along w's direction. Multiply that by ‖w‖ and you have x·w. Negative when the angle passes 90°.

From neuron to matrix: y = Wx

Stack out neurons that read the same in inputs and their weight rows form a matrix W of shape [out, in]. There are two equally true ways to read the product, and you will use both.

PauseStep

Row 0 is the neuron from panel (b) with its starting weights; rows 1–3 are made-up toy weights. x is image 2's input. Each output is one row·x dot product: 5 multiplies + 5 adds ≈ 2·in FLOPs, so the whole thing is 2·in·out = 40 FLOPs.

PyTorch's convention: weights are [out, in], and y = xWᵀ

nn.Linear(in_features, out_features) stores weight with shape [out, in] and computes y = x @ W.T + b. That is why every tensor name in a Hugging Face checkpoint lists out first: q_proj is [4096, 4096] and k_proj is [1024, 4096] in Llama-3-8B. Rows are contiguous in memory, so one output neuron's weights sit side by side, and quantization blocks (Lessons 16 and 17) run along those rows, which is the input dimension.

Operation	Shape rule	Cost
Linear (x @ W.T)	[..., in] → [..., out]; weight [out, in]; all leading dims act as one batch of B·T rows	2·(B·T)·in·out FLOPs; reads in·out weights once
Batched matmul (A @ B)	[..., n, k] @ [..., k, m] → [..., n, m]; leading dims broadcast	2·(batch)·n·k·m FLOPs
view / reshape	Product of dims preserved: [B, T, 4096] ↔ [B, T, 32, 128]	Free when memory is contiguous
transpose / permute	Reorders axes: [B, T, 32, 128] → [B, 32, T, 128]	Free (stride change), but a later .contiguous() copies
Elementwise (SiLU, ⊙, +)	Shapes must broadcast	∝ number of elements, tiny vs matmuls
softmax(dim=−1)	Shape unchanged; each last-axis row sums to 1	A few ops per element

Following shapes through a real block (Llama-3-8B)

Real numbers from Meta-Llama-3-8B's config.json: hidden_size 4096, 32 layers, 32 attention heads and 8 key-value heads (so head_dim = 4096 / 32 = 128), intermediate_size 14336, vocab_size 128256, untied embeddings. Click a row or let it play. Change T to see the two regimes: weight matmuls grow linearly in tokens, attention scores grow with T².

PauseStep

B = 1B = 8

T = 1T = 128T = 2048T = 8192

x: residual stream in[1, 2048, 4096]—RMSNorm(x)[1, 2048, 4096]input_layernorm [4096]q = x·W_qᵀ[1, 2048, 4096]q_proj [4096, 4096]68.72 GFLOPk, v = x·W_kᵀ, x·W_vᵀ2 × [1, 2048, 1024]k_proj, v_proj [1024, 4096]34.36 GFLOPsplit heads (view + transpose)q [1, 32, 2048, 128] · k,v [1, 8, 2048, 128]no weights, no FLOPsscores = q·kᵀ / √128[1, 32, 2048, 2048]each K head serves 4 query heads34.36 GFLOPsoftmax(scores + mask) · v[1, 32, 2048, 128]no weights34.36 GFLOPmerge heads[1, 2048, 4096]view, no FLOPsh = x + o_proj(·)[1, 2048, 4096]o_proj [4096, 4096]68.72 GFLOPgate, up = Norm(h)·W_gᵀ, ·W_uᵀ2 × [1, 2048, 14336]gate_proj, up_proj [14336, 4096]481.04 GFLOPSiLU(gate) ⊙ up[1, 2048, 14336]elementwisey = h + down_proj(·)[1, 2048, 4096]down_proj [4096, 14336]240.52 GFLOP

Weight-matmul FLOPs, this layer

893.35 GFLOP = 2 · 2048 tokens · 218,103,808 weights

Attention-score FLOPs, this layer

68.72 GFLOP = 2 · 2 · B · 32 · T² · 128

Attention share of the block

7.1%

Score tensor, if materialized in bf16

268 MB per layer

Shapes from Meta-Llama-3-8B's config.json (hidden 4096, 32 query heads, 8 KV heads, head_dim 128, intermediate 14336). Blue = weight matmuls (scale with T); yellow = attention-score matmuls (scale with T²). Score FLOPs are counted for the full T×T square; a causal kernel can skip the masked half. Norms, SiLU, softmax and adds are omitted because they are tiny next to the matmuls.

Count the block's weights from the shapes: q and o are 4096 × 4096 = 16,777,216 each; k and v are 1024 × 4096 = 4,194,304 each; gate, up and down are 14336 × 4096 = 58,720,256 each. That sums to 218,103,808 per layer, × 32 = 6,979,321,856. Add embed_tokens and lm_head at 128256 × 4096 = 525,336,576 each, plus 266,240 norm gains, and you get 8,030,261,248: the "8B".

The embedding is a lookup, not a matmul.

Why high dimensions change the intuition

In 2-D, two random arrows are often nearly parallel. In 4096-D they almost never are: the cosine between random directions concentrates around 0 with standard deviation about 1/√d, which is 0.0156 at d = 4096. Two consequences you will meet again:

1. 1

   The √d in attention

   If q and k have independent unit-variance entries, q·k has variance d_k, so its std is √d_k (√128 ≈ 11.3). Dividing scores by √d_k brings them back to unit scale before the softmax, which would otherwise saturate into a near one-hot.

2. 2

   Room for many features

   A 4096-dimensional stream has only 4096 exactly orthogonal directions, but vastly more almost-orthogonal ones. Models exploit that to store more features than dimensions (superposition, Lesson 20), at the price of small interference in every dot product.

d = 2d = 16d = 128 (head_dim)d = 4096 (Llama-3-8B)

PlayResample

Measured std of cos θ

0.0925

1/√d

0.0884

Raw dot std (unnormalized, unit-variance entries)

√d = 11.3

Cosine similarity of 300 random Gaussian vector pairs, sampled live in your browser. Dashed lines mark ±1/√d. Press Play to keep resampling.

Lesson 2: key numbers

[out, in]

nn.Linear weight shapeforward is x @ W.T + b

2·in·out

FLOPs per token per linearone multiply + one add per weight

218.1M

weights per Llama-3-8B layer× 32 layers = 6.98B

8.03B

Llama-3-8B total parametersincl. 2 × 525.3M embed/lm_head

128

Llama-3-8B head_dim4096 / 32 heads; 8 KV heads

√d_k ≈ 11.3

attention score scalestd of q·k for unit-variance entries

1/√d

typical |cos| of random pair0.0156 at d = 4096

~15 GFLOP

Llama-3-8B forward per tokenbefore the T-dependent attention term

Lesson 2: self-check

Lesson 2: resources

• Dot products and duality — Essence of linear algebra, Ch. 9YouTube · 3Blue1Brown
• Matrix multiplication as composition — Essence of linear algebra, Ch. 4YouTube · 3Blue1Brown
• torch.nn.Linear (weight shape and y = xAᵀ + b)PyTorch docs