Transformers from the Weights Up, Part 3: Activation functions and nonlinearity

Composition of affine maps is affine. Worse than useless at scale: Llama-3-8B's up_proj [14336, 4096] followed by down_proj [4096, 14336] with nothing in between would be 117.4M weights computing a single 4096 × 4096 matrix (16.8M weights). Insert an elementwise nonlinearity between them and the collapse is gone.

Image 3 is the picture this lesson starts from: a weighted sum, then a sigmoid.

The activation zoo

GELU and SiLU are both "the input times a soft on/off probability": Φ(z) and σ(z) are similar S-curves, and σ(1.702z) is a classic close approximation to Φ(z). Many implementations of GELU use the tanh form below, which differs from the exact z·Φ(z) by at most about 4.7 × 10⁻⁴ (checked numerically over [−6, 6]). That gap is why checkpoints record which variant they were trained with (gelu vs gelu_new / gelu_pytorch_tanh in Hugging Face configs).

Saturation and the vanishing gradient

Backprop (Lesson 6) multiplies local derivatives along the path from the loss to each weight. Every elementwise activation contributes a diagonal Jacobian, diag(f′(z)), at every layer it passes through. For sigmoid that factor is at most 0.25 and far smaller once |z| grows, so the signal reaching early layers shrinks geometrically. Ten sigmoid layers at the very best cap the gradient at 0.25¹⁰ ≈ 9.5 × 10⁻⁷ of what it was; in practice the weights' own factors and saturated units make it worse or erratic.

What nonlinearity buys: folding space

A ReLU layer partitions input space into regions by the sign pattern of W₁x + b₁ and is affine inside each region. More units mean more kinks, and enough kinks fit any continuous function on a bounded domain to any tolerance (the universal approximation theorem, Cybenko 1989 for sigmoids, Hornik 1991 more generally). The theorem says nothing about how many units you need or whether gradient descent will find them; depth helps because each layer can fold the regions created by the previous one.

Gating: from GLU to SwiGLU

A gated linear unit (Dauphin et al., 2016) computes two projections of the same input and multiplies them, using a squashed version of one as a per-unit valve on the other. Shazeer (2020) swapped in different gate functions inside the transformer FFN and found the gated variants beat the ReLU and GELU baselines at equal parameter count. The Swish (SiLU) version became the default. His paper's explanation, verbatim: "We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence."

Why a gate helps, as intuition rather than proof: the output of hidden unit i is now a product of two learned linear features of x, so the FFN computes quadratic-style interactions that a ReLU unit (one feature, clipped) cannot. The gate also gives a smooth, nonzero gradient for slightly negative inputs, so units are much less prone to dying.

That is the code path in Meta's reference model.py: hidden_dim = int(2 · hidden_dim / 3), then multiplied by ffn_dim_multiplier, then rounded up to a multiple of multiple_of. The result: 3 × 4096 × 14336 = 176,160,768 FFN weights per layer, 80.8% of the block's 218.1M from Lesson 2.

Engineering consequences you will see later

1. 1

   FLOP-free, not byte-free

   SiLU and ⊙ are a few operations per element of a [T, 14336] tensor, negligible next to the matmuls. But a naive implementation writes gate and up to memory and reads them back, so fast kernels fuse SiLU(gate) ⊙ up into one pass, or into the matmul epilogue.

2. 2

   Gate and up can be one matmul

   They read the same input, so many engines concatenate gate_proj and up_proj into a single [2 · 14336, 4096] weight and split the output. llama.cpp's GGUF keeps them as separate ffn_gate and ffn_up tensors; vLLM merges them at load time.

3. 3

   Products make outliers

   Multiplying two projections can produce occasional very large values in down_proj's 14336-wide input. Fishman et al. (2024) traced FP8 training instabilities on trillion-token runs to this SwiGLU amplification. It is one reason quantization recipes handle ffn_down carefully (the Q4_K_M mix in Lesson 1 promotes some ffn_down tensors to Q6_K; Lessons 16–17 revisit).

4. 4

   Activations are where sparsity lives

   After a ReLU, many hidden units are exactly zero for a given token; after SiLU they are merely small. That difference matters for interpretability (Lesson 5's key–value view) and for sparsity-exploiting inference tricks.

Lesson 3: key numbers

0.25

max slope of sigmoidσ′(0) = σ(0)(1 − σ(0))

9.5e-7

0.25¹⁰best-case gradient through 10 sigmoids

−0.278

SiLU minimumat z ≈ −1.278

−0.170

GELU minimumat z ≈ −0.752

3

FFN matrices with SwiGLUgate, up, down; no biases

⅔

d_ff shrink for equal params3 · d · 8d/3 = 2 · d · 4d

14336

Llama-3-8B d_ff⌊⅔·16384⌋ × 1.3, rounded up

176.2M

FFN weights per layer80.8% of the 218.1M block

Lesson 3: self-check

Lesson 3: resources

• GLU Variants Improve Transformer (Shazeer, 2020)arXiv
• Gaussian Error Linear Units (Hendrycks & Gimpel)arXiv
• Building makemore Part 3: Activations & Gradients, BatchNormYouTube · Andrej Karpathy
• Llama 3 reference model.py (FeedForward class)GitHub · meta-llama