Deep Learning Architecture Notes: CNNs, Sequence Models, Attention & Transformers

From residual convolutional networks (ResNets) and 1x1 bottleneck projections to scaled dot-product attention in Transformers, modern deep learning architectures are built on a core set of inductive biases that stabilize gradient flow across deep computational graphs.

1. Why Residual Connections & Layer Normalization Enable Deep Stacks

By formulating blocks as $\mathcal{F}(x) + x$, identity skip connections create an unimpeded gradient highway during backpropagation, preventing vanishing gradients even across 100+ layers.

2. Scaled Dot-Product Attention & Key-Value Caching

In autoregressive Transformer inference, computing $\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$ benefits massively from caching historical Key ($K$) and Value ($V$) projections across decoding steps—reducing per-token complexity from quadratic to linear in sequence length.