Transformer Architecture

Multi-Head Self-Attention is the core innovation that allows the model to assess every token's relevance to every other token when creating contextualized representations. For each position, attention computes a weighted sum across all positions, with weights determined by learned compatibility between tokens.

Multiple attention heads operate in parallel, each specializing in different relationship types—some focusing on syntax, others on semantics, entity relationships, or discourse patterns. This allows the model to simultaneously capture diverse types of information.

Self-attention represents tokens as queries, keys, and values—conceptually similar to information retrieval where the model determines which input parts are most relevant to each position. When analyzing "bank" in "I deposited money in the bank," the model directly attends to "money" and "deposited" regardless of distance, recognizing the financial context.

This parallel computation captures long-range dependencies more effectively while leveraging hardware designed for parallel computation, addressing a fundamental limitation of previous sequential approaches.

Since attention has no inherent notion of token order, transformers add position information directly to token embeddings using sinusoidal functions of different frequencies. This allows understanding token positions without reintroducing sequential bottlenecks.

Position encodings embed each token's position in the sequence as a vector that gets added to the token's embedding. This ensures the model can distinguish between the same word appearing in different positions and understand concepts like word order and proximity.

The original implementation used fixed sinusoidal functions, though many modern variants use learned position embeddings that the model adjusts during training to capture position information optimally for its specific tasks.

These components address training challenges in deep networks. Residual connections create shortcuts for gradient flow by adding sublayer inputs to their outputs, while layer normalization stabilizes activations, enabling much deeper models.

Residual connections (or skip connections) help mitigate the vanishing gradient problem in deep networks by allowing gradients to flow directly through the network. Each sublayer's output is added to its input, creating direct paths for backpropagation.

Layer normalization standardizes the inputs to each layer, reducing internal covariate shift and stabilizing the learning process. This normalization operates across the feature dimension for each token, helping maintain consistent scale throughout the network regardless of depth.

Between attention layers, point-wise feed-forward networks with non-linear activations provide additional representational capacity, transforming attention-weighted information through learned projections.

Each feed-forward network consists of two linear transformations with a ReLU activation in between, applied independently to each position. While attention layers capture interactions between positions, these feed-forward layers process each position's information independently.

Despite their simplicity, these networks significantly increase the model's capacity to represent complex functions and are often where much of a transformer's parameter count resides. They can be viewed as position-wise fully-connected layers that transform each token's representation.

BERT (Bidirectional Encoder Representations from Transformers), introduced by Google AI in 2018, represented a watershed moment in NLP by applying the transformer architecture to create deep bidirectional representations. Unlike previous models that processed text either left-to-right or right-to-left, BERT simultaneously considers both directions to develop rich contextual understanding.

The key innovation in BERT was its training approach. Using masked language modeling, BERT randomly hides words in the input and tasks the model with predicting these masked tokens based on their context. This forced the model to develop deep bidirectional representations capturing subtle meanings and relationships.

BERT's pre-training on massive text corpora (including Wikipedia and books) allowed it to develop general language understanding capabilities that could be fine-tuned for specific downstream tasks with relatively small amounts of labeled data. This transfer learning approach dramatically reduced the data requirements for achieving state-of-the-art performance across NLP tasks.

The impact of BERT was immediate and profound. It shattered performance records across a wide range of NLP benchmarks, including question answering, sentiment analysis, textual entailment, and named entity recognition. Its architecture spawned numerous variants and extensions, including RoBERTa, ALBERT, and DistilBERT, each optimizing different aspects of the original design.

GPT (Generative Pre-trained Transformer) models, developed by OpenAI beginning in 2018, applied the transformer architecture to autoregressive language modeling—predicting the next token given previous tokens. This approach created powerful text generation capabilities while still enabling effective transfer learning to downstream tasks.

Unlike BERT's bidirectional approach, GPT models use a causal attention mask that prevents tokens from attending to future positions, preserving the left-to-right generation capability. This unidirectional approach creates models that excel at text generation tasks while still developing useful representations for understanding.

The GPT series has followed a clear scaling trajectory, with each generation substantially increasing parameter count, training data, and capabilities. GPT-1 (2018) demonstrated the potential of the approach, GPT-2 (2019) showed surprisingly coherent text generation, and GPT-3 (2020) revealed emergent capabilities like few-shot learning that weren't explicitly trained.

GPT models have profoundly influenced NLP by demonstrating that a single architecture and training objective can develop general-purpose language capabilities applicable across diverse tasks. Rather than creating specialized architectures for different problems, GPT showed that scaled autoregressive models can perform translation, summarization, question-answering, and even reasoning through simple text prompting.

Transformer-based encoder-decoder models combine elements of both BERT-like bidirectional encoders and GPT-like autoregressive decoders to create architectures optimized for sequence-to-sequence tasks like translation, summarization, and question answering.

In these models, the encoder processes the entire input sequence bidirectionally to create context-rich representations, while the decoder generates the output sequence autoregressively, attending both to previously generated tokens and the encoded input through cross-attention mechanisms.

T5 (Text-to-Text Transfer Transformer) exemplifies this approach by framing all NLP tasks as text-to-text problems, using a consistent encoder-decoder architecture regardless of whether the task involves classification, translation, or question answering. This unified approach allows a single model to handle diverse tasks through appropriate prompting.

BART (Bidirectional and Auto-Regressive Transformer) similarly combines a bidirectional encoder with an autoregressive decoder but introduces more sophisticated pre-training techniques like text infilling, where spans of text are replaced with a single mask token. This approach created models particularly effective for tasks requiring both understanding and generation capabilities.