Transformer Implementation (Attention Is All You Need)

This project implements the Transformer model as described in the paper "Attention Is All You Need". The model is built using PyTorch and includes an encoder-decoder architecture for sequence-to-sequence tasks such as machine translation.

Features

• Self-Attention Mechanism: Implements multi-head self-attention for efficient sequence learning.
• Positional Encoding: Adds positional embeddings to handle sequential data.
• Layer Normalization & Dropout: Ensures stable training and prevents overfitting.
• Masking Mechanisms: Implements padding and look-ahead masks for proper training.
• Scalable Architecture: Supports configurable depth, embedding size, and number of attention heads.

Architecture Overview

The Transformer consists of:

1. Encoder: A stack of self-attention and feed-forward layers.
2. Decoder: Similar to the encoder but includes an additional cross-attention layer to process encoded representations.
3. Full Model: Combines the encoder and decoder to process input and generate output sequences.

Installation

Prerequisites

• Python 3.8+
• PyTorch

Setup

1. Clone the repository:

   git clone https://github.com/your-repo/transformer-pytorch.git
   cd transformer-pytorch

2. Install dependencies:

   pip install torch numpy

Usage

Run the example script to test the model:

python main.py

The script will generate a sample output tensor to verify the implementation.

Code Structure

├── main.py  # Main script to define and run the Transformer model
├── README.md  # Project documentation

Example Output

The model will output a tensor of shape (batch_size, sequence_length, vocab_size), representing predicted probabilities for each token:

torch.Size([2, 7, 10])

References

• Vaswani et al., "Attention Is All You Need" (2017): https://arxiv.org/abs/1706.03762