Who how-to-train-your-gpt is for#
Python developers who use LLM APIs and want to understand the internals
You call OpenAI or Anthropic APIs daily but cannot explain what happens when a prompt is tokenized, how attention selects which context to use, or why temperature changes the output distribution. This guide takes you through every step with real numbers and runnable code so you can reason about model behavior rather than treating it as a black box.
Skip if:
Skip if you are already comfortable reading ML papers and implementing Transformer components from scratch. The guide is calibrated for developers starting from Python basics, not ML practitioners.
Students building intuition for Transformer architecture
You have read the "Attention Is All You Need" paper and understood most of it, but the gaps involve KV cache, RoPE, RMSNorm, and why modern LLMs diverged from the original 2017 architecture. This guide fills those gaps with worked numeric examples and annotated implementations that trace from the formula to running code.
Skip if:
Skip if your goal is deploying a model for a production use case. The trained model here is a learning artifact, not a production-quality language model.
ML engineers evaluating Transformer architecture tradeoffs
You need to choose between RoPE and learned positional encoding, or between RMSNorm and LayerNorm for a specific use case. This guide implements and explains each choice with citations to the source models that made each tradeoff, so you can make informed decisions rather than copying the latest architecture without understanding the reasoning.
Skip if:
Skip if your immediate need is fine-tuning an existing model for a downstream task. The main guide builds from scratch; LoRA and QLoRA are covered separately in the fine-tuning section.
Self-study learners preparing to read modern ML papers
You want to read the LLaMA 3, Mistral, or Qwen 2.5 technical reports but cannot follow the architecture sections. This guide covers every component those papers reference: RoPE, SwiGLU, pre-norm, AdamW, mixed precision, and grouped query attention. After finishing, you can read architecture tables in those papers with recognition rather than confusion.
Skip if:
Skip if you want a quick conceptual overview without writing code. Getting the full benefit requires reading approximately 3,500 lines of commented code across the 12 chapters.
The problem it solves#
Most resources for learning about large language models fall into two traps. API-only tutorials teach you to call model.generate() without explaining what generates the output. Academic papers give you the theory but assume prior ML expertise and skip directly to dense notation. The result is a gap that most developers cannot close: they use AI daily but cannot explain attention, cannot read an architecture paper, and cannot debug a training loop.
The deeper challenge is that paid training platforms like OpenAI's fine-tuning API, Google Cloud Vertex AI, and Azure Machine Learning abstract away exactly what you need to understand. You submit a dataset and get a model back. When something goes wrong, you have no visibility into what happened or how to fix it. For teams evaluating whether to self-host an open source model, fine-tune a base model, or rely on API providers, that knowledge gap creates real operational risk.
How it solves it#
12-chapter annotated textbook
A structured 7,500+ line guide covering tokenization, embeddings, positional encoding, attention, transformer blocks, training, and inference end to end. Each chapter follows a four-step format: plain-English analogy, worked numeric example, annotated code with WHAT and WHY comments on every line, and a diagram. You build every component yourself rather than importing it from a library.
LLaMA 3 style architecture built from scratch
Implements the techniques used in modern production LLMs: RoPE positional encoding, RMSNorm, SwiGLU activations, pre-norm, AdamW with cosine warmup, and mixed precision training. The completed model is 151M parameters at GPT-2 scale (768 dims, 12 layers), with a smaller 17M parameter CPU-friendly default (256 dims, 4 layers) for getting started without a GPU.
28 standalone topic explainers
Each major technique has its own dedicated deep-dive file: attention, RoPE, BPE tokenization, KV cache, flash attention, mixture of experts, speculative decoding, and more. Two narrative walkthroughs trace a single sentence through the entire model step by step. All explainers follow the same structure: what, where, why, when, and how, each with a runnable code example.
Runnable training script with configurable scale
A complete main.py runs the entire training pipeline in one command. The default tiny configuration (256 dims, 4 layers, 17M params) trains on CPU in minutes. The GPT-2 scale configuration (768 dims, 12 layers, 151M params) runs on GPU. Includes AdamW, cosine warmup, mixed precision, and gradient accumulation.
Jupyter notebooks and Google Colab support
Each of the 12 chapters has a companion Jupyter notebook with clean, runnable code stripped of prose explanations. A dedicated Colab notebook lets you train the model in the cloud with no local GPU required. Notebooks are self-contained: open any chapter's notebook and run all cells without completing prior notebooks first.
Fine-tuning guide covering LoRA and QLoRA
A separate fine-tuning section explains how to adapt a pre-trained base model without full retraining. Covers LoRA (Low-Rank Adaptation), QLoRA, and data preparation with a companion Jupyter notebook. This bridges the gap from understanding the architecture to customizing a model for a specific task using techniques found in production fine-tuning workflows.
Strengths and trade-offs#
Strengths
- MIT licensed with no prerequisites beyond Python basicsThe guide is MIT licensed: clone, run, and modify freely with no restrictions. The only prerequisite is Python basics (variables, functions, classes, pip install). No ML background, no calculus, and no prior PyTorch experience required. The guide introduces those concepts inline as you encounter them in the code.
- Modern LLaMA 3 architecture, not 2019-era GPT-2Most tutorials teach the 2019 GPT-2 architecture. This guide implements the techniques used in LLaMA 3, Mistral, and Qwen 2.5: RoPE, RMSNorm, SwiGLU, and pre-norm. Each architectural choice is traced to its source model in the glossary chapter, with the mathematical reasoning (variance argument for attention scaling, geometric intuition for RoPE) rather than just stating the technique.
- Every line annotated with WHAT and WHYTypical tutorial code shows you what to type. This codebase annotates every single line twice: what it does and why it is there. The attention chapter alone is 713 lines. That annotation level closes the gap between following along and understanding deeply enough to modify the architecture, adapt it to a new problem, or debug your own training run.
- CPU-friendly start, GPU-ready at full scaleThe tiny default configuration (256 dims, 4 layers, 17M params) trains on a standard laptop CPU in minutes, with no cloud GPU required to get started. Switching to the GPT-2 scale configuration (768 dims, 12 layers, 151M params) requires editing one config object in main.py. The Colab notebook provides a free GPU path when local hardware is not available.
Trade-offs
- -Educational resource, not a production training frameworkThis is a textbook with runnable code for learning purposes, not a framework for training production LLMs. It does not include distributed training, FSDP, DeepSpeed, or model parallelism for multi-GPU clusters. For training at scale on a large dataset, this guide serves as preparation before reaching for tools like Megatron-LM or a managed cloud training platform.
- -Full-scale model requires GPU accessThe GPT-2 scale configuration (151M params, 768 dims, 12 layers) requires a GPU. The guide notes this requirement but does not provide detailed infrastructure guidance for cloud GPU setup. Google Colab is mentioned as an option, but longer training runs may exceed Colab's free session time limits.
- -RLHF and instruction tuning are listed as future experimentsThe fine-tuning section covers LoRA and QLoRA for supervised adaptation, but RLHF (Reinforcement Learning from Human Feedback) and DPO (Direct Preference Optimization) are listed as next-steps experiments rather than implemented chapters. Developers who need instruction-tuning skills require additional resources beyond what is currently in the guide.
how-to-train-your-gpt vs alternatives#
How to Train Your GPT vs OpenAI Fine-Tuning API
OpenAI's fine-tuning API lets you adapt GPT-4o Mini or GPT-3.5 Turbo on a custom dataset with no infrastructure setup. You upload a JSONL file, trigger a training job, and get a model endpoint back. It is a managed service: you pay per token for the fine-tuning run and for inference, with no visibility into the training loop.
How to Train Your GPT is not a fine-tuning service. It is a 12-chapter guide that teaches you to build the training loop yourself, from tokenization to inference. If your goal is to ship a customized model quickly without understanding the internals, OpenAI's fine-tuning API is faster. If your goal is to understand what that API does and to build the skills to implement training pipelines yourself, this is the open source alternative.
| How to Train Your GPT | OpenAI Fine-Tuning | |
|---|---|---|
| Cost | Free (MIT licensed) | Pay-per-token |
| Setup | Clone and pip install | API key + JSONL upload |
| Purpose | Education, deep understanding | Production model customization |
| Architecture control | Full (you write the code) | None (black box) |
| Self-hostable | Yes | No |
How to Train Your GPT vs Hugging Face AutoTrain
Hugging Face AutoTrain is a managed fine-tuning service that trains models on your dataset without requiring you to write training code. It supports causal language model fine-tuning and classification tasks through a web interface or API, with pricing by compute hour on managed hardware.
How to Train Your GPT teaches you to write that training code yourself: AdamW optimizer, cosine learning rate warmup, mixed precision training, and gradient accumulation. AutoTrain is the practical choice when you need a fine-tuned model on a deadline. How to Train Your GPT is the educational alternative for developers who want to understand what AutoTrain does under the hood, and who may eventually build or maintain their own training infrastructure.
| How to Train Your GPT | Hugging Face AutoTrain | |
|---|---|---|
| Cost | Free | Paid (compute hours) |
| Purpose | Learning by building | Fine-tuning existing models |
| Architecture | LLaMA 3 style, built from scratch | Loads from HuggingFace Hub |
| Compute | CPU default, GPU for full scale | Managed GPU cloud |
| Self-hosting | Yes | Limited to higher tiers |
Install and self-host#
Clone the repository and install dependencies to run the training script locally.
```bash
git clone https://github.com/raiyanyahya/how-to-train-your-gpt.git
cd how-to-train-your-gpt
pip install -r requirements.txt
python main.py
```What it's built on#
- Languages
- Python
- Frameworks
- PyTorch
FAQ#
Does How to Train Your GPT require a GPU?
No. The default training configuration (256 dims, 4 layers, 17M params) runs on a CPU and completes in minutes. A GPU is only needed for the full GPT-2 scale configuration (151M params, 768 dims, 12 layers). The guide also provides a Google Colab notebook for cloud-based GPU training with no local setup required.
What prior knowledge do I need to work through this guide?
Python basics only: variables, functions, classes, and pip install. No machine learning background, no calculus, and no prior PyTorch experience is assumed. The guide introduces concepts like backpropagation, attention, and normalization from first principles as they come up in the code.
How does this compare to HuggingFace AutoTrain or OpenAI fine-tuning?
Those are managed services for fine-tuning existing models; they do not teach you how models work internally. How to Train Your GPT is an educational resource for building a Transformer from scratch. The two serve different goals: use this guide to build deep understanding of the architecture, then use managed services when you need a fine-tuned model deployed quickly.
Can I use this to fine-tune an existing language model like LLaMA?
The main guide builds and trains a GPT from scratch on a small dataset, not from a pre-trained checkpoint. A separate fine-tuning section in the repository covers LoRA and QLoRA for adapting existing base models. For production fine-tuning of LLaMA or similar models, you would use those techniques alongside tools like Hugging Face's PEFT library.
Is the model trained in this guide usable for real tasks?
No. The model is a learning artifact trained on a small dataset to demonstrate how training mechanics work. It is not a production-quality language model. The guide states explicitly that its purpose is education. After finishing, you have the skills to work with production models from HuggingFace or to contribute to open source LLM projects, but the trained artifact itself is a demo.
Similar open-source tools#
Soup
Fine-tune any LLM on a 4 GB GPU, one YAML config
open-notebook
Self-host private AI research notebooks
Ollama
Run large language models locally on Mac, Linux, or Windows
Unsloth
Train LLMs locally without code using a browser-based interface
Ploomber
Build reproducible Python data pipelines with DAG orchestration
TinyLLaMA
Compact 1.1B LLaMA model trained on 3 trillion tokens