Nanochat

59.9K stars
MIT
Active recently
Last commit 56 days ago

Last verified Nov 3, 2026

Build GPT-class LLMs end-to-end on a single GPU node for under $100, with MIT-licensed Python covering tokenization, pretraining, finetuning, and inference.

Visit website GitHub59.9K

Read the full breakdownWho it is for, how it works, trade-offs, the quick start and the FAQ

who it's for

Who Nanochat is for#

ML researchers reproducing GPT-scale training

Nanochat's speedrun script packages the full recipe to train a GPT-2-class model in one shell script. Researchers run runs/speedrun.sh on an 8-GPU node to reproduce the baseline, then modify individual stages (data pipeline, optimizer, architecture) to study their effect on the CORE metric and validation bits-per-byte. The leaderboard tracks which changes improve time-to-GPT-2.

Skip if:

You need a model larger than GPT-2 scale (depth 26 equivalent). Nanochat targets micro models accessible for under $1,000; larger model training requires elastic multi-node distributed training, which nanochat does not support.

Engineers fine-tuning a base model for specific tasks

After pretraining, nanochat's SFT and RL scripts (chat_sft.py, chat_rl.py) let you fine-tune on custom datasets to add task-specific abilities. The codebase includes a Python code execution tool (execution.py) as an example of an added ability. Evaluation tasks covering MMLU, GSM8K, ARC, and HumanEval let you measure progress.

Skip if:

You want to adapt an existing large foundation model (LLaMA, Mistral, Gemma, etc.) using LoRA or PEFT. Nanochat trains from scratch; for fine-tuning pretrained models, tools like HuggingFace trl or Unsloth are a better fit.

Scaling-laws researchers running depth sweeps

The runs/scaling_laws.sh and runs/miniseries.sh scripts generate a series of compute-optimal models at different depths. Setting depths from 6 to 26 produces models at GPT-1 through GPT-2 scale with automatically tuned hyperparameters. Wandb integration tracks validation bits-per-byte (val_bpb), CORE score, VRAM utilization, and model FLOPS utilization across the sweep.

Skip if:

You need multi-node scaling data beyond a single 8-GPU server. Nanochat runs on a single node only; scaling-laws studies requiring hundreds of GPUs or petaFLOP budgets need a different training framework.

the problem

The problem it solves#

Training a language model from scratch has historically required either a large research lab budget or deep distributed-training expertise. The original GPT-2 training run cost approximately $43,000 in 2019, and reproduction efforts faced a wall of framework complexity, with giant configuration objects, opaque model factories, and dozens of hyperparameters that interact in non-obvious ways.

For researchers and engineers who want to study how each training stage affects model behavior, existing tools present a difficult tradeoff. Large, heavily abstracted frameworks hide the training loop behind configuration APIs. Low-level PyTorch requires writing the entire pipeline from scratch. Neither supports fast iteration on individual stages.

how Nanochat solves it

How it solves it#

Single --depth complexity dial

Training a compute-optimal model requires setting only --depth (the number of transformer layers). Nanochat automatically derives transformer width, number of attention heads, learning rate schedule, training horizon, and weight decay from that single integer. GPT-2-class models land around depth 24-26; researchers can sweep depths from 12 upward to produce the full miniseries of compute-optimal models at various sizes.

Full pipeline from tokenization to RL inference

The codebase covers every major LLM stage in single, readable Python files. BPE tokenizer training lives in tok_train.py, base model pretraining in base_train.py, supervised finetuning in chat_sft.py, reinforcement learning in chat_rl.py, and CLI inference in chat_cli.py. Each stage is independently runnable without touching the others.

Explicit mixed-precision control via COMPUTE_DTYPE

Precision is managed through a global COMPUTE_DTYPE environment variable rather than torch.amp.autocast. On H100/A100 hardware the default is bfloat16 compute with fp32 weights stored for optimizer precision. Embeddings store in COMPUTE_DTYPE to reduce memory. Override per-run with the NANOCHAT_DTYPE environment variable without touching model code.

GPT-2 speedrun leaderboard

Nanochat maintains a public leaderboard for the wall-clock time needed to train a model to GPT-2 CORE score (0.2565) on an 8xH100 node. The current record is 1.65 hours, down from the initial 3.04-hour baseline in January 2026. The runs/speedrun.sh script always reflects the reference training recipe, giving contributors a reproducible baseline.

Distributed training with automatic single-GPU fallback

Multi-GPU runs use torchrun with all-reduce for gradient synchronization across 8xH100 or 8xA100 nodes. Omitting torchrun switches the code automatically to gradient accumulation on a single GPU, producing approximately identical results at 8 times the wall-clock time. No code changes are required to switch between the two modes.

strengths · trade-offs

Strengths and trade-offs#

Strengths

GPT-2-class training for under $100GPT-2-class capability (CORE score above 0.256525) costs approximately $48 on an 8xH100 node, about 2 hours at current GPU rates. On spot instances the cost drops to around $15. The original 2019 OpenAI GPT-2 training run cost approximately $43,000; nanochat achieves equivalent results through modern hardware efficiency, improved data quality (NVIDIA ClimbMix), and optimizer advances including the Muon optimizer.
Minimal, auditable codebaseThe core library (nanochat/) is twelve files, with no giant configuration objects or model factory abstractions. Every training stage is a single Python script under scripts/. A researcher can read the entire codebase in a session and understand every architectural decision, from the custom AdamW + Muon optimizer in optim.py to the KV-cache inference engine in engine.py.
Compute-optimal hyperparameter schedulingSetting one integer (--depth) produces a model where all other hyperparameters (transformer width, head count, learning rate schedule, training horizon, weight decay) are automatically tuned for compute-optimal results. This means any candidate improvement to the codebase is principled enough to work across all model sizes, not just GPT-2 scale.

Trade-offs

-Requires a dedicated 8xH100 or 8xA100 node for the full runThe full GPT-2 speedrun requires an 8xH100 or 8xA100 GPU node with 80GB VRAM per GPU. Single-GPU runs work but take roughly 8 times longer (about 12-16 hours). Machines with less than 80GB VRAM per GPU need a reduced --device-batch-size, which can affect training quality. CPU and MPS paths are available but will not produce GPT-2-grade results in reasonable time.
-No hosted version and no production serving infrastructureNanochat covers training and CLI inference only. There is no REST API server, no model-serving layer, and no checkpointing-resume from arbitrary hardware failures. Teams who want to deploy a trained model into a production serving stack will need to export the checkpoint and set up inference separately. The model architecture is a standard GPT transformer, so it can be loaded with custom serving code.
-Active development with evolving training defaultsWith 28 open issues and commits as recent as September 2026, nanochat is under active development. Training scripts and hyperparameter defaults in the reference speedrun have changed multiple times since the initial October 2025 launch. Researchers who build on nanochat should pin to a specific commit to ensure reproducibility.

versus alternatives

Nanochat vs alternatives#

Nanochat vs AWS SageMaker Training

AWS SageMaker Training is a managed ML training service where you supply a training script, and Amazon provisions and manages the compute cluster. Nanochat runs on any GPU node you provision yourself.

Feature	Nanochat	AWS SageMaker Training
License	MIT	Proprietary
Self-hosting	Yes, any GPU node	No, AWS-managed only
Pretraining from scratch	Yes	Limited; primarily fine-tuning
Code transparency	Full source	Managed training loop
GPT-2 scale cost	~$48 on 8xH100 node	Per-hour compute + platform fees
Multi-node distributed	No (single-node only)	Yes

Nanochat is the better choice for researchers who need full code visibility, train from scratch on raw text data, or want to iterate on optimizer and architecture without framework abstractions. SageMaker Training is worth considering when you need fault-tolerant multi-node distributed training, already run workloads on AWS infrastructure, or need managed checkpointing and job orchestration for production training runs.

Nanochat vs OpenAI Fine-tuning API

OpenAI's fine-tuning API lets you adapt GPT-3.5-turbo or GPT-4o on custom datasets for a per-token training fee, with no self-hosting option. Nanochat trains models from scratch on raw text data.

Feature	Nanochat	OpenAI Fine-tuning API
License	MIT	Proprietary
Self-hosting	Yes	No
Trains from scratch	Yes	No (adapts existing models)
Code access	Full	None
Model ownership	You own the weights	OpenAI hosts and runs inference

Nanochat is the better fit when you need a fully self-hosted model whose weights you own, want to train on data that cannot leave your infrastructure, or need to study the full training pipeline. OpenAI's fine-tuning API is simpler when you only need to adapt an existing model's behavior for a task and do not need self-hosting, code access, or pretraining from scratch.

install · quick start

Quick start#

bash

Install nanochat by cloning the repository and syncing dependencies with uv on a CUDA-capable GPU node.

```bash
git clone https://github.com/karpathy/nanochat
cd nanochat
uv sync --extra gpu
source .venv/bin/activate
```

tech stack · detected from GitHub

What it's built on#

Languages: Python

frequently asked

FAQ#

How much does it cost to train a GPT-2-class model with nanochat?

Approximately $48 on an 8xH100 node (about 2 hours at roughly $24/hr for the node), or closer to $15 on spot instances. The original 2019 OpenAI GPT-2 training cost approximately $43,000; the cost reduction comes from both improved hardware efficiency and seven years of algorithmic advances. The runs/speedrun.sh script reflects the current reference recipe.

What GPU hardware does nanochat require?

The full GPT-2 speedrun requires an 8xH100 or 8xA100 node with 80GB VRAM per GPU. Single-GPU runs work with the same code by omitting torchrun; results are approximately identical but take about 8 times longer. GPUs with less than 80GB VRAM need a reduced --device-batch-size. CPU and Apple Silicon (MPS) paths are available for small experiments but will not produce GPT-2-grade results.

Does nanochat support fine-tuning existing models like LLaMA or Mistral?

No. Nanochat trains models from scratch, starting with pretraining on raw text data. It does not provide a path for loading and fine-tuning existing HuggingFace checkpoints. For LoRA or PEFT fine-tuning of existing foundation models, other tools such as the HuggingFace trl library are a better fit.

Can nanochat run on a single GPU?

Yes. Omitting torchrun from the training command switches the code automatically to gradient accumulation on a single GPU. Results are approximately identical to multi-GPU runs, but training takes about 8 times longer. For the GPT-2 speedrun, that means roughly 12-16 hours instead of 1.5-2 hours. Less than 80GB VRAM requires reducing --device-batch-size.

What license does nanochat use and what can I do with it?

MIT license. You can use, modify, distribute, and build commercial products on top of the code without restriction. The only requirement is preserving the license text and copyright notice. Academic use has a BibTeX citation in the README for attribution.

also worth a look

Similar open-source tools#

Hermes Agent

Featured

Alternative toGrok Bot

Self-hosted AI agent: persistent memory, multi-channel chat, model choice.

259,499 GitHub stars
Licence MIT

Deepagents

Alternative toLangGraph Platform

Batteries-included agent library for complex, multi-step AI work.

30,917 GitHub stars
Licence MIT

Mlx Serve

Alternative toLM Studio

Local LLM server for Apple Silicon, faster than LM Studio

2,206 GitHub stars
Licence MIT

TensorFold

InAI & Machine Learning

Local LLM inference for Apple Silicon and NVIDIA GPUs, no Python.

1,567 GitHub stars
Licence Apache-2.0

Skillshare

InDeveloper Tools

AI coding setup manager for skills, agents, MCP, and hooks

2,872 GitHub stars
Licence MIT

Agent Desktop

InAI & Machine Learning

Desktop computer use for AI agents via OS accessibility tree

3,850 GitHub stars
Licence Apache-2.0

by category

Open source tools in these categories#

453

12,889 GitHub stars
Licence AGPL-3.0
#Email

rakazo

Alternative toGrok Bot

AI teammates you own: your keys, your model, your machine.

6,568 GitHub stars
Licence Apache-2.0
#AI Agents

SnitchFeed

Live monitoring on LinkedIn, X, Reddit & more. Agentic social listening for startups.

Marketing & Growth

Advertise your product

The projects we add and every new guide. Free, unsubscribe any time.

Find open source alternatives to paid software

Nanochat vs AWS SageMaker Training

Feature	Nanochat	AWS SageMaker Training
License	MIT	Proprietary
Self-hosting	Yes, any GPU node	No, AWS-managed only
Pretraining from scratch	Yes	Limited; primarily fine-tuning
Code transparency	Full source	Managed training loop
GPT-2 scale cost	~$48 on 8xH100 node	Per-hour compute + platform fees
Multi-node distributed	No (single-node only)	Yes

Nanochat vs OpenAI Fine-tuning API

OpenAI's fine-tuning API lets you adapt GPT-3.5-turbo or GPT-4o on custom datasets for a per-token training fee, with no self-hosting option. Nanochat trains models from scratch on raw text data.

Feature	Nanochat	OpenAI Fine-tuning API
License	MIT	Proprietary
Self-hosting	Yes	No
Trains from scratch	Yes	No (adapts existing models)
Code access	Full	None
Model ownership	You own the weights	OpenAI hosts and runs inference

Who Nanochat is for#

ML researchers reproducing GPT-scale training

Skip if:

Engineers fine-tuning a base model for specific tasks

Skip if:

Scaling-laws researchers running depth sweeps

Skip if:

The problem it solves#

How it solves it#

Single --depth complexity dial

Full pipeline from tokenization to RL inference

Explicit mixed-precision control via COMPUTE_DTYPE

GPT-2 speedrun leaderboard

Distributed training with automatic single-GPU fallback

Strengths and trade-offs#

Strengths

Trade-offs

Nanochat vs alternatives#

Nanochat vs AWS SageMaker Training

Nanochat vs OpenAI Fine-tuning API

Quick start#

What it's built on#

FAQ#

Similar open-source tools#

Open source tools in these categories#

AI & Machine Learning

LLMOps & AI Tooling

Developer Tools

Find open source alternatives to paid software

Who Nanochat is for#

ML researchers reproducing GPT-scale training

Skip if:

Engineers fine-tuning a base model for specific tasks

Skip if:

Scaling-laws researchers running depth sweeps

Skip if:

The problem it solves#

How it solves it#

Single --depth complexity dial

Full pipeline from tokenization to RL inference

Explicit mixed-precision control via COMPUTE_DTYPE

GPT-2 speedrun leaderboard

Distributed training with automatic single-GPU fallback

Strengths and trade-offs#

Strengths

Trade-offs

Nanochat vs alternatives#

Nanochat vs AWS SageMaker Training

Nanochat vs OpenAI Fine-tuning API

Quick start#

What it's built on#

FAQ#

Similar open-source tools#

Open source tools in these categories#

AI & Machine Learning

LLMOps & AI Tooling

Developer Tools

Find open source alternatives to paid software