Andrej Karpathy

Neural Networks: Zero to Hero

Build neural networks from scratch in code, up to a GPT and its tokenizer, for programmers with intro-level maths.

Taught by Andrej KarpathyFreeVerified 11 Oct 2026

Open the course

From The spelled-out intro to neural networks and backpropagation: building micrograd to Let's build the GPT Tokenizer: 8 videos
What it uses: PyTorch
Before you startSolid programming (Python) and intro-level math (e.g. derivative, gaussian).: Any background
8 video lectures, about 14.5 hours: Self-paced

Neural Networks: Zero to HeroFreeOpen the course

What you’ll learn

Learn how backpropagation trains a neural network by building a tiny autograd engine, micrograd, by hand.
Build character-level language models, from a simple bigram model up to an MLP and a WaveNet-style network.
Look inside deep networks at activations and gradients, and see why BatchNorm makes training easier.
Build a GPT model from scratch, following the 'Attention is All You Need' paper and GPT-2 and GPT-3.
Build the GPT tokenizer with Byte Pair Encoding and see how tokenization causes odd LLM behavior.

Skills you’ll gain

PythonNeural networksTokenizersDeep Learning

Tools it uses

PyTorch.

Details to know

No certificateAndrej Karpathy lists none
Free to takeNo fee on karpathy.ai
Taught in EnglishNo other languages listed
VerifiedOct 2026

Lessons

8 video lectures, about 14.5 hours

A video course by Andrej Karpathy where you build neural networks from scratch in Python, coding along with him. It starts with backpropagation on single numbers and works up to a GPT and the tokenizer that feeds it.

Most of the course follows makemore, a character-level language model that grows from a bigram counter into a deeper network over five videos. Each video's code sits in a notebook, Colab file or companion repo, and most descriptions list exercises.

The videos, in order:

The spelled-out intro to neural networks and backpropagation: building micrograd: Build a tiny autograd engine and a small neural net, and see how backpropagation trains it.
The spelled-out intro to language modeling: building makemore: Build a bigram character model by counting, then as a neural net, using torch.Tensor.
Building makemore Part 2: MLP: Turn makemore into a multilayer perceptron and learn about learning rates, data splits and overfitting.
Building makemore Part 3: Activations & Gradients, BatchNorm: Inspect activations and gradients in a deeper net and add batch normalization to steady training.
Building makemore Part 4: Becoming a Backprop Ninja: Write the backward pass by hand through every layer instead of calling loss.backward().
Building makemore Part 5: Building a WaveNet: Make the network deeper with a tree-like, WaveNet-style structure and see how torch.nn works.
Let's build GPT: from scratch, in code, spelled out.: Build a GPT with self-attention, following Attention is All You Need and GPT-2 and GPT-3.
Let's build the GPT Tokenizer: Build the Byte Pair Encoding tokenizer used by GPT models and see how tokens cause odd LLM behavior.

Applied learning project

The course is meant to be coded along, not just watched, and each video leaves you work to do.

Notebooks: Rebuild the code from each video; the notebooks and Colab files are linked from the repo and the descriptions.
Exercises: Try the exercises in the descriptions, such as a trigram model, beating the video's loss or folding BatchNorm into a layer.
Backprop by hand: Work through the Part 4 Colab exercise yourself and check the video only when stuck.
Your own GPT data: Train the GPT on a dataset you choose, or pretrain on a large set and fine-tune on tiny Shakespeare.

By the end you can write and train a small GPT and its tokenizer from scratch in PyTorch, and explain how gradients move through it.

The spelled-out intro to neural networks and backpropagation: building microgradVideo 1 • 2 h 25 minVideo details

What you’ll learn

Build micrograd, a tiny autograd engine, and use it to backpropagate through math expressions.
Train a small neural network with it and watch the loss fall step by step.
Test yourself with the Colab exercise linked in the video description.

The spelled-out intro to language modeling: building makemoreVideo 2 • 1 h 57 minVideo details

What you’ll learn

Build a bigram character-level language model, first by counting and then as a simple neural net.
Learn how torch.Tensor works and why its details matter for fast evaluation.
Sample from the model and score it with the negative log likelihood loss.

Building makemore Part 2: MLPVideo 3 • 1 h 15 minVideo details

What you’ll learn

Build a multilayer perceptron that predicts the next character from the ones before it.
Tune the learning rate and other settings, and split data into train, dev and test sets.
Spot underfitting and overfitting, then try to beat the video's validation loss.

Building makemore Part 3: Activations & Gradients, BatchNormVideo 4 • 1 h 55 minVideo details

What you’ll learn

Inspect forward activations and backward gradients to see when a deep network is badly scaled.
Use plots and statistics to check the health of a network during training.
Add batch normalization and see why it makes deep networks easier to train.

Building makemore Part 4: Becoming a Backprop NinjaVideo 5 • 1 h 55 minVideo details

What you’ll learn

Backpropagate by hand through cross entropy, linear layers, tanh, BatchNorm and the embedding table.
Work with gradients on whole tensors rather than single numbers.
Do the Colab exercise alongside the video and pause whenever you get stuck.

Building makemore Part 5: Building a WaveNetVideo 6 • 56 minVideo details

What you’ll learn

Deepen the MLP into a tree-shaped network similar to DeepMind's 2016 WaveNet.
Learn what torch.nn does under the hood.
Follow a typical development loop: reading docs, tracking tensor shapes and moving from notebook to code.

Let's build GPT: from scratch, in code, spelled out.Video 7 • 1 h 56 minVideo details

What you’ll learn

Build a GPT step by step: a baseline language model, self-attention, then the full Transformer.
Follow the Attention is All You Need paper and OpenAI's GPT-2 and GPT-3 as you go.
Train it on tiny Shakespeare text and see how it relates to ChatGPT.

Let's build the GPT TokenizerVideo 8 • 2 h 13 minVideo details

What you’ll learn

Build the Byte Pair Encoding tokenizer used in OpenAI's GPT series from scratch.
Write encode() and decode() to move between strings and tokens.
Trace common LLM quirks back to tokenization.