<aside> 📌

Notes from implementing GPT from scratch following Karpathy's tutorial, with extra annotations on shapes and intuitions.

</aside>

Notebook + Code

Google Colab

https://github.com/archit-manek/gpt_scratch

Data Loading

• Reading the input.txt (Tiny Shakespeare).
• Building the vocabulary (sorted list of unique characters).
• Creating the stoi (string-to-integer) and itos mappings.

GPT Data Batching & Shapes

1. Tensors vs. Python Lists

• The Issue: PyTorch tensors (e.g., tensor(5)) are not the same as Python integers (5).
• The Fix: Use .tolist() to extract values before passing them to standard Python functions (like string decoders).

# Don't loop over the tensor directly
decode(out.tolist())

2. Key Dimensions (B, T, C)

In Mechanical Interpretability, always track the shapes: