All about GPUs

A learning journey, August 15, 2026–August 2027

Learning journal

Aug 15, 2026

I started All about GPUs, a year-long journal for documenting what I learn, the questions I run into, the experiments I try, and how my understanding changes. I also outlined the roadmap below. My starting point is the foundation: how GPU hardware turns massive parallelism into throughput and how software exposes that hardware to programmers.

New entries will appear here in reverse chronological order. Each entry will record what I studied, what I built, what surprised me, and what I still need to understand.

About this journey

From today through August 2027, I want to learn GPUs from the transistors and memory hierarchy up through kernels, runtimes, compilers, and real applications. This page is my public notebook for that process. The goal is not just to collect definitions, but to develop an intuition for why GPUs are designed the way they are, how to reason about performance, and how to write software that uses the hardware well.

I will learn by alternating between theory and practice: reading architecture and programming material, implementing small kernels, measuring them, inspecting the generated code, and explaining the results in my own words. When my understanding is incomplete or changes, I will document that too.

Roadmap

This roadmap follows the progression of Programming Massively Parallel Processors, Fourth Edition (PMPP), with practical extensions from the GPU MODE lectures. A linked topic opens a dedicated notes page so this overview can stay compact.

Part I — Fundamental concepts (August–October 2026)

• PMPP Chapters 1–3: Heterogeneous parallel computing and CUDA foundations — data parallelism, CUDA program structure, grids, and matrix multiplication. [GPU MODE Lectures 2–3]
• PMPP Chapters 4–6: GPU architecture, scheduling, memory locality, and performance considerations. [GPU MODE Lectures 4, 8, 16, and 37]

Part II — Parallel patterns (November 2026–January 2027)

• PMPP Chapters 7–9: Convolution, stencil computations, histograms, caching, and atomic operations.
• PMPP Chapters 10–12: Reduction, prefix sum, merge, divergence, and work efficiency. [GPU MODE Lectures 9, 20–21, and 24]

Part III — Advanced patterns and applications (February–April 2027)

• PMPP Chapters 13–15: Sorting, sparse matrix computation, and graph traversal.
• PMPP Chapters 16–19: Deep-learning kernels, application case studies, and computational thinking. [GPU MODE Lectures 11–12 and 23]

Part IV — Advanced practices (May–June 2027)

• PMPP Chapters 20–23: CUDA streams, heterogeneous clusters, dynamic parallelism, and host/device execution.
• Multi-GPU systems: Collective communication, NCCL, NVSHMEM, and overlapping communication with computation. [GPU MODE Lectures 17, 67, 70, and 78]

PyTorch and torch.compile (ongoing alongside Parts II–IV)

• Framework foundations: Tensors, autograd, modules, operators, dispatch, and the eager execution model.
• Compiler stack: PyTorch, torch.compile, and GPU code generation — TorchDynamo, FX graphs, AOTAutograd, TorchInductor, and Triton.
• Correctness and performance: Graph breaks, guards, recompilation, dynamic shapes, profiling, benchmarking, and custom operators. [GPU MODE Lectures 1, 6, 8, 18, 29, 39, and 42]

Applied GPU systems track (July–August 2027)

• Profiling and production CUDA: Integrating custom kernels, diagnosing bottlenecks, and building production-ready CUDA libraries. [GPU MODE Lectures 8, 10, and 16]
• Kernels and compilers: Triton, CUTLASS, fused kernels, Tensor Cores, SASS, and compiler-driven optimization. [GPU MODE Lectures 14–15, 18, 23, 29, 36–37, 42, 57, and 79]
• ML systems: Attention, quantization, inference, distributed training, and a measured final project. [GPU MODE Lectures 12–13, 30, 32, 35, and 39–40]

Projects and experiments

This section will collect small kernels, benchmarks, visual explanations, profiler traces, and larger projects as the journey develops. For every experiment, I plan to record the hardware and software setup, the question being tested, the result, and the lesson I can carry into the next project.

• Coming first: a small set of baseline kernels that compare serial CPU execution with parallel GPU execution.