Heterogeneous Parallel Computing

Why GPUs trade latency for throughput

Learning goals

This page will document my introduction to heterogeneous parallel computing and the CUDA programming model. I want to understand the ideas well enough to explain not only how to launch a kernel, but why a GPU executes the work differently from a CPU.

• Explain the roles of the host and device in a heterogeneous program.
• Distinguish latency-oriented CPU execution from throughput-oriented GPU execution.
• Describe data parallelism and identify problems that map well to a GPU.
• Understand CUDA threads, blocks, grids, kernel launches, and device memory.
• Implement and test vector addition as a first CUDA kernel.

Notes

My notes will go here as I work through the reading and lectures. I will use this section for explanations, diagrams, code excerpts, and questions that need more study.

Heterogeneous computing

Notes to come.

Data parallelism

Notes to come.

The CUDA programming model

Notes to come.

Experiments

For each experiment I will record the hardware and software environment, the question being tested, the implementation, correctness checks, measurements, and takeaways.

• First experiment: CPU and CUDA implementations of vector addition.
• Follow-up: map a two-dimensional data set onto a CUDA grid.

Takeaways and open questions

This section will be updated after completing the reading and experiments.

Resources

• Programming Massively Parallel Processors, Fourth Edition , Chapters 1–3.
• GPU MODE Lecture 2: PMPP Chapters 1–3 recap .
• GPU MODE Lecture 3: Getting Started With CUDA .