A GPU, or Graphics Processing Unit, is a specialized electronic circuit designed to rapidly manipulate and alter memory to accelerate the creation of images in a frame buffer intended for output to a display device. Initially developed to handle graphics rendering for video games and other multimedia applications, GPUs have evolved into powerful parallel processors capable of handling a wide range of tasks beyond graphics processing, including scientific simulations, machine learning, and cryptocurrency mining.

The difference between GPU and CPU

While both GPUs and CPUs are crucial components of modern computing systems, they differ significantly in their design, functionality, and usage. CPUs, or Central Processing Units, are optimized for sequential processing tasks, featuring a few powerful cores capable of executing instructions one after another. They excel at tasks that require complex decision-making and high clock speeds. In contrast, GPUs are designed for parallel processing, boasting thousands of smaller, more efficient cores optimized for handling multiple tasks simultaneously. They are highly adept at tasks that involve massive parallelism, such as rendering graphics, processing large datasets for machine learning, and accelerating certain computational tasks.

Traditional CPUs are structured with only a few cores. For example, the Xeon X5670 CPU has six cores. However, a modern GPU chip can be built with hundreds of processing cores.

Distributed and Cloud Computing by Kai Hawang

The world’s first GPU, the GeForce 256, was marketed by NVIDIA in 1999. These GPU chips can process a minimum of 10 million polygons per second

Distributed and Cloud Computing by Kai Hawang

Detailed architecture of GPU

The architecture of a GPU typically consists of several key components, including

Processing Cores: GPUs contain a large number of processing cores (often referred to as shader cores or CUDA cores) that work in parallel to perform computations. These cores are organized into streaming multiprocessors (SMs) or compute units, each capable of executing multiple threads simultaneously.

Memory Hierarchy: GPUs feature multiple levels of memory hierarchy, including on-chip caches, high-speed VRAM (Video Random Access Memory), and sometimes system memory (RAM). This hierarchy is crucial for efficiently accessing and storing data during computations.

Compute Units: Compute units within a GPU are responsible for executing instructions and performing calculations. They consist of arithmetic logic units (ALUs), registers, and control units.

Instruction Pipeline: GPUs utilize a pipelined architecture to efficiently process instructions. This involves breaking down tasks into smaller stages and executing them in parallel across multiple cores.

Memory Controllers: Memory controllers manage the flow of data between the GPU’s processing cores and memory subsystems, ensuring efficient data access and transfer.

How GPU works

At the core of GPU functionality lies parallel processing, which enables it to handle multiple tasks simultaneously. When a task is sent to a GPU, it is divided into smaller, independent instructions called threads. These threads are then executed concurrently across the numerous processing cores within the GPU.

Task Parallelism: Unlike CPUs, which focus on executing instructions sequentially, GPUs excel at task parallelism. They can execute thousands of threads simultaneously, making them ideal for computationally intensive tasks that can be divided into smaller units of work.

Streaming Multiprocessors (SMs): The processing cores in a GPU are organized into streaming multiprocessors (SMs). Each SM contains multiple processing cores, along with specialized units for tasks such as texture mapping and memory access. SMs manage the execution of threads, scheduling them for execution and coordinating their access to shared resources.

Data Parallelism: In addition to task parallelism, GPUs leverage data parallelism to further enhance performance. Data parallelism involves applying the same operation to multiple pieces of data simultaneously. This is achieved through SIMD (Single Instruction, Multiple Data) execution, where a single instruction is applied to multiple data elements in parallel.

Memory Hierarchy: GPUs feature a hierarchy of memory subsystems optimized for different types of data access. This includes on-chip caches for fast access to frequently used data, high-speed VRAM for storing large datasets, and sometimes access to system memory (RAM) for additional capacity. Efficient management of data movement between these memory tiers is crucial for maximizing performance.

Compute APIs: To harness the power of GPUs, developers use compute APIs (Application Programming Interfaces) such as CUDA (Compute Unified Device Architecture) for NVIDIA GPUs and OpenCL (Open Computing Language) for various GPU architectures. These APIs provide a programming model for writing parallel code and managing GPU resources.

Overall, the parallel architecture and specialized design of GPUs enable them to deliver high performance across a wide range of computational tasks, from scientific simulations and machine learning to real-time graphics rendering in video games and multimedia applications.