Memory bandwidth

Also known as: GB/s, TB/s

How fast the GPU reads its memory, in GB/s or TB/s. Generating LLM tokens is usually limited by bandwidth rather than FLOPS, which is why PetaPrice can rank GPUs by bandwidth per dollar.

Best value right now

Ranked by $ per TB/s of memory bandwidth per hour at each GPU's lowest on-demand price.

1RTX 309024 GB$0.12/h$0.12/TB/s·h
2RTX 306012 GB$0.06/h$0.17/TB/s·h
3RTX 3080 Ti12 GB$0.18/h$0.20/TB/s·h
4RTX A500024 GB$0.16/h$0.21/TB/s·h
5V100 SXM2 16GB16 GB$0.19/h$0.21/TB/s·h
6V100 PCIe16 GB$0.19/h$0.21/TB/s·h
7RTX 30708 GB$0.09/h$0.21/TB/s·h
8RTX 4070 SUPER12 GB$0.11/h$0.21/TB/s·h

Compare every GPU →

Related terms

HBMHigh Bandwidth Memory, stacked next to the chip on datacenter GPUs (HBM2e, HBM3, HBM3e). Several times faster than the GDDR memory on consumer and workstation cards.
VRAMThe GPU's own memory. The model weights, activations and KV cache must fit in it (or be split across GPUs), so VRAM often decides which GPU you can use at all.
KV cacheMemory an LLM keeps for every token in its context during inference. Long contexts and many parallel requests need a lot of it, on top of the model weights.

← All terms

…