What the Memory Bottleneck Involves
Neural network computation requires continuously supplying weights and activation data to the compute units. If memory bandwidth can't keep pace with compute throughput, the compute units sit idle waiting for data – a condition engineers call being 'memory-bound.' This is especially pronounced for workloads with low arithmetic intensity (relatively little computation performed per byte of data moved), which describes many common neural network layers.
Why It Matters
An accelerator that looks impressive on a compute-throughput spec sheet may deliver disappointing real-world performance if its memory subsystem can't keep the compute units fed. Understanding a workload's arithmetic intensity – and designing the memory hierarchy accordingly – is often more important to actual performance than simply adding more compute units.