Why AI chips are limited by memory, not just speed
The most quoted number on an AI chip is compute. For running large models, how fast it can fetch data often matters just as much.
By Team Inspiropedia

Key points
- Large models must be read from memory for every token they generate.
- High-bandwidth memory (HBM) stacks memory beside the processor to move data faster.
- Memory supply and advanced packaging can limit how many AI accelerators get built.
Ask what makes an AI accelerator good and you will usually hear about compute: how many operations per second it can perform. For training giant models that matters a great deal. For serving them to users, a second number often matters as much, how quickly the chip can fetch the model from memory.
The memory wall
Generating text means reading a large share of the model’s weights from memory for every token. If the processor can calculate faster than memory can deliver data, it sits idle, waiting. Engineers call this being memory-bound, and it is common when answering a single user’s request at a time.
What HBM is
High-bandwidth memory, or HBM, stacks several memory dies vertically and places them next to the processor on the same package. The short, wide connections deliver far more bandwidth than conventional memory modules on a circuit board. Today’s leading AI accelerators use it, and the main makers are SK hynix, Samsung and Micron.
Why it affects supply and price
- HBM is harder to manufacture than standard memory, so supply is limited.
- Joining processor and memory stacks needs advanced packaging, which has been a bottleneck at times.
- Memory per chip sets how large a model fits without being split across several chips, which adds communication cost.
What to look for
When a company announces a new accelerator, look beyond peak compute. Ask how much memory it carries, how fast that memory is, and how chips connect to one another. For many real workloads those figures decide the cost of each answer.


