Wiki / concepts / wiki

RAM-to-parameters rule

tina-huang's rough way to size a local model: subtract 25% of the advertised RAM for the operating system and other programs, then divide what's left by 0.6. The result is the largest model, in billions of parameters, that fits.

  • 32 GB MacBook Pro: 24 GB usable, 24 / 0.6 = 40B.
  • They leave the 64 GB Mac Studio as a viewer quiz. The rule gives 48 GB usable and 80B, though the language models they run on it are 27B to 32B.
  • The 128 GB AMD Ryzen AI Halo has about 96 GB usable by their count, and they run 70B and 120B models on it.

They call it a very rough method and say asking a chatbot also works. The rule covers capacity only; it says nothing about speed (capacity-vs-bandwidth) or the memory a long context needs. For that, Kai's Mac mini video puts the KV cache of a 70B model at about 10 GB for 32,000 tokens and 40 GB for 128,000 (report).

Related: model-parameters.

Source: report

Linked from

Capacity vs bandwidthTina Huang