large language models are large and must be loaded into memory to train or to use for inference if we want to keep them fast. older models like gpt3 have around 175 billion parameters. at float32s that comes out to something like 700GB of memory. newer models are even larger. and openai wants to run them as consumer web services.

▲

lysace an hour ago | parent [-]

I mean, I know that much. The numbers still don't make sense to me. How is my internal model this wrong?

For one, if this was about inference, wouldn't the bottleneck be the GPU computation part?

	▲	ssl-3 24 minutes ago \| parent [-]
		Concurrency? Suppose some some parallelized, distributed task requires 700GB of memory (I don't know if it does or does not) per node to accomplish, and that speed is a concern. A singular pile of memory that is 700GB is insufficient not because it lacks capacity, but instead because it lacks scalability. That pile is only enough for 1 node. If more nodes were added to increase speed but they all used that same single 700GB pile, then RAM bandwidth (and latency) gets in the way.