this post was submitted on 25 Jul 2024
658 points (100.0% liked)
196
16925 readers
1477 users here now
Be sure to follow the rule before you head out.
Rule: You must post before you leave.
If you have any questions, feel free to contact us on our matrix channel.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I don't have access to llama 3.1 405b but I can see that llama 3 70b takes up ~145 gb, so 405b would probably take 840 gigabytes, just to download the uncompressed fp16 (16 bits / weight) model. With 8 bit quantization it would probably take closer to 420 gb, and with 4 bit it would probably take closer to 210 gb. 4 bit quantization is really going to start harming the model outputs, and its still probably not going to fit in your RAM, let alone VRAM.
So yes, it is a crazy model. You'd probably need at least 3 or 4 a100s to have a good experience with it.