this post was submitted on 25 Jul 2024
658 points (100.0% liked)
196
16925 readers
1374 users here now
Be sure to follow the rule before you head out.
Rule: You must post before you leave.
If you have any questions, feel free to contact us on our matrix channel.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Yes, but 200 gb is probably already with 4 bit quantization, the weights in fp16 would be more like 800 gb IDK if its even possible to quantize more, if it is, you're probably better of going with a smaller model anyways