this post was submitted on 09 Jan 2025
480 points (99.0% liked)
Opensource
1681 readers
3 users here now
A community for discussion about open source software! Ask questions, share knowledge, share news, or post interesting stuff related to it!
⠀
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Source?
I don't have a source for that, but the most that any locally-run program can cost in terms of power is basically the sum of a few things: maxed-out gpu usage, maxed-out cpu usage, maxed-out disk access. GPU is by far the most power-consuming of these things, and modern video games make essentially the most possible use of the GPU that they can get away with.
Running an LLM locally can at most max out usage of the GPU, putting it in the same ballpark as a video game. Typical usage of an LLM is to run it for a few seconds and then submit another query, so it's not running 100% of the time during typical usage, unlike a video game (where it remains open and active the whole time, GPU usage dips only when you're in a menu for instance.)
Data centers drain lots of power by running a very large number of machines at the same time.
Training the model yourself would take years on a single machine. If you factor that into your cost per query, it blows up.
The data centers are (currently) mainly used for training new models.
But if you divide the cost of training by the number of people using the model, it should be pretty low.