I found a VRAM calculator for LLMs here: https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator
Wow it seems like for 128K context size you do need a lot of VRAM (~55 GB). Qwen 72B will take up ~39 GB so you would either need 4x 24GB Nvidia cards or the Mac Pro 192 GB RAM. Probably the cheapest option would be to deploy GPU instances on a service like Runpod. I think you would have to do a lot of processing before you get to the breakeven point of your own machine.
The context cache doesn't take up too much memory compared to the model. The main benefit of having a lot of VRAM is that you can run larger models. I think you're better off buying a 24 GB Nvidia card from a cost and performance standpoint.
I would suggest an Intel N100 mini PC if you are planning to transcode video files with Plex. Intel Quick Sync performs better than AMD for media transcoding.
I wasn't sure if it was AI or not. According to the description on GitHub:
Utilizes state-of-the-art algorithms to identify duplicates with precision based on hashing values and FAISS Vector Database using ResNet152.
Isn't ResNet152 a neural network model? I was careful to say neural network instead of AI or machine learning.
Yeah, the duplicate finder uses a neural network to find duplicates I think. I went through my wedding album that had a lot of burst shots and it was able to detect similar images well.
Not sure if you're aware, but Immich has a duplicate finder
Dockge for docker compose stacks. Glances for system resource usage because it has a Homepage widget.
All good recs so far but haven't seen anyone mention Infernal Affairs.
I am running my disks as mirrors.
I have a Terramaster 4-bay DAS
I'm running a mini-PC with the N100, 12GB RAM, and 2x18TB mirrored drives on ZFS and it seems to work well.
One thing I would do differently is setup LDAP and OIDC so you can use the same authentication credentials for different apps (at least the ones that support them). I use LLDAP and Authelia for this purpose.