Lemmy Support

4753 readers

4 users here now

Support / questions about Lemmy.

Matrix Space: #lemmy-space

founded 5 years ago

MODERATORS

[email protected]

Lemmy Pict-rs storage requirements? (lemmy.ml)

submitted 2 years ago* (last edited 2 years ago) by [email protected] to c/[email protected]

26 comments fedilink hide all child comments

I might take over one of these one-year-free hosted Lemmy instances on my server infrastructure, but I read several times now that Lemmy's image hosting system Pict-rs is using a lot of storage quickly.

The server I could run this on is limited to 32gb ssd storage with no easy way to expand it.

Is there some way to limit the image storage use and automatically prune old images that are not user or community icons or such?

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 7 points 2 years ago (4 children)

pict-rs doesn't keep track of how often it serves different images, so there's not a good metric for pruning old images. That said, 0.4 will introduce functionality for cleaning up processed images (e.g. resizes/thumbnails), removing their files & metadata. If they are viewed again, they will be re-generated.

0.4 will also include the ability to scale down images on upload, rather than storing the original resolution. This is not yet implemented, but it's on my roadmap.

All this said, it is already possible to use pict-rs with object storage (s3-compatible), rather than block storage. That's a good option if your hosting provider offers it

[–] [email protected] 4 points 2 years ago (1 children)

Actually S3 compatible interface might be interesting to link Pict-rs to Garage...

[–] [email protected] 2 points 2 years ago

I am aware of garage, but haven't tested it yet with pict-rs. It's a cool project for sure

[–] [email protected] 3 points 2 years ago* (last edited 2 years ago) (1 children)

That sounds promising. Any idea when 0.4 will be released?

Object-storage on large cloud providers is not an option for me for various reasons (privacy, legal etc.).

[–] [email protected] 4 points 2 years ago (1 children)

I can only say "when it's ready." I think most of what I want to include in 0.4 is there, but I don't have a ton of time to work on it currently. I might see if I can get my last feature changes in this weekend, then it will be a matter of ensuring the 0.3 -> 0.4 upgrade is smooth, and that storage migration is solid

[–] [email protected] 1 points 2 years ago

Update on this: I got the feature work done this weekend, so now I'll be testing it a bunch for upgrades and storage migrations

[–] [email protected] 1 points 2 years ago

Hello there! I am one of the administrators at Beehaw. If I'm reading and understanding your comment correctly, then this could solve our most pressing problem of running out of server disk space.

Is there a time-frame when you expect to have pict-rs 0.4 available?

[–] [email protected] 1 points 2 years ago (1 children)

Is deduplication supported by re-using images already in storage if newly upload images share the same hash with them?

[–] [email protected] 1 points 2 years ago (1 children)

Yes. It uses sha256 rather than perceptual hashing, but that's Good Enough™️

[–] [email protected] 1 points 2 years ago (1 children)

Why not SHA-512 or SHA3?

[–] [email protected] 1 points 2 years ago (1 children)

I chose it at the start of the project 🤷

[–] [email protected] 1 points 2 years ago

Maybe is it worthy to make a smooth change to this in the future? https://en.wikipedia.org/wiki/SHA-3#Comparison_of_SHA_functions