this post was submitted on 27 Jan 2025
125 points (99.2% liked)

Lemmy.ca's Main Community

2903 readers
6 users here now

Welcome to lemmy.ca's c/main!

Since everyone on lemmy.ca gets subscribed here, this is the place to chat about the goings on at lemmy.ca, support-type items, suggestions, etc.

Announcements can be found at https://lemmy.ca/c/meta

For support related to this instance, use https://lemmy.ca/c/lemmy_ca_support

founded 4 years ago
MODERATORS
smorks
TruckBC
Shadow
125
Server Maintenance - Jan 29th and Jan 30th at 9AM PT (self.main)
submitted 1 week ago* (last edited 1 week ago) by Shadow to c/main
35 comments fedilink hide all child comments
 

Hello everyone!

I'll be taking the site down for two maintenance windows this week to complete our server migration.

  • Weds Jan 29th - 09:00 - 11:00 PT (12:00 - 14:00 ET)
  • Thurs Jan 30th - 09:00 - 11:00 PT (12:00 - 14:00 ET)

During the first window I'll be migrating us from OVH to our new dedicated hardware. After this migration there will likely be some temporarily broken images, as it takes approximately 8 hours to resync our object storage from OVH.

This is a major change and despite my testing, may have some unintended side effects. If you run into any problems that aren't just a broken image, please let us know.

The second maintenance window is to migrate our pict-rs database from it's local sled-db into our primary postgres DB. This is a much smaller change but since pict-rs checks every image as it goes through them, it takes about 1.5 hours.

As usual, you can check https://status.lemmy.ca/ for updates.

you are viewing a single comment's thread
view the rest of the comments
[–] Shadow 10 points 1 week ago (3 children)

Second move complete, all done for now!

[–] Darkassassin07 1 points 6 days ago* (last edited 6 days ago) (1 children)

I've noticed my profile pic is partially broken.

The mini-thumbnail beside my name in comments appears fine; but the image in 'about' on my profile is broken. (viewed from Jerboa).

It opens properly in full-res when you click on the broken image though.

Haven't tried re-uploading it yet. Just figured I'd report it.

/edit Broken in the side bar too

Fixed with a re-upload.

[–] Shadow 1 points 6 days ago (2 children)

Do you happen to have the broken image url still?

[–] Darkassassin07 1 points 6 days ago (2 children)

It looks like your profile is doing the same; but only when viewed from Jerboa.

Jerboa:

Web browser:

I didn't check mine in a browser at the time.

[–] Shadow 3 points 6 days ago

Thanks, I see it too. Digging.

[–] Shadow 2 points 6 days ago (1 children)

I think this is fixed now. I'm not sure why but pictrs was expecting a bunch of resized images that don't exist in either the old or new object stores. I've manually cleaned up the DB.

[–] Darkassassin07 2 points 6 days ago* (last edited 6 days ago) (2 children)

Many of them are, but scrolling through this thread and checking the profiles of various lemmy.ca accounts; some are still broken.

@[email protected] for example

[–] Shadow 1 points 6 days ago (1 children)

I've hacked in a dirty dirty fix for now. It looks like the pictrs database migration failed horribly and cloudflare was caching the images, so nobody noticed. Now I've got two copies of pictrs running (old + new) and some code in place to try both of them.

Sigh.

[–] Darkassassin07 1 points 6 days ago

Oh shit.... Well at least you found it before everything just stopped working? :(

Thanks for your efforts m8; I appreciate the hard work. Best of luck fixing that.

[–] Shadow 1 points 6 days ago

Well shit, cloudflare's been caching all our images and deceiving me into thinking stuff was working properly. Digging, but I made a mess =(

[–] Darkassassin07 1 points 6 days ago* (last edited 6 days ago)

I don't, but I'll pm you an IP address. Check the logs for 404s from that ip? (around 4 hours ago)

[–] ininewcrow 1 points 1 week ago

Beautiful work ... everything purring like a well tuned motor over here

Thanks!

[–] GlassHalfHopeful 3 points 1 week ago

Great news. I hope it was relatively non-eventful.