Lemmy.ca's Main Community

3283 readers

9 users here now

Welcome to the lemmy.ca/c/main community!

All new users on lemmy.ca are automatically subscribed to this community, so this is the place to read announcements, make suggestions, and chat about the goings-on of lemmy.ca.

For support requests specific to lemmy.ca, you can use [email protected].

founded 4 years ago

MODERATORS

148

About the outages (July 1st and 2nd 2024) (self.main)

submitted 9 months ago* (last edited 9 months ago) by mp3 to c/main

15 comments fedilink hide all child comments

Happy Canada Day everyone.

Related to the outage that happened last night, we rebooted the Lemmy services but we're still trying to figure out the root cause, which seems to point to an out of memory issue in the logs. However it's not what we see in our monitoring console.

In the meantime, we will monitor the service more closely until we are confident the issue is resolved, and we will improve our tools to detect such a problem faster.

EDIT: Also happened at night on July 2nd, still trying to find the root cause..

Apologies for the extended downtime.

you are viewing a single comment's thread
view the rest of the comments

[–] Shadow 13 points 9 months ago* (last edited 9 months ago)

Something got into a weird state and restarting either the backend or frontend didn't help. Taking the entire stack down and then bringing it back up, resolved it.

It's weird since it crashed at 1am and at 3am we gradually restart all backend and frontends, so that automatic restart should have fixed it too. All the containers reported healthy, but nginx wasn't reporting any available frontends.

I suspect some sort of weird lemmy bug, but we'll just have to improve monitoring for now and try to debug this more if it happens again.