Shadow

Unfortunately the pictrs database migration yesterday seems to have skipped migrating a ton of data.

Since cloudflare is caching our images, almost nobody noticed. We've now been running on the new instance for over a day, collecting new images into a new incomplete db. Fun.

I now have two copies of pictrs running, one with the postgres db and one with the legacy sled-db. Requests that fail against our updated one, will be retried against the legacy one. This should result in all images working properly again for now, while I figure out a better long term solution.

Please let me know if you still see any issues with images loading!

More details - https://www.justice.gov/opa/pr/former-senior-adviser-federal-reserve-indicted-charges-economic-espionage

I'm curious if anyone here actually finds value in the reddit posts brought over by lemmit.online, since I'd like to defederate from it otherwise.

It feels actively harmful to lemmy, since so many of the posts it brings over are questions that the original poster will never see. It encourages a conversation that will never happen, so if someone does reply they're going to feel disengaged.

The bot rarely gets any upvotes or engagement, and I suspect a majority of people (like myself) have just blocked it. TBH I forgot it existed until Tesseract showed me its posts again.

Hi everyone!

Tesseract is now available as an alternative front end at https://tess.lemmy.ca/

Hello everyone!

I'll be taking the site down for two maintenance windows this week to complete our server migration.

• Weds Jan 29th - 09:00 - 11:00 PT (12:00 - 14:00 ET)
• Thurs Jan 30th - 09:00 - 11:00 PT (12:00 - 14:00 ET)

During the first window I'll be migrating us from OVH to our new dedicated hardware. After this migration there will likely be some temporarily broken images, as it takes approximately 8 hours to resync our object storage from OVH.

This is a major change and despite my testing, may have some unintended side effects. If you run into any problems that aren't just a broken image, please let us know.

The second maintenance window is to migrate our pict-rs database from it's local sled-db into our primary postgres DB. This is a much smaller change but since pict-rs checks every image as it goes through them, it takes about 1.5 hours.

As usual, you can check https://status.lemmy.ca/ for updates.

Hello everyone, we're long overdue for an update on how things have been going!

Finances

Since we started accepting donations back in July we've received a total of $1350, as well as $1707 in older donations from smorks. We haven't had any expenses other than OVH (approx $155/mo) since then, leaving us $2152 in the bank.

We still owe TruckBC $1980 for the period he was covering hosting, and I've contributed $525 as well (mostly non-profit registration related stuff, plus domain renewals). We haven't yet discussed reimbursing either of us, we're both happy to build up a contingency fund for a while.

New Server

A few weeks ago, we experienced a ~26-hour outage due to a failed power supply and extremely slow response times from OVH support. This was followed by an unexplained outage the next morning at the same time. To ensure Lemmy’s growth remains sustainable for the long term and to support other federated applications, I’ve donated a new physical server. This will give us a significant boost in resources while keeping the monthly cost increase minimal.

Our system specs today:

• Undoubtedly the cheapest hardware OVH could buy
• Intel Xeon E-2386G (6 cores @ 3.5ghz)
• 32gb of ram
• 2x 512gb Samsung nvme in raid 1
• 1gb network
• $155/month

The new system:

• Dell R7525
• AMD EPYC 7763 (64 cores @ 2.45ghz)
• 1tb of ram
• 3x 120gb sata ssd (hw raid 1 with a hot spare, for proxmox)
• 4x 6.4tb nvme (zfs mirrored + striped, for data)
• 1gb network with a 50mbit commit (See 95th percentile billing)
• Redundant power supplies
• Next day hardware support until Aug 2027
• $166/month + tax

This means instead of renting an entire server and having them be responsible for the hardware, we'll be renting co-location space at a Vancouver datacenter PDF via a 3rd party service provider I know.

These servers are extremely reliable but if there is a failure, either Otter or myself will be able to get access reasonably quickly. We also have full OOB access via idrac, so it's pretty unlikely we'll ever need to go on site.

Server Migration

Phase 1 is currently planned for Jan 29th or 30th and will completely move us out of OVH and onto our own hardware. I'm expecting probably a 2-3 hour outage, followed by an 6-8 hour window where some images may be missing as the object store resyncs. I'll make another follow up post in a week with specifics.

Phases 2+ I'm not 100% decided on yet and have not planned a timeline around. It would get us into a fully redundant (excluding hardware) setup that's easier to scale and manage down the road, but it does add a little bit of complexity.

• Dark mode diagram
• Light mode diagram

Let me know if you have any questions or comments, or feedback on the architecture!

Morning all!

I'm going to be taking the site down for about 5 minutes, so that I can get a consistent copy of our databases (postgres + pict-rs sled).

Will do it at about 10am PST.

Hey everyone, and happy new year!

Sorry about that super long downtime there. Yesterday (Sunday) morning at 10:03AM PST our server suffered a physical hardware failure, apparently a power supply failure. Unfortunately despite opening a ticket with our hosting vendor (OVH) a few minutes later and them claiming to have 24/7 support, nobody looked at our ticket until this morning when their phone support lines opened and I called them.

They've now replaced a defective power supply and we're back online, after ~26 hours of being offline. Some pretty disappointing response times, to put it nicely.

We're planning to move away from OVH at the end of this month, onto proper enterprise grade hardware that we own and control. This will give us a HUGE boost in server resources and allow us to scale for the foreseeable future, while also giving us the control to resolve problems like this much quicker. Expect another follow up post about this in the next couple weeks once I've put together the migration plan.

Timeline:

• Jan 5th 10:03am PST - We get alerts to the server being non-responsive.
• Jan 5th 10:05am PST - I pull up the console via IPMI and it's completely non-responsive. Attempting to power off / on the server or do anything, does not work.
• Jan 5th 10:15am PST - Initial support ticket created with OVH. I followed up a couple times over the next few hours, and got no response.
• Jan 6th 6:32am PST - Called OVH, gave them the case number and asked them to investigate
• Jan 6th 7:34am PST - I get notified they'll start their "intervention" in 15 minutes.
• Jan 6th 11:04am PST - Call them again, the tech is still working on it and they'll get back to me with an update
• Jan 6th 11:34am PST - "I was informed by our data centre technician that there is an issue with the power supply unit for the rack on which your server resides. Your server will come back online once they have replaced the power supply."
• Jan 6th 12:17pm PST - We're back up finally!

Edit on Jan 7th @ 8:40am PST: We just had another outage of about an hour. Investigating with OVH.