I want to say that I've been blown away by the amount of support that's I've received in the last 24 hours. You all have left me speechless. Thank you everyone who have donated and those who couldn't, know that being a good member of this community already more than I could ask for. Thank you!
Hey there, I considered this at first but my understanding with tesseract is that this would mean we'd be proxying account credentials/tokens of other instances. I'm comfortable in doing this for our own instance but it doesn't feel I'd be a responsible human to enable it for other instances. If there was a way to be more transparent to users using it for external instances I'd reconsider. I'm open to any feedback.
edit: typo
This is odd. I did not upgrade pictrs (the service responsible for image hosting). There may have been an error for some specific tables rows during the database migrates. Another possibility could have been an error during the schema update that I believe the new version had to go through.
Would you be able to DM me links to other posts that you come across with broken images? I will look to see if I can identify any common trends that could tell us more.
Thank you
Hello sh.itjust.works community,
It’s been a while since my last update, and I owe you all a more detailed one soon. However, today I’m here to give you a heads-up about the upcoming upgrade.
Over the next 7 days, I’ll be upgrading the instance to v19.5. While this update doesn’t strictly require it, I’ve decided to take this opportunity to upgrade our Postgres database from version 15 to 16 as well. This process can be time-consuming and have take up to 2 hours to complete on other instance.
I'm optimistic that we'll complete it in less time, but to err on the side of caution, I've scheduled the maintenance window for 120 minutes.
To minimize downtime, I've split the upgrade into two phases: Phase one will cover the database update, and Phase two will handle the Lemmy code upgrade.
Note: If the database upgrade is completed in under 60 minutes, I’ll proceed with the Lemmy code upgrade during the same maintenance window.
Here's the tentative timeline:
| Task | Date | Expected Downtime |
|---|---|---|
| Upgrade of Postgres Database | Friday September 20 2024 @ 8:00PM ET | 60-120 Minutes |
| Upgrade to V19.5 | ~~Monday or Tuesday September 23 or 24 @ 8:00PM ET~~ | ~~Up to 30 Minutes~~ |
During these two planned events those who want to provide moral support or who want to get periodic updates are more than welcome to join us on our matrix channel
edit: 9:23EDT - Hey all, the database and lemmy codebase has been upgraded to v19.5. The database upgrade took a lot less time than expected so I went right into upgrading the codebase. A whole bunch of background tasks are still running which might make your experience slower than usual. Things should be back to normal in the next couple of hours.
It's coming. This instance is deployed using lemmy-ansible but with some slight modifications. I need to review a few things to make sure the transition is smooth. As you also pointed out, it's also summer time which isn't helping with my free time situation.
I have a long term solution that will speed up upgrades but I only expect to get it in place by the fall. In the meanwhile, I'm planning on getting this instance upgraded to 19.5 in the coming weeks.
Hey,
I'm due to make a post about the instances finances soon. I'll get one posted in the next bit.
I'm still covering costs out of pocket but my goal has been to have this instance fully funded by its members. I want this instance to stay true to its members, the fediverse and be put in a position where it can continue to thrive transparently without me. The admin team has been doing an outstanding job keeping this instance safe and moderated for everyone. I don't thank them enough but they are the true heroes of this instance.
As some may know, I've been working with a local non profit for the past 6 months to have them leveraged in accepting accepting donations for this instance. They have been slow to get things done but its progressing. I may end up doing what I was trying to avoid and create my own non-profit instead if that's what it takes. That direction does have its own challenges and would add additional responsibilities to my plate.
In the meanwhile, while some of the costs of the services have gone up slightly, I get joy knowing that I can continue providing this instance to all of you. I'm OK paying the costs to keeping the lights on until we can transition to donations.
There was a little hiccup this morning unrelated to the migration tonight. The Lemmy services like to be restarted every once in a while. Once we move to the new hardware I'll be able to look into implementing a better logging system and hopefully be more proactive when situations like these happen.
The biggest consumer of storage on this instance is related to the image hosting which we use an external object storage provider for. The second is the database which is no were near the 2TB capacity. 1TB SSDs are cheaper than 2TB SSDs and I also didn't want to spend more than I needed. As other mentioned if we need more space or IOPs in the future, I could accomplish this by adding more drives as a quick fix. This server does not support NVME unless I leverage its PCIe ports but I don't plan on doing that. By the time this instance gets to the point where 10 SSD drives just isn't cutting it anymore I'll probably have come across another opportunities on getting a new server with better NVME support.
Hello sh.itjust.works community,
Many of you have been eager to get an update about when the sh.itjust.works instance will get it's upgrade to the latest version of lemmy. Here's a update along with a tentative timeline.
In December 2023 I purchased a new server for this community. It took me awhile but I eventually made the time to get it racked at the local datacenter. For the sysadmins lingering and those interested here are the specs:
- Dual Xeon 2.9Ghz CPUs (32 cores total)
- 256GB ram
- 4 x 1TB SSD in raid 10 (with room to add 6 more disks)
- 10gbit networking
While I'm ready to proceed with the upgrade, I've decided to first migrate this instance over to the new hardware. Here are two reasons.
- Those of you who have been around long enough may remember that I've been running this instance on "borrowed" unused resources that were available at the time. There are no more resources available for this instance to grow.
- There are reports that the latest version of lemmy may use more resources. Given we are among the bigger instances, should I end up in a situation where I need to increase resources to keep things fast I'll be restricted.
Here's the tentative timeline:
| Task | Date | Expected Downtime |
|---|---|---|
| Migration to new server | Tuesday February 27 2024 @ 8:00PM ET | 90 Minutes |
| Upgrade to V19.3 | Thursday February 29 2024 @ 8:00PM ET | Up to 120 Minutes |
- If anything major goes wrong on the 27th I will revert back the changes and bring the instance back up on the current server.
- If anything major goes wrong on the 29th I will revert back using an earlier snapshot. If that fails, I will restore from a backup.
During these two planned events those who want to provide moral support or who want to get periodic updates are more than welcome to join us on our matrix channel
=========================================================
Update February 29 2024
We've successfully completed the upgrade to v1.9.3. I'm happy to announce that we did it in an astonishing 27 minutes, a whole 93 minutes under what was expected. The extra leg work that was done over the last few weeks combined with the better hardware definitely played a part. Looking over the processes, it looks like the service responsible for images is still doing some work so it's possible that you will come across some broken images. I'll be keeping on eye on that over the next bit and make adjustments if needed. Thank you all for the support and to all of you who kept me company on our matrix channel. Have a good evening.
=========================================================
Update February 27 2024
We've successfully completed the migration. I'm happy to announce that this instance is now running on its new hardware dedicated solely to this community! We experienced just under 40 minutes of downtime which is a whole 50 minutes less than expected. Please give this instance a chance to catch up what it missed but we should be good within the next 30 or so minutes. Thank you
view more: next ›
This one was received. Thanks!