Lemmy just reached a new milestone: 1 million posts, across 1,323 servers.

Source: https://lemmy.fediverse.observer/dailystats&days=90

I'm tired about reading about reddit here.

We left. Let's move on.

Please note this is just a beta and there are going to be bugs, but it works and it works nicely. Have fun.

Source: https://lemmy.fediverse.observer/dailystats&days=90

AccidentalRenaissance has no active moderators due to Reddit's unprecedented API changes, and has thus been privated to prevent vandalism.

Resignation letters:

Openminded_Skeptic - https://imgur.com/a/WwzQcac

VoltasPistol - https://imgur.com/a/lnHSM4n

We welcome you to join us in our new homes:

https://kbin.social/m/AccidentalRenaissance

https://lemmy.blahaj.zone/c/accidentalrenaissance

Thank you for all your support!

Original post from r/ModCoord

I hate that everything now is a subscription service instead of buying it and do whatever you want.

Looks like it works.

Edit still see some performance issues. Needs more troubleshooting

Update: Registrations re-opened We encountered a bug where people could not log in, see https://github.com/LemmyNet/lemmy/issues/3422#issuecomment-1616112264 . As a workaround we opened registrations.

Thanks

First of all, I would like to thank the Lemmy.world team and the 2 admins of other servers @[email protected] and @[email protected] for their help! We did some thorough troubleshooting to get this working!

The upgrade

The upgrade itself isn't too hard. Create a backup, and then change the image names in the docker-compose.yml and restart.

But, like the first 2 tries, after a few minutes the site started getting slow until it stopped responding. Then the troubleshooting started.

The solutions

What I had noticed previously, is that the lemmy container could reach around 1500% CPU usage, above that the site got slow. Which is weird, because the server has 64 threads, so 6400% should be the max. So we tried what @[email protected] had suggested before: we created extra lemmy containers to spread the load. (And extra lemmy-ui containers). And used nginx to load balance between them.

Et voilà. That seems to work.

Also, as suggested by him, we start the lemmy containers with the scheduler disabled, and have 1 extra lemmy running with the scheduler enabled, unused for other stuff.

There will be room for improvement, and probably new bugs, but we're very happy lemmy.world is now at 0.18.1-rc. This fixes a lot of bugs.

Hello there!

It has been a while since our last update, but it's about time to address the elephant in the room: downtimes. Lemmy.World has been having multiple downtimes a day for quite a while now. And we want to take the time to address some of the concerns and misconceptions that have been spread in chatrooms, memes and various comments in Lemmy communities.

So let's go over some of these misconceptions together.

"Lemmy.World is too big and that is bad for the fediverse".

While one thing is true, we are the biggest Lemmy instance, we are far from the biggest in the Fediverse. If you want actual numbers you can have a look here: https://fedidb.org/network

The entire Lemmy fediverse is still in its infancy and even though we don't like to compare ourselves to Reddit it gives you something comparable. The entire amount of Lemmy users on all instances combined is currently 444,876 which is still nothing compared to a medium sized subreddit. There are some points that can be made that it is better to spread the load of users and communities across other instances, but let us make it clear that this is not a technical problem.

And even in a decentralised system, there will always be bigger and smaller blocks within; such would be the nature of any platform looking to be shaped by its members. 

"Lemmy.World should close down registrations"

Lemmy.World is being linked in a number of Reddit subreddits and in Lemmy apps. Imagine if new users land here and they have no way to sign up. We have to assume that most new users have no information on how the Fediverse works and making them read a full page of what's what would scare a lot of those people off. They probably wouldn't even take the time to read why registrations would be closed, move on and not join the Fediverse at all. What we want to do, however, is inform the users before they sign up, without closing registrations. The option is already built into Lemmy but only available on Lemmy.ml - so a ticket was created with the development team to make these available to other instance Admins. Here is the post on Lemmy Github.

Which brings us to the third point:

"Lemmy.World can not handle the load, that's why the server is down all the time"

This is simply not true. There are no financial issues to upgrade the hardware, should that be required; but that is not the solution to this problem.

The problem is that for a couple of hours every day we are under a DDOS attack. It's a never-ending game of whack-a-mole where we close one attack vector and they'll start using another one. Without going too much into detail and expose too much, there are some very 'expensive' sql queries in Lemmy - actions or features that take up seconds instead of milliseconds to execute. And by by executing them by the thousand a minute you can overload the database server.

So who is attacking us? One thing that is clear is that those responsible of these attacks know the ins and outs of Lemmy. They know which database requests are the most taxing and they are always quick to find another as soon as we close one off. That's one of the only things we know for sure about our attackers. Being the biggest instance and having defederated with a couple of instances has made us a target.  

"Why do they need another sysop who works for free"

Everyone involved with LW works as a volunteer. The money that is donated goes to operational costs only - so hardware and infrastructure. And while we understand that working as a volunteer is not for everyone, nobody is forcing anyone to do anything. As a volunteer you decide how much of your free time you are willing to spend on this project, a service that is also being provided for free.

We will leave this thread pinned locally for a while and we will try to reply to genuine questions or concerns as soon as we can.

Lemmy.ml has now blocked Threads.net

I strongly encourage instance admins to defederate from Facebook/Threads/Meta.

They aren't some new, bright-eyed group with no track record. They're a borderline Machiavellian megacorporation with a long and continuing history of extremely hostile actions:

• Helping enhance genocides in countries
• Openly and willingly taking part in political manipulation (see Cambridge Analytica)
• Actively have campaigned against net neutrality and attempted to make "facebook" most of the internet for members of countries with weaker internet infra - directly contributing to their amplification of genocide (see the genocide link for info)
• Using their users as non-consenting subjects to psychological experiments.
• Absolutely ludicrous invasions of privacy - even if they aren't able to do this directly to the Fediverse, it illustrates their attitude.
• Even now, they're on-record of attempting to get instance admins to do backdoor discussions and sign NDAs.

Yes, I know one of the Mastodon folks have said they're not worried. Frankly, I think they're being laughably naive >.<. Facebook/Meta - and Instagram's CEO - might say pretty words - but words are cheap and from a known-hostile entity like Meta/Facebook they are almost certainly just a manipulation strategy.

In my view, they should be discarded as entirely irrelevant, or viewed as deliberate lies, given their continued atrocious behaviour and open manipulation of vast swathes of the population.

Facebook have large amounts of experience on how to attack and astroturf social media communities - hell I would be very unsurprised if they are already doing it, but it's difficult to say without solid evidence ^.^

Why should we believe anything they say, ever? Why should we believe they aren't just trying to destroy a competitor before it gets going properly, or worse, turn it into yet another arm of their sprawling network of services, via Embrace, Extend, Extinguish - or perhaps Embrace, Extend, Consume would be a better term in this case?

When will we ever learn that openly-manipulative, openly-assimilationist corporations need to be shoved out before they can gain any foothold and subsume our network and relegate it to the annals of history?

I've seen plenty of arguments claiming that it's "anti-open-source" to defederate, or that it means we aren't "resilient", which is wrong ^.^:

• Open source isn't about blindly trusting every organisation that participates in a network, especially not one which is known-hostile. Threads can start their own ActivityPub network if they really want or implement the protocol for themselves. It doesn't mean we lose the right to kick them out of most - or all - of our instances ^.^.
• Defederation is part of how the fediverse is resilient. It is the immune system of the network against hostile actors (it can be used in other ways, too, of course). Facebook, I think, is a textbook example of a hostile actor, and has such an unimaginably bad record that anything they say should be treated as a form of manipulation.

Edit 1 - Some More Arguments

In this thread, I've seen some more arguments about Meta/FB federation:

• Defederation doesn't stop them from receiving our public content:
  • This is true, but very incomplete. The content you post is public, but what Meta/Facebook is really after is having their users interact with content. Defederation prevents this.
• Federation will attract more users:
  • Only if Threads makes it trivial to move/make accounts on other instances, and makes the fact it's a federation clear to the users, and doesn't end up hosting most communities by sheer mass or outright manipulation.
  • Given that Threads as a platform is not open source - you can't host your own "Threads Server" instance - and presumably their app only works with the Threads Server that they run - this is very unlikely. Unless they also make Threads a Mastodon/Calckey/KBin/etc. client.
  • Therefore, their app is probably intending to make itself their user's primary interaction method for the Fediverse, while also making sure that any attempt to migrate off is met with unfamiliar interfaces because no-one else can host a server that can interface with it.
  • Ergo, they want to strongly incentivize people to stay within their walled garden version of the Fediverse by ensuring the rest remains unfamiliar - breaking the momentum of the current movement towards it. ^.^
• We just need to create "better" front ends:
  • This is a good long-term strategy, because of the cycle of enshittification.
  • Facebook/Meta has far more resources than us to improve the "slickness" of their clients at this time. Until the fediverse grows more, and while they aren't yet under immediate pressure to make their app profitable via enshittification and advertising, we won't manage >.<
  • This also assumes that Facebook/Meta won't engage in efforts to make this harder e.g. Embrace, Extend, Extinguish/Consume, or social manipulation attempts.
  • Therefore we should defederate and still keep working on making improvements. This strategy of "better clients" is only viable in combination with defederation.

PART 2 (post got too long!)

Edit: obligatory explanation (thanks mods for squaring me away)...

What you see via the UI isn't "all that exists". Unlike Reddit, where everything is a black box, there are a lot more eyeballs who can see "under the hood". Any instance admin, proper or rogue, gets a ton of information that users won't normally see. The attached example demonstrates that while users will only see upvote/downvote tallies, admins can see who actually performed those actions.

Edit: To clarify, not just YOUR instance admin gets this info. This is ANY instance admin across the Fediverse.

Source by joedeandev

Another day, another update.

More troubleshooting was done today. What did we do:

• Yesterday evening @phiresky@[email protected] did some SQL troubleshooting with some of the lemmy.world admins. After that, phiresky submitted some PRs to github.
• @[email protected] created a docker image containing 3PR's: Disable retry queue, Get follower Inbox Fix, Admin Index Fix
• We started using this image, and saw a big drop in CPU usage and disk load.
• We saw thousands of errors per minute in the nginx log for old clients trying to access the websockets (which were removed in 0.18), so we added a return 404 in nginx conf for /api/v3/ws.
• We updated lemmy-ui from RC7 to RC10 which fixed a lot, among which the issue with replying to DMs
• We found that the many 502-errors were caused by an issue in Lemmy/markdown-it.actix or whatever, causing nginx to temporarily mark an upstream to be dead. As a workaround we can either 1.) Only use 1 container or 2.) set ~~proxy_next_upstream timeout;~~ max_fails=5 in nginx.

Currently we're running with 1 lemmy container, so the 502-errors are completely gone so far, and because of the fixes in the Lemmy code everything seems to be running smooth. If needed we could spin up a second lemmy container using the ~~proxy_next_upstream timeout;~~ max_fails=5 workaround but for now it seems to hold with 1.

Thanks to @[email protected] , @[email protected] , @[email protected], @[email protected] , @[email protected] , @[email protected] for their help!

And not to forget, thanks to @[email protected] and @[email protected] for their continuing hard work on Lemmy!

And thank you all for your patience, we'll keep working on it!

Oh, and as bonus, an image (thanks Phiresky!) of the change in bandwidth after implementing the new Lemmy docker image with the PRs.

Edit So as soon as the US folks wake up (hi!) we seem to need the second Lemmy container for performance. So that's now started, and I noticed the proxy_next_upstream timeout setting didn't work (or I didn't set it properly) so I used max_fails=5 for each upstream, that does actually work.

Welcome to the fediverse!