Linux

6583 readers

499 users here now

A community for everything relating to the GNU/Linux operating system

Also check out:

Original icon base courtesy of [email protected] and The GIMP

founded 2 years ago

MODERATORS

[email protected]

114

FOSS infrastructure is under attack by AI companies (thelibre.news)

submitted 10 hours ago* (last edited 10 hours ago) by [email protected] to c/[email protected]

20 comments fedilink hide all child comments

LLM scrapers are taking down FOSS projects' infrastructure, and it's getting worse.

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 44 points 9 hours ago* (last edited 9 hours ago) (9 children)

Wow that was a frustrating read. I dd not know it was quite that bad. Just to highlight one quote

they don’t just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don’t give a single flying fuck about robots.txt, because why should they. [...] If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet.

[–] [email protected] 18 points 9 hours ago (8 children)

the solution here is to require logins. thems the breaks unfortunately. it'll eventually pass as the novelty wears off.

[–] [email protected] 8 points 8 hours ago (2 children)

Next you'll have to invest in preventing automated signups

[–] [email protected] 3 points 8 hours ago

Signups in most platforms are quite hard. Straight up give your phone and do SMS verification, or at least give email and to register that email you will have to provide phone anyway. Captchas nowadays became so hard that even humans struggle with them and it often takes multiple attempts to get it right.

load more comments (1 replies)

load more comments (6 replies)