Pulse of Truth

648 readers

287 users here now

Cyber Security news and links to cyber security stories that could make you go hmmm. The content is exactly as it is consumed through RSS feeds and wont be edited (except for the occasional encoding errors).

This community is automagically fed by an instance of Dittybopper.

founded 1 year ago

MODERATORS

[email protected]

130

AI haters build tarpits to trap and trick AI scrapers that ignore robots.txt (arstechnica.com)

submitted 2 days ago by [email protected] to c/[email protected]

15 comments fedilink hide all child comments

Attackers explain how an anti-spam defense became an AI weapon.

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 18 points 2 days ago (11 children)

I can save you a lot of trouble, actually. You don't need all of this!

Just make a custom 404 page that returns 13 MBs of junk along with status code 200 and has a few dead links (404, so it just goes to itself)

There are no bots on the domain I do this on anymore. From swarming to zero in under a week.

You don't need tar pits or heuristics or anything else fancy. Just make your website so expensive to crawl that it's not worth it so they filter themselves.

[–] [email protected] 1 points 1 day ago (1 children)

Surely any competent web scraper will avoid an infinite loop?

[–] [email protected] 1 points 1 day ago

Critics debating Nepenthes' utility on Hacker News suggested that most AI crawlers could easily avoid tarpits like Nepenthes, with one commenter describing the attack as being "very crawler 101." Aaron said that was his "favorite comment" because if tarpits are considered elementary attacks, he has "2 million lines of access log that show that Google didn't graduate."

You assume incorrectly that bots, scrapers and drive-by malware attacks are made by competent people. I have years worth of stories I'm not going to post on the open internet that says otherwise. I also have months worth of access logs that say otherwise. AhrefsBot in particular is completely unable to deal with anything you throw at it. It spent weeks in a tarpit I made very similar to the one in the article, looping links, until I finally put it out of its misery.

load more comments (9 replies)