this post was submitted on 02 Jun 2025
34 points (100.0% liked)

TechTakes

1901 readers
179 users here now

Big brain tech dude got yet another clueless take over at HackerNews etc? Here's the place to vent. Orange site, VC foolishness, all welcome.

This is not debate club. Unless it’s amusing debate.

For actually-good tech, you want our NotAwfulTech community

founded 2 years ago
MODERATORS
dgerard@awful.systems
34
Fighting the AI scraper bots at Pivot to AI and RationalWiki (pivot-to-ai.com)
submitted 3 days ago* (last edited 3 days ago) by dgerard@awful.systems to c/techtakes@awful.systems
6 comments fedilink hide all child comments
 

video version

that s3kr1t method: https://www.jwz.org/blog/2025/05/user-agent-blocking/#comment-259266

you are viewing a single comment's thread
view the rest of the comments
[–] HedyL@awful.systems 7 points 3 days ago* (last edited 3 days ago)

This is just naive web crawling: Crawl a page, extract all the links, then crawl all the links and repeat.

It's so ridiculous - supposedly these people have access to a super-smart AI (which is supposedly going to take all our jobs soon), but the AI can't even tell them which pages are worth scraping multiple times per second and which are not. Instead, they appear to kill their hosts like maladapted parasites regularly. It's probably not surprising, but still absurd.

Edit: Of course, I strongly assume that the scrapers don't use the AI in this context (I guess they only used it to write their code based on old Stackoverflow posts). Doesn't make it any less ridiculous though.