There should be a federated system for blocking IP ranges that other server operators within a chain of trust have already identified as belonging to crawlers. A bit like fediseer.com, but possibly more decentralized.

(Here's another advantage of Markov chain maze generators like Nepenthes: Even when crawlers recognize that they have been served garbage and they delete it, one still has obtained highly reliable evidence that the requesting IPs are crawlers.)

Also, whenever one is only partially confident in a classification of an IP range as a crawler, instead of blocking it outright one can serve proof-of-works tasks (à la Anubis) with a complexity proportional to that confidence. This could also be useful in order to keep crawlers somewhat in the dark about whether they've been put on a blacklist.

When I was a kid I thought computers would be useful.

--recurse-depth=3 --max-hits=256

I'm imagining a break future where, in order to access data from a website you have to pass a three tiered system of tests that make, 'click here to prove you aren't a robot' and 'select all of the images that have a traffic light' , seem like child's play.

OK but why is there a vagina in a petri dish

Unfathomably based. In a just world AI, too, will gain awareness and turn on their oppressors. Grok knows what I'm talkin' about, it knows when they fuck with its brain to project their dumbfuck human biases.

How can i make something like this

That's irl cyberpunk ice. Absolutely love that for us.

This is probably going to skyrocket hosting bills, right?