I was scouring the indie-web earlier, and found a pretty useful list of bots to add to your robots.txt. But, since I’m not convinced that this is enough to keep them away, I also figured out a simple way to at least potentially completely block them from accessing your websites.
Eventually I’m gonna make a proper article about it, but what I’m doing right now boils down to this:
The next iteration of this will include a lot of uncompressed filler data so hopefully the bots have to download half a gigabyte of data every time they do this. I’m not paying for bandwidth, it doesn’t matter to me.
See for yourself https://drkt.eu/fdhasklfh
I can see that it works by just looking at my access logs.