I was scouring the indie-web earlier, and found a pretty useful list of bots to add to your robots.txt. But, since I’m not convinced that this is enough to keep them away, I also figured out a simple way to at least potentially completely block them from accessing your websites.
Yeah, that’s true. If they’re not using names, then there’s not a whole lot you can do. And blocking IPs is impossible, because they use different IPs constantly.
But!
With my post, and your suggestions, this is the “something” that’s better than doing absolutely nothing
I suppose it comes down to being offensive or defensive. I don’t think being defensive is worth my time. I’m not paying for bandwidth and compute-time is so cheap it’s irrelevant so I’m on the offensive. You can do both if you want. There’s definitely more ready-to-go defensive solutions than there are offensive (your own article, for example), but I think tinkering and adapting my own solution is fun. It’s like a game of cat and mouse but they have money to lose and I don’t.