It’s an attempt to stop poorly behaved AI crawlers written by someone who had their Git server flooded Amazon’s crawler. The worst part is that it seems these companies are so desperate to suck up data to feed their models that they’ll happily disregard “good etiquette” and summarily workaround the website owner’s attempt to slow them down.

I don’t want to have to close off my Gitea server to the public, but I will if I have to. It’s futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more. I just want the requests to stop.

Source: Amazon’s AI crawler is making my git server unstable by Xe Iaso

Given the Gnome gitlab is a rather known website, I can only imagine they’ve suffered from the same fate, probably even worse.