Reddit says that it doesn’t want companies scraping the site for AI. Microsoft says it’s not doing that.
  • Moonrise2473@feddit.it
    10·
    2 months ago

    for common people they respect and even warn a webmaster if they submit a sitemap that has paths included in robots.txt