Andy Reid@lemmy.world to Technology@lemmy.worldEnglish · 5 months agoAI companies are violating a basic social contract of the web and and ignoring robots.txtwww.theverge.comexternal-linkmessage-square201fedilinkarrow-up11.09Karrow-down115cross-posted to: [email protected][email protected]technology
arrow-up11.07Karrow-down1external-linkAI companies are violating a basic social contract of the web and and ignoring robots.txtwww.theverge.comAndy Reid@lemmy.world to Technology@lemmy.worldEnglish · 5 months agomessage-square201fedilinkcross-posted to: [email protected][email protected]technology
minus-squaregayhitler420@lemm.eelinkfedilinkEnglisharrow-up1·5 months agoRobots.txt is supposed to help web crawlers to not waste their time, not enforce some kind of social norm. The logic was that if it’s actually important, you’ll put it behind a login.
Robots.txt is supposed to help web crawlers to not waste their time, not enforce some kind of social norm.
The logic was that if it’s actually important, you’ll put it behind a login.