cm0002@lemmy.world to Technology@lemmy.worldEnglish · 2 months agoOpen Source devs say AI crawlers dominate traffic, forcing blocks on entire countriesarstechnica.comexternal-linkmessage-square25linkfedilinkarrow-up1389arrow-down14cross-posted to: OpenSource@europe.pubtechnology@lemmy.ziptechnology@lemmy.mlopensource@programming.dev
arrow-up1385arrow-down1external-linkOpen Source devs say AI crawlers dominate traffic, forcing blocks on entire countriesarstechnica.comcm0002@lemmy.world to Technology@lemmy.worldEnglish · 2 months agomessage-square25linkfedilinkcross-posted to: OpenSource@europe.pubtechnology@lemmy.ziptechnology@lemmy.mlopensource@programming.dev
minus-squaretallinkfedilinkEnglisharrow-up11·2 months agoThey can just interleave requests to different hosts. Honestly, someone spidering the whole Web probably should be doing that regardless.
They can just interleave requests to different hosts. Honestly, someone spidering the whole Web probably should be doing that regardless.