Drunk & Root@sh.itjust.works to Selfhosted@lemmy.worldEnglish · 2 months agoHow to combat large amounts of Ai scrapersmessage-squaremessage-square54linkfedilinkarrow-up1120arrow-down14file-text
arrow-up1116arrow-down1message-squareHow to combat large amounts of Ai scrapersDrunk & Root@sh.itjust.works to Selfhosted@lemmy.worldEnglish · 2 months agomessage-square54linkfedilinkfile-text
everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
minus-squareDrunk & Root@sh.itjust.worksOPlinkfedilinkEnglisharrow-up1·2 months agosince its the frontends i run getting scraped its the robots.txt included there
since its the frontends i run getting scraped its the robots.txt included there