Bots are currently scraping the internet for LLM training data at unprecedented rates[1][2][3], driving up costs and destabilizing public-facing websites. I want to talk about how this has been particularly difficult for wikis, and has gotten much worse in the last few months.
Ai scrapers are making it kinda suck to run anything on the internet to be honest.
Making it kinda suck to use the Internet at all. I try to look things up and its page after page of AI slop. We are going to end up back in the days of Altavista.
For some reason, for the past month or so, when I google stuff the first result is ‘Can’t generate an AI overview right now, try again later’. Maybe one time in 50 it’ll do the AI response.
I don’t know if I changed a setting and forgot, or if they just don’t like me, or if their local processing can’t hack it. Either way, it makes me actually go to the websites, like the good old days of a couple of years ago! And I find better info doing that.
Honestly, the AI overview is crap. One time, searching for legal stuff about a potentially precedent setting case I’m involved in, it responded referring specifically to my case - as though it had been resolved and the precedent was set.
So yeah, I wouldn’t trust that shit to provide me a recipe for biscuits.
i am actively looking for a google alternative, they are horrible with their summaries and because it sounds confident, it throws you off and derails actually finding an answer. their search has been seo optimized clickbait for years but now with reddit becoming ai slop and another billionaire persona cleansing space its looking bleak
If you don’t mind paying for it, I like Kagi. There are some people who don’t for reasons outside of its performance, so make sure you do your research if you care about those things.
For free, DDG isn’t bad, but there’s also options that can be self-hosted or you can find other people’s hosted instances: SearXNG and LibreY for example. (I’d link to some but I don’t want to accidentally get them hugged to death since they’re being run by small independent servers.) I don’t use any of the above free options so I can’t comment on which one’s better or not but at least I am aware they exist. I know SearXNG lets you also pick and choose which search engine services are used in your searches.