There are numerous reports that Google may be using a more generic and undocumented Googlebot, Google crawler. The crawler user agent is just named "Google," and it is not listed in any of the documented crawlers that Google has provided.
  • Flyswat@lemmy.dbzer0.com
    English
    16·
    5 days ago

    Can we get rid of the current model where websites are financed by ads and have instead these AI companies pay the websites they crawl?

    I could really do with less ads if they are going to crawl the web anyway.