

do they just want everything to be crawled
Yes. Web crawling has been a normal and vital part of the web from day 1. We’d have no search engines without crawlers.
The web is user-centric by design. I’m sick of tech companies trying to flip the script and hoard information, most of which is not theirs to begin with (e.g. Google, Reddit, Twitter, Facebook, etc.).
SQLite would definitely be smaller, faster, and require less memory.
Thing is, it’s 2025, roughly 20 years since anybody’s given half a shit about storage efficiency, memory efficiency, or even CPU efficiency for anything so small. Presumably this is not something they need to query dynamically.