I noted that crawling the entire web while keeping SEO‑spam out “costs a fortune”, which I see as the main economic barrier to building a truly AI‑driven Google‑style search engine.
crawling the web without getting buried in SEO spam costs a fortune
Web Crawling Costs
From x.com/dheekshithoff