ChatGPT is once again trying to compete with Google Search, and this time it has a partner with a rare view of the web. OpenAI, which makes ChatGPT, keeps a store of web pages the chatbot draws on for answers, called a search index. Cloudflare, which operates one of the world’s largest networks, has explained how data from that network is helping fill it. Helpful pages, the company points out, sometimes have few other sites linking to them, and Google uses links to find new pages.
The two companies began a research pilot in July. OpenAI uses software that visits web pages and reads them for the index, called a crawler. Cloudflare’s data shows the crawler which pages have changed and which addresses lead to missing or redirected pages. The crawler uses that to decide where to go next, while OpenAI alone ranks the pages. Data from sites making up about 1% of the network helped the crawler find hundreds of millions of new pages.
That data also tells the crawler what to skip, because revisiting every page more often would cost everyone too much. Filters drop missing and redirected pages before the crawler requests them, and unchanged pages wait for a change before the next visit.
Choosing what to fetch and keep is a problem for Google Search too. With no central list of the web, it must keep hunting for new and updated pages. Its official guide says no page is guaranteed a place in the index, and a page left out never appears in results, even a helpful one.
The pilot’s data covers traffic quality, how recent a page’s content is and whether it has changed, none of which depends on who links to a page. The design points toward an index with less need for links, though Cloudflare stops short of saying so. Such an index would give rarely linked pages another way in.
Google starts with a large lead in data and experience, but Cloudflare brings data of another kind. More than a fifth of the web sits behind its network, which the company says gives it a unique, real-time view of page changes and traffic. A crawler learns about a change only by visiting a page again. From inside the network, the change shows up as traffic moves through.
