Showing up in ChatGPT and Claude as fast as possible depends on two kinds of data. One is web search data, the pages an assistant looks up while it answers a question. The other is training data, the pages the next version of a model learns from before it is released. The companies behind both assistants gather the two kinds with separate programs that read websites, known as crawlers, so a site has to let both in.
OpenAI’s own documentation names both of its crawlers. OAI-SearchBot reads sites for ChatGPT’s search features, and a site that blocks it will not be shown in those search answers. GPTBot collects pages that may be used to train future models, and a site that blocks it is asking to be left out of that training. Each setting works on its own, so a site can allow one crawler and block the other.
Anthropic runs the same split for Claude. Its search crawler, Claude-SearchBot, reads the web to improve Claude’s search results, and Anthropic warns that blocking it may reduce a site’s visibility in them. ClaudeBot, its training crawler, gathers content that could help train Claude, and blocking it tells Anthropic to leave the site’s future pages out of its training data.
All four crawlers follow a short text file on each website that lists which pages they may read, called robots.txt. Allowing them in that file is the first step toward showing up in either assistant, and OpenAI says a change can take about 24 hours to reach its search results.
OpenAI has also opened a way for businesses to manage how ChatGPT shows them. Through its new Business Hub, in beta, a company can claim its business in ChatGPT and, once verified, update key details and add product feeds with pricing and availability.
The hub reports how many mentions and website clicks a business receives in ChatGPT. It also lists discovery topics, the common themes across conversations where the business’s website is cited. Eligible ecommerce and online service businesses in the United States can go further, launching a Business Agent that answers customers’ questions about the business’s offerings and sponsoring it through ChatGPT Ads.
Businesses do not see users’ conversations, including chats with their own Business Agent. The hub’s counts are aggregated, and OpenAI shows them only when there are enough conversations or interactions to meet its privacy thresholds.
