AI Search News · issue 19
AI Search News: Week 46
AI Agents Start Paying to Crawl Your Site
This week the business model behind AI crawlers came into sharper focus, with Cloudflare describing deals where OpenAI pays for efficient access to web pages. OpenAI also previewed a dashboard that lets businesses claim their presence inside ChatGPT, while Google tightened its stance on mass AI-generated content and reshuffled how AI answers display ads. Separately, competing data sets disagreed sharply on how much real traffic ChatGPT actually sends to websites compared to Google. Together, these stories sketch a shift from blocking AI bots on principle to treating them as a new, sometimes paying, audience.
This week's stories
What changed this week
Cloudflare: AI Agent Optimization Is the New SEO
Who it affects: Every website that gets crawled by AI bots
Cloudflare CEO Matthew Prince said on the company's earnings call that optimizing for AI agents is becoming a new version of SEO. OpenAI is already paying Cloudflare for hints about when a webpage changes, so its crawler avoids re-fetching the same page repeatedly, and Prince expects other AI companies to do the same. He said some Cloudflare customers want as many AI agents visiting as possible, while others see them as a cost with no benefit and choose to block them for free. Looking ahead, he expects micropayments where agents that explain their purpose, like planning a family vacation, could get cheaper or free access because the visit creates business value. Cloudflare has already moved two payment products, Monetization Gateway and Pay Per Use, into beta.
What this means for your website
Decide now whether you want AI agent traffic on your site or want it blocked, since both are real options with different costs and benefits; check your Cloudflare dashboard's AI Crawl Control settings to allow or deny individual AI crawlers.
OpenAI Builds a Search-Console-Style Tool for ChatGPT
Who it affects: Businesses that want visibility in ChatGPT answers
OpenAI is developing a Business Hub feature that lets companies claim their business inside ChatGPT after verifying ownership, for example by connecting a Shopify store. The tool tracks how often a business is mentioned in ChatGPT conversations and shows impressions and clicks on links to the business's website, charted daily. It does not show which prompts triggered mentions, user-level data, or competitor benchmarks, so it cannot explain why numbers rise or fall. It appears separate from ChatGPT's advertising products and is aimed at organic responses.
What this means for your website
Start building a verifiable, up-to-date business presence, including a connectable storefront like Shopify, so you can claim your Business Hub listing as soon as it launches and get first-party data on ChatGPT citations.
Google Flags Mass AI Content, Tweaks Indexing and AI Ad Placement
Who it affects: Sites publishing large volumes of AI content, all sites in Google Search
This week's roundup covered Facebook's latest wave of AI-generated pages, which is likely to run into Google's scaled content abuse policy. Google confirmed that important URLs can stay in Search temporarily even after returning a 400 error status, and it updated its documentation on the crawled, currently not indexed status that some sites see in Search Console. Reddit's stock dropped sharply after its CEO confirmed declining Google Search traffic. On the results page, Google moved AI Overviews for brands to the right column and now allows up to four ads inside AI Mode.
What this means for your website
If you publish large amounts of AI-generated content, review Google's scaled content abuse policy before adding more, and check Search Console for a crawled, currently not indexed status on pages you expect to rank.
ChatGPT's Search Volume Looks Big, But Real Website Traffic Still Doesn't
Who it affects: Any site tracking where its traffic actually comes from
ChatGPT consumer product lead Adam Fry said ChatGPT's search-like queries are now roughly one-fifth the volume of Google's, counting every question that sends ChatGPT out to the web. But Ahrefs' tracker of real website visits found Google drove 24.74% of tracked traffic in October, versus just 0.36% from ChatGPT. Fry did not say which Google figure he used or over what time period his comparison covers. The two numbers measure different things: OpenAI is counting queries made inside ChatGPT, while Ahrefs is counting actual visits landing on websites.
What this means for your website
Do not let headline comparisons of search query volume distract you from what matters, your own traffic numbers; keep watching your analytics for ChatGPT referral share, which remains tiny next to Google's.
ChatGPT Ads Switches to Target-Cost Bidding Starting November 9
Who it affects: Businesses running ads inside ChatGPT
OpenAI emailed advertisers that starting Monday, November 9, Target CPA and Target CPC bidding options are being added to ChatGPT Ads Manager, replacing Max Bid and Max CPC for new campaigns. Existing campaigns keep their old bidding method for now, but Max Bid and Max CPC fully retire on January 1, 2027. OpenAI recommends advertisers confirm their Pixel or Conversions API is sending the right event data, and review recent actual cost-per-action or cost-per-click figures before setting new targets.
What this means for your website
If you advertise inside ChatGPT, review your conversion tracking and set realistic Target CPA or CPC figures now, rather than waiting for the January 2027 deadline to force the switch.
Weekly · 100% free
AI search: all you need to know, every Monday
One short email a week on what changed in AI search and what it means for your website. Plain language, always free. Also on LinkedIn.
Sources
Every source this week
- searchenginejournal.comCloudflare: AI Agent Optimization Is the New SEO
- seroundtable.comOpenAI Builds a Search-Console-Style Tool for ChatGPT
- seroundtable.comGoogle Flags Mass AI Content, Tweaks Indexing and AI Ad Placement
- searchenginejournal.comChatGPT's Search Volume Looks Big, But Real Website Traffic Still Doesn't
- seroundtable.comChatGPT Ads Switches to Target-Cost Bidding Starting November 9
Free GEO audit · about 60 seconds
Is your site ready for AI search?
Run a free GEO audit to see exactly what AI crawlers can and can't read on your page, with the fixes ranked.