Robots.txt & AI Crawlers in 2026: How to Control GPTBot, ClaudeBot & Perplexity Without Hurting Google SEO

Hamza Fazal, Founder, DEESUHamza FazalFounder, DEESU10 min readWeb Development

For two decades, robots.txt was one of the sleepiest files in any repository. You added User-agent: *, blocked /admin and /search, pointed to sitemap.xml, and forgot about it for five years. That era ended abruptly. Across production web logs in 2026, automated AI crawlers and scraping fleets now account for 35% to 60% of total non-cached origin requests on documentation hubs, engineering blogs, SaaS knowledge bases, and e-commerce catalogs.

Left unmanaged, aggressive AI crawlers create two simultaneous headaches: skyrocketing serverless/origin compute bills from millions of uncached dynamic route fetches, and unauthorized ingestion of proprietary technical content into commercial foundation models. Yet swinging to the opposite extreme—blocking every AI user-agent blindly—cuts your brand out of AI Search and live citation engines like ChatGPT Search, Perplexity, and Google AI Overviews, which increasingly drive high-intent developer and enterprise referral traffic.

Modern crawler governance requires surgical precision. In this guide, we map the 2026 AI crawler ecosystem, explain RFC 9309 precedence rules and wildcard pitfalls, show how to combine robots.txt with X-Robots-Tag headers and edge WAF rules, and optimize crawl budget for Next.js and large dynamic sites. To build and validate a tailored crawler policy interactively, open our Robots.txt & AI Bot Blocker.

2026 AI Crawler Governance Architecture separating Search Citation Bots from Foundation Model Training Scrapers
2026 Bot Governance Decision Flow: Allow Search & Citation crawlers (Googlebot, OAI-SearchBot, PerplexityBot) for organic discovery while isolating Model Training scrapers (GPTBot, ClaudeBot, Google-Extended, CCBot) via robots.txt and edge WAF enforcement.

1. Separating Search & Citation Bots from Foundation Model Training Scrapers

The single most important concept in 2026 technical SEO is that major AI labs now split their crawling infrastructure into distinct user-agent tokens with legally and operationally separate purposes. Confusing a lab's search citation bot with its model training bot can either accidentally hide you from AI search citations or inadvertently grant blanket training access.

Take Google as the canonical example. `Googlebot` crawls pages for traditional Google Search indexing, rankings, and Featured Snippets. `Google-Extended` is a standalone product control token that governs whether Google may use your crawled pages to train future Gemini models and Vertex AI generative APIs. Disallowing Google-Extended in robots.txt has zero negative effect on your Google Search rankings, crawl frequency, or organic clicks—Google officially guarantees complete separation between the two controls. Apple follows an identical model: Applebot powers Siri and Spotlight search results, while Applebot-Extended controls Apple Intelligence foundation model training.

OpenAI draws an equally important boundary between `GPTBot` (bulk web scraping used to pre-train and fine-tune GPT foundation models), `OAI-SearchBot` (indexing pages to surface clickable citations inside ChatGPT Search), and `ChatGPT-User` (real-time on-demand fetches triggered when a live user pastes your URL into a chat prompt). Most publishers in 2026 adopt a Allow Search, Block Training posture: permitting OAI-SearchBot and PerplexityBot to earn high-converting referral links while disallowing GPTBot, ClaudeBot, Google-Extended, CCBot, and Bytespider.

User-Agent TokenOperatorPrimary PurposeAffects Search SEO?Typical Publisher Policy
GooglebotGoogleGoogle Search indexing & rankingYes (critical)Allow (optimize crawl budget)
Google-ExtendedGoogleGemini & Vertex AI model trainingNo (strictly separated)Disallow to protect IP
OAI-SearchBotOpenAIChatGPT Search links & citationsYes (AI Search visibility)Allow for referral traffic
GPTBotOpenAIFoundation model training datasetNoDisallow unless licensed
ClaudeBotAnthropicClaude foundation model trainingNoDisallow (aggressive volume)
PerplexityBotPerplexity AILive search index & cited answersYes (AI Search citations)Allow (rate-limit if needed)
Applebot-ExtendedAppleApple Intelligence model trainingNo (Applebot handles search)Disallow to opt out of training
CCBotCommon CrawlOpen web archive used by dozens of LLMsNoDisallow (blocks downstream training)
BytespiderByteDanceDoubao/TikTok AI training & scrapingNoDisallow + WAF rate-limit
Major Search, Citation, and AI Training Crawlers in 2026: Purpose, Impact, and Recommended Policy

2. RFC 9309 Precedence Rules and Subtle Wildcard Bugs

In September 2022, the IETF formally standardized the Robots Exclusion Protocol as RFC 9309, codifying exact parsing rules that Google, Bing, OpenAI, and modern compliant bots follow. Unfortunately, two RFC 9309 behaviors routinely trip up even senior engineers:

  • **Specific User-agent groups do NOT inherit User-agent: * rules:** This is the #1 bug in production robots.txt files. Suppose you block /admin/ and /api/ under User-agent: *, and then add a separate block User-agent: OAI-SearchBot with Allow: /blog/. Because OAI-SearchBot matches its own explicit group, it completely ignores the User-agent: * group—meaning OAI-SearchBot is now permitted to crawl /admin/ and /api/ unless you repeat those Disallow lines inside its group!
  • Longest-match-wins precedence (not top-to-bottom order): Within a matched group, crawlers do not stop at the first matching rule. Instead, RFC 9309 compares the character length of matching Allow and Disallow path prefixes; the longest match wins, and if both are identical in length, Allow wins. Thus Allow: /blog/public/ overrides Disallow: /blog/ regardless of which line comes first.
  • **Wildcards (*) and end-of-URL anchors ($):** Trailing $ anchors match exact paths or file extensions (Disallow: /*.json$), while * matches zero or more arbitrary characters. Remember that all paths in robots.txt are case-sensitive (/Admin does not block /admin) and apply strictly to the exact host, scheme, and port (https://docs.example.com/robots.txt is independent from https://example.com/robots.txt).
# 1. Foundation Model Training Scrapers — Explicit Opt-Out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Meta-ExternalAgent
Disallow: /

# 2. Search & AI Citation Crawlers — Allow Public Content, Protect Crawl Budget
User-agent: Googlebot
User-agent: Bingbot
User-agent: OAI-SearchBot
User-agent: PerplexityBot
Allow: /
Allow: /_next/static/
Disallow: /api/
Disallow: /admin/
Disallow: /dashboard/
Disallow: /*?*sort=
Disallow: /*?*filter=

# 3. Default Fallback for All Other Compliant Crawlers
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /dashboard/
Disallow: /search

Sitemap: https://example.com/sitemap.xml
Production-grade RFC 9309 robots.txt: Allow Search & Citations, Block Training Scrapers

3. Why `robots.txt` Is NOT Access Control (WAF + `X-Robots-Tag`)

A critical security truth every engineering team must internalize: `robots.txt` is an honor system, never an access-control boundary. Publishing Disallow: /internal-staging/ does not prevent anyone from requesting /internal-staging/—worse, it acts as a public reconnaissance map telling attackers and rogue scrapers exactly where sensitive endpoints live. Furthermore, robots.txt only stops *crawling*, not *indexing*: if external sites link to a disallowed URL, Google can still display that URL in search results with a placeholder snippet unless the page serves a noindex directive.

To enforce genuine boundaries against non-compliant AI scrapers and prevent private assets (PDFs, staging environments, JSON feeds) from appearing in search indexes, layer two technical controls alongside robots.txt:

  • `X-Robots-Tag` HTTP response headers: Unlike HTML <meta name="robots"> tags, X-Robots-Tag works on non-HTML responses including PDFs, images, API endpoints, and entire staging subdomains. Sending X-Robots-Tag: noindex, nofollow, noai, noimageai guarantees search engines drop the URL from their index immediately—just remember you must NOT block the URL in robots.txt if you want search bots to crawl the header and process the noindex instruction.
  • Edge WAF & Verified Bot Enforcement (Cloudflare / AWS WAF): Rogue scrapers frequently spoof Mozilla/5.0 Chrome user-agents or ignore robots.txt entirely. Use Cloudflare Bot Management or reverse-DNS/IP-range verification (OpenAI, Google, Anthropic, and Perplexity all publish official JSON IP range lists) to verify legitimate bots while challenging or rate-limiting unverified datacenter ASNs that scrape HTML at high velocity.
import { NextRequest, NextResponse } from "next/server";

export function middleware(request: NextRequest) {
  const response = NextResponse.next();
  const host = request.headers.get("host") ?? "";

  // Prevent indexing of preview deployments and staging subdomains
  if (host.includes("staging.") || host.endsWith(".vercel.app")) {
    response.headers.set("X-Robots-Tag", "noindex, nofollow, noarchive");
  }

  // Keep downloadable customer PDFs out of search and AI indexes
  if (request.nextUrl.pathname.startsWith("/exports/")) {
    response.headers.set("X-Robots-Tag", "noindex, noai, noimageai");
  }

  return response;
}
Next.js Middleware applying X-Robots-Tag on staging environments and private file routes

4. Crawl Budget Optimization for Large Sites and Next.js Apps

For sites with more than 10,000 indexable URLs—or dynamic Next.js applications with faceted catalogs, tags, and search filters—crawl budget directly governs how quickly new pages rank and how much server infrastructure you waste. Search bots allocate a finite host load budget based on your Time-to-First-Byte (TTFB) and 5xx error rate.

A notorious Next.js mistake is accidentally blocking /_next/ in robots.txt. Googlebot runs a headless Chromium rendering phase to evaluate layout shift, mobile usability, and hydrated content; blocking /_next/static/ prevents Googlebot from loading your JavaScript and CSS chunks, causing pages to render as broken unstyled HTML and tanking Core Web Vitals signals. Always explicitly allow /_next/static/ while disallowing /_next/data/ (if unused for public rendering), /api/, and infinite query-string permutations (?sort=, ?page= beyond reasonable depth, ?utm_* canonical noise). And once search and AI citation bots successfully crawl your public pages, make sure every route emits crisp social and citation previews by verifying your metadata with our Open Graph & Meta Tag Generator.

5. Production Crawler Governance & SEO Checklist

Audit your crawler setup against these six operational standards before shipping changes to production:

  • Decide explicitly between Search/Citation visibility (OAI-SearchBot, PerplexityBot, Googlebot) and Model Training opt-outs (GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended).
  • Duplicate critical Disallow paths (/api/, /admin/, faceted query params) inside every dedicated User-agent group because RFC 9309 does not inherit rules from User-agent: *.
  • Keep /_next/static/ and public CSS/JS/image assets crawlable so Googlebot's Web Rendering Service evaluates your pages identically to real users.
  • Never rely on robots.txt alone to hide sensitive URLs or de-index existing pages; enforce authentication for private data and use X-Robots-Tag: noindex for crawlable de-indexing.
  • Verify legitimate high-volume bots by official IP ranges or reverse DNS at your CDN/WAF layer, and apply rate limits to unverified datacenter ASNs ignoring robots.txt.
  • Include the absolute canonical Sitemap: https://yourdomain.com/sitemap.xml URL at the bottom of robots.txt and monitor 5xx responses on /robots.txt with synthetic alerting.

Frequently asked questions

No. Google explicitly maintains Google-Extended as a standalone product control token completely separate from Googlebot. Adding User-agent: Google-Extended followed by Disallow: / stops Google from using your published pages to train future Gemini foundation models and Vertex AI generative APIs, while leaving your standard Google Search crawling, indexing, organic rankings, Featured Snippets, and Google Discover eligibility completely untouched as long as Googlebot remains explicitly allowed.

No, provided you keep OAI-SearchBot explicitly allowed in your robots.txt file. OpenAI separates its crawler infrastructure cleanly by operational purpose: GPTBot gathers bulk web training corpora for future foundation models, whereas OAI-SearchBot indexes pages specifically to surface clickable links and live source citations inside ChatGPT Search. Blocking GPTBot while allowing OAI-SearchBot protects your intellectual property from model training while still capturing high-intent conversational search referral traffic.

Because robots.txt controls crawling, not indexing. If other websites or internal pages link to a disallowed URL, Google can still index the URL based on anchor text without ever fetching its body—displaying the infamous 'No information is available for this page' snippet. To remove a page from search results cleanly, allow it to be crawled temporarily and serve an X-Robots-Tag: noindex HTTP header or <meta name="robots" content="noindex"> tag.

Do not rely on Crawl-delay as your primary defense. Googlebot completely ignores Crawl-delay, and many aggressive AI scrapers either ignore it or run distributed worker pools that still saturate your origin. While Bingbot and Yandex honor Crawl-delay, the reliable engineering solution for protecting origin capacity is CDN caching for public HTML paired with Cloudflare or AWS WAF rate-limiting rules targeting high-frequency bot ASNs.

Under RFC 9309, crawlers are only required to read the first 500 KiB (512,000 bytes) of a robots.txt file and may silently ignore any directives beyond that limit, so keep your rules concise using prefix patterns rather than listing thousands of individual URLs. If /robots.txt returns a 404 Not Found, crawlers interpret that as full permission to crawl the entire site—whereas a 5xx server error causes Googlebot to halt crawling altogether.

Sources

ShareLinkedInX
Hamza Fazal, Founder, DEESU

Written by

Hamza Fazal

Founder, DEESU

Muhammad Hamza Fazal is the Founder of DEESU. An Android and full-stack web developer and digital marketer based in Islamabad, he founded DEESU in May 2022 and leads its engineering operations, product strategy, and client software delivery.

More from Hamza

Free tools for this topic

Need help with this?

Reply within 1 business day

Tell us what you're building.

Send the problem, not a polished brief. We'll tell you what it actually takes to ship it: stack, timeline and cost, before you commit to anything.