For two decades, robots.txt was one of the sleepiest files in any repository. You added User-agent: *, blocked /admin and /search, pointed to sitemap.xml, and forgot about it for five years. That era ended abruptly. Across production web logs in 2026, automated AI crawlers and scraping fleets now account for 35% to 60% of total non-cached origin requests on documentation hubs, engineering blogs, SaaS knowledge bases, and e-commerce catalogs.
Left unmanaged, aggressive AI crawlers create two simultaneous headaches: skyrocketing serverless/origin compute bills from millions of uncached dynamic route fetches, and unauthorized ingestion of proprietary technical content into commercial foundation models. Yet swinging to the opposite extreme—blocking every AI user-agent blindly—cuts your brand out of AI Search and live citation engines like ChatGPT Search, Perplexity, and Google AI Overviews, which increasingly drive high-intent developer and enterprise referral traffic.
Modern crawler governance requires surgical precision. In this guide, we map the 2026 AI crawler ecosystem, explain RFC 9309 precedence rules and wildcard pitfalls, show how to combine robots.txt with X-Robots-Tag headers and edge WAF rules, and optimize crawl budget for Next.js and large dynamic sites. To build and validate a tailored crawler policy interactively, open our Robots.txt & AI Bot Blocker.

1. Separating Search & Citation Bots from Foundation Model Training Scrapers
The single most important concept in 2026 technical SEO is that major AI labs now split their crawling infrastructure into distinct user-agent tokens with legally and operationally separate purposes. Confusing a lab's search citation bot with its model training bot can either accidentally hide you from AI search citations or inadvertently grant blanket training access.
Take Google as the canonical example. `Googlebot` crawls pages for traditional Google Search indexing, rankings, and Featured Snippets. `Google-Extended` is a standalone product control token that governs whether Google may use your crawled pages to train future Gemini models and Vertex AI generative APIs. Disallowing Google-Extended in robots.txt has zero negative effect on your Google Search rankings, crawl frequency, or organic clicks—Google officially guarantees complete separation between the two controls. Apple follows an identical model: Applebot powers Siri and Spotlight search results, while Applebot-Extended controls Apple Intelligence foundation model training.
OpenAI draws an equally important boundary between `GPTBot` (bulk web scraping used to pre-train and fine-tune GPT foundation models), `OAI-SearchBot` (indexing pages to surface clickable citations inside ChatGPT Search), and `ChatGPT-User` (real-time on-demand fetches triggered when a live user pastes your URL into a chat prompt). Most publishers in 2026 adopt a Allow Search, Block Training posture: permitting OAI-SearchBot and PerplexityBot to earn high-converting referral links while disallowing GPTBot, ClaudeBot, Google-Extended, CCBot, and Bytespider.
| User-Agent Token | Operator | Primary Purpose | Affects Search SEO? | Typical Publisher Policy |
|---|---|---|---|---|
| Googlebot | Google Search indexing & ranking | Yes (critical) | Allow (optimize crawl budget) | |
| Google-Extended | Gemini & Vertex AI model training | No (strictly separated) | Disallow to protect IP | |
| OAI-SearchBot | OpenAI | ChatGPT Search links & citations | Yes (AI Search visibility) | Allow for referral traffic |
| GPTBot | OpenAI | Foundation model training dataset | No | Disallow unless licensed |
| ClaudeBot | Anthropic | Claude foundation model training | No | Disallow (aggressive volume) |
| PerplexityBot | Perplexity AI | Live search index & cited answers | Yes (AI Search citations) | Allow (rate-limit if needed) |
| Applebot-Extended | Apple | Apple Intelligence model training | No (Applebot handles search) | Disallow to opt out of training |
| CCBot | Common Crawl | Open web archive used by dozens of LLMs | No | Disallow (blocks downstream training) |
| Bytespider | ByteDance | Doubao/TikTok AI training & scraping | No | Disallow + WAF rate-limit |
2. RFC 9309 Precedence Rules and Subtle Wildcard Bugs
In September 2022, the IETF formally standardized the Robots Exclusion Protocol as RFC 9309, codifying exact parsing rules that Google, Bing, OpenAI, and modern compliant bots follow. Unfortunately, two RFC 9309 behaviors routinely trip up even senior engineers:
- **Specific
User-agentgroups do NOT inheritUser-agent: *rules:** This is the #1 bug in productionrobots.txtfiles. Suppose you block/admin/and/api/underUser-agent: *, and then add a separate blockUser-agent: OAI-SearchBotwithAllow: /blog/. BecauseOAI-SearchBotmatches its own explicit group, it completely ignores theUser-agent: *group—meaningOAI-SearchBotis now permitted to crawl/admin/and/api/unless you repeat thoseDisallowlines inside its group! - Longest-match-wins precedence (not top-to-bottom order): Within a matched group, crawlers do not stop at the first matching rule. Instead, RFC 9309 compares the character length of matching
AllowandDisallowpath prefixes; the longest match wins, and if both are identical in length,Allowwins. ThusAllow: /blog/public/overridesDisallow: /blog/regardless of which line comes first. - **Wildcards (
*) and end-of-URL anchors ($):** Trailing$anchors match exact paths or file extensions (Disallow: /*.json$), while*matches zero or more arbitrary characters. Remember that all paths inrobots.txtare case-sensitive (/Admindoes not block/admin) and apply strictly to the exact host, scheme, and port (https://docs.example.com/robots.txtis independent fromhttps://example.com/robots.txt).
# 1. Foundation Model Training Scrapers — Explicit Opt-Out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Meta-ExternalAgent
Disallow: /
# 2. Search & AI Citation Crawlers — Allow Public Content, Protect Crawl Budget
User-agent: Googlebot
User-agent: Bingbot
User-agent: OAI-SearchBot
User-agent: PerplexityBot
Allow: /
Allow: /_next/static/
Disallow: /api/
Disallow: /admin/
Disallow: /dashboard/
Disallow: /*?*sort=
Disallow: /*?*filter=
# 3. Default Fallback for All Other Compliant Crawlers
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /dashboard/
Disallow: /search
Sitemap: https://example.com/sitemap.xml3. Why `robots.txt` Is NOT Access Control (WAF + `X-Robots-Tag`)
A critical security truth every engineering team must internalize: `robots.txt` is an honor system, never an access-control boundary. Publishing Disallow: /internal-staging/ does not prevent anyone from requesting /internal-staging/—worse, it acts as a public reconnaissance map telling attackers and rogue scrapers exactly where sensitive endpoints live. Furthermore, robots.txt only stops *crawling*, not *indexing*: if external sites link to a disallowed URL, Google can still display that URL in search results with a placeholder snippet unless the page serves a noindex directive.
To enforce genuine boundaries against non-compliant AI scrapers and prevent private assets (PDFs, staging environments, JSON feeds) from appearing in search indexes, layer two technical controls alongside robots.txt:
- `X-Robots-Tag` HTTP response headers: Unlike HTML
<meta name="robots">tags,X-Robots-Tagworks on non-HTML responses including PDFs, images, API endpoints, and entire staging subdomains. SendingX-Robots-Tag: noindex, nofollow, noai, noimageaiguarantees search engines drop the URL from their index immediately—just remember you must NOT block the URL inrobots.txtif you want search bots to crawl the header and process thenoindexinstruction. - Edge WAF & Verified Bot Enforcement (Cloudflare / AWS WAF): Rogue scrapers frequently spoof
Mozilla/5.0Chrome user-agents or ignorerobots.txtentirely. Use Cloudflare Bot Management or reverse-DNS/IP-range verification (OpenAI, Google, Anthropic, and Perplexity all publish official JSON IP range lists) to verify legitimate bots while challenging or rate-limiting unverified datacenter ASNs that scrape HTML at high velocity.
import { NextRequest, NextResponse } from "next/server";
export function middleware(request: NextRequest) {
const response = NextResponse.next();
const host = request.headers.get("host") ?? "";
// Prevent indexing of preview deployments and staging subdomains
if (host.includes("staging.") || host.endsWith(".vercel.app")) {
response.headers.set("X-Robots-Tag", "noindex, nofollow, noarchive");
}
// Keep downloadable customer PDFs out of search and AI indexes
if (request.nextUrl.pathname.startsWith("/exports/")) {
response.headers.set("X-Robots-Tag", "noindex, noai, noimageai");
}
return response;
}4. Crawl Budget Optimization for Large Sites and Next.js Apps
For sites with more than 10,000 indexable URLs—or dynamic Next.js applications with faceted catalogs, tags, and search filters—crawl budget directly governs how quickly new pages rank and how much server infrastructure you waste. Search bots allocate a finite host load budget based on your Time-to-First-Byte (TTFB) and 5xx error rate.
A notorious Next.js mistake is accidentally blocking /_next/ in robots.txt. Googlebot runs a headless Chromium rendering phase to evaluate layout shift, mobile usability, and hydrated content; blocking /_next/static/ prevents Googlebot from loading your JavaScript and CSS chunks, causing pages to render as broken unstyled HTML and tanking Core Web Vitals signals. Always explicitly allow /_next/static/ while disallowing /_next/data/ (if unused for public rendering), /api/, and infinite query-string permutations (?sort=, ?page= beyond reasonable depth, ?utm_* canonical noise). And once search and AI citation bots successfully crawl your public pages, make sure every route emits crisp social and citation previews by verifying your metadata with our Open Graph & Meta Tag Generator.
5. Production Crawler Governance & SEO Checklist
Audit your crawler setup against these six operational standards before shipping changes to production:
- Decide explicitly between Search/Citation visibility (
OAI-SearchBot,PerplexityBot,Googlebot) and Model Training opt-outs (GPTBot,ClaudeBot,Google-Extended,CCBot,Applebot-Extended). - Duplicate critical
Disallowpaths (/api/,/admin/, faceted query params) inside every dedicatedUser-agentgroup because RFC 9309 does not inherit rules fromUser-agent: *. - Keep
/_next/static/and public CSS/JS/image assets crawlable so Googlebot's Web Rendering Service evaluates your pages identically to real users. - Never rely on
robots.txtalone to hide sensitive URLs or de-index existing pages; enforce authentication for private data and useX-Robots-Tag: noindexfor crawlable de-indexing. - Verify legitimate high-volume bots by official IP ranges or reverse DNS at your CDN/WAF layer, and apply rate limits to unverified datacenter ASNs ignoring
robots.txt. - Include the absolute canonical
Sitemap: https://yourdomain.com/sitemap.xmlURL at the bottom ofrobots.txtand monitor 5xx responses on/robots.txtwith synthetic alerting.
Frequently asked questions
Google-Extended as a standalone product control token completely separate from Googlebot. Adding User-agent: Google-Extended followed by Disallow: / stops Google from using your published pages to train future Gemini foundation models and Vertex AI generative APIs, while leaving your standard Google Search crawling, indexing, organic rankings, Featured Snippets, and Google Discover eligibility completely untouched as long as Googlebot remains explicitly allowed.OAI-SearchBot explicitly allowed in your robots.txt file. OpenAI separates its crawler infrastructure cleanly by operational purpose: GPTBot gathers bulk web training corpora for future foundation models, whereas OAI-SearchBot indexes pages specifically to surface clickable links and live source citations inside ChatGPT Search. Blocking GPTBot while allowing OAI-SearchBot protects your intellectual property from model training while still capturing high-intent conversational search referral traffic.robots.txt controls crawling, not indexing. If other websites or internal pages link to a disallowed URL, Google can still index the URL based on anchor text without ever fetching its body—displaying the infamous 'No information is available for this page' snippet. To remove a page from search results cleanly, allow it to be crawled temporarily and serve an X-Robots-Tag: noindex HTTP header or <meta name="robots" content="noindex"> tag.Crawl-delay as your primary defense. Googlebot completely ignores Crawl-delay, and many aggressive AI scrapers either ignore it or run distributed worker pools that still saturate your origin. While Bingbot and Yandex honor Crawl-delay, the reliable engineering solution for protecting origin capacity is CDN caching for public HTML paired with Cloudflare or AWS WAF rate-limiting rules targeting high-frequency bot ASNs.robots.txt file and may silently ignore any directives beyond that limit, so keep your rules concise using prefix patterns rather than listing thousands of individual URLs. If /robots.txt returns a 404 Not Found, crawlers interpret that as full permission to crawl the entire site—whereas a 5xx server error causes Googlebot to halt crawling altogether.Sources

Written by
Hamza Fazal
Founder, DEESU
Muhammad Hamza Fazal is the Founder of DEESU. An Android and full-stack web developer and digital marketer based in Islamabad, he founded DEESU in May 2022 and leads its engineering operations, product strategy, and client software delivery.
More from HamzaFree tools for this topic
Need help with this?

