Free tool

Robots.txt Generator & AI Bot Blocker

Generate a compliant RFC 9309 robots.txt file, selectively block AI training crawlers like GPTBot and ClaudeBot without hurting SEO, and test URL rules live.

Robots.txt Generator & AI Bot Blocker: the tool

Quick Policy Presets:

AI Crawler Control Grid (6/8 Blocked)

General Crawler Rules (`User-agent: *`) & Sitemap

Disallow/admin/
Disallow/api/
Disallow/private/

Generated robots.txt

# Generated with DEESU Robots.txt & AI Bot Blocker
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /private/

# AI Crawlers — Blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Applebot-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

Live URL Crawl Tester (RFC 9309)

ALLOWED
Googlebot

Path "/blog/how-to-optimize-seo" did not match any Disallow rule in the * (Wildcard Fallback) group.

Group: * (Wildcard Fallback)
Matched Rule: No blocking rule matched

Need technical SEO architecture, automated sitemaps, or AI search readiness?

DEESU builds fast, crawl-optimized web applications with clean structured data, SSR/prerendering, and disciplined crawler governance.

Improve My Technical SEO

How it works

How RFC 9309 Robots.txt Matching Actually Works

The Robots Exclusion Protocol—formally standardized in 2022 as RFC 9309—governs how compliant search engines and AI bots discover which URLs on your origin they are permitted to fetch. When a crawler visits your domain, it requests /robots.txt over HTTPS and looks for the most specific User-agent group matching its product token (for instance, GPTBot or Googlebot). If no dedicated group exists, it falls back to the wildcard User-agent: * block.

Crucially, modern crawlers do NOT evaluate Allow and Disallow rules top-to-bottom. Instead, RFC 9309 specifies longest-prefix matching: the rule with the longest matching path pattern wins regardless of line order. If an Allow rule and a Disallow rule match with the exact same character length, Allow wins. That is why WordPress sites can write Disallow: /wp-admin/ alongside Allow: /wp-admin/admin-ajax.php and safely expose the AJAX endpoint while blocking the rest of the admin dashboard.

Separating Search Indexing from AI Model Training Crawlers

Historically, blocking search bots meant disappearing from Google or Bing. Today, major AI labs operate separate user-agents for search retrieval versus foundation model training so publishers can stay visible in search results without donating proprietary content to training datasets:

  • OpenAI splits crawling between GPTBot (used to train future foundation models) and OAI-SearchBot (used to surface live citations in ChatGPT Search). Blocking GPTBot while allowing OAI-SearchBot preserves search referrals.
  • Google uses Googlebot for Search indexing and Google-Extended as an independent opt-out control for Gemini and Vertex AI training—blocking Google-Extended does not impact your Google Search rankings.
  • Anthropic's ClaudeBot, Common Crawl's CCBot (which feeds dozens of open-weight LLM datasets), ByteDance's Bytespider, and Apple's Applebot-Extended can each be targeted individually via explicit User-agent blocks.

Common Robots.txt Mistakes That Hurt SEO and Crawl Budget

A single typo in /robots.txt can deindex an entire site overnight. The most frequent production incident occurs when a staging environment's Disallow: / file is accidentally deployed to production, cutting off organic traffic within days. Always test your generated file against critical landing pages before pushing to main.

Another common misconception is using robots.txt to hide private data or remove already-indexed pages from Google. Remember that robots.txt is publicly readable at https://yourdomain.com/robots.txt—listing secret URLs like /secret-investor-deck/ exposes them to attackers. Moreover, if external sites link to a disallowed URL, Google may still index the bare URL without a snippet. Use proper authentication (401/403) for private pages and <meta name="robots" content="noindex"> or X-Robots-Tag: noindex headers to deindex public URLs.

Declaring XML Sitemaps and Optimizing Crawl-Delay

Including an absolute Sitemap: https://yourdomain.com/sitemap.xml directive at the bottom of your robots.txt gives Googlebot, Bingbot, and AI search crawlers an immediate discovery path to every canonical URL without relying solely on link discovery. Note that Google ignores Crawl-delay (use Google Search Console for crawl rate), whereas Bingbot, DuckDuckBot, and Yandex honor it—so only add Crawl-delay if non-Google bots are straining your origin server.

FAQ

Questions, answered

No. Google Search relies strictly on Googlebot. Blocking Google-Extended, GPTBot, ClaudeBot, CCBot, or Bytespider only opts your site out of AI training crawlers and has zero negative effect on Google Search indexing or rankings.

It must be served at the exact root path of each host and protocol—for example https://example.com/robots.txt. In Next.js, Vite, or React static sites, place it inside your public/robots.txt folder (or use app/robots.ts in Next.js App Router). Subdomains such as docs.example.com require their own separate /robots.txt.

No. Under RFC 9309, once a crawler finds a specific User-agent group matching its own name (such as User-agent: GPTBot), it exclusively follows that specific block and ignores the generic User-agent: * group.

No. robots.txt is a voluntary protocol respected by reputable search engines and major AI labs, not a security firewall. Malicious scrapers ignore it. Always enforce server-side authentication, WAF/Cloudflare Bot Fight rules, and rate limiting for sensitive endpoints.

Yes. RFC 9309 supports * to match any sequence of characters (e.g., Disallow: /*?sort=) and $ to anchor the end of a URL (e.g., Disallow: /*.pdf$). Our Live URL Rule Tester supports both * and $ anchors accurately.

More free tools

Reply within 1 business day

Tell us what you're building.

Send the problem, not a polished brief. We'll tell you what it actually takes to ship it: stack, timeline and cost, before you commit to anything.