Robots.txt Tester & Crawl Directive Auditor

ROBOTS.TXT DIRECTIVES & TEST URL

Robots.txt Content

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Disallow: /*.json$

User-agent: GPTBot
Disallow: /

User-agent: Googlebot
Allow: /
Disallow: /internal-search/

Sitemap: https://nadhebe.com/sitemap-index.xml

User-Agent Crawler

Test URL Path

CRAWL DIRECTIVE AUDIT RESULT

Matching Directive Rule

Evaluated User-Agent: Googlebot

Evaluated Path: /admin/dashboard

Matching Rule: Disallow: /admin/

Detected Sitemap Directives

Robots.txt Crawl Directive Auditor — Guide & Best Practices

The robots.txt file serves as the traffic controller for search engine web crawlers. It instructs Googlebot, Bingbot, and AI web scrapers which directories, files, or URL parameters they are permitted (or forbidden) to crawl.

Key Robots.txt Directives Explained

User-agent: — Specifies which search engine crawler the following rules apply to. Using * targets all crawlers.
Disallow: — Blocks crawlers from accessing URLs that match the path pattern.
Allow: — Explicitly permits access to a subpath within a previously disallowed directory.
Sitemap: — Provides absolute URLs pointing to XML sitemap index files.

Blocking AI Web Scrapers vs Search Engine Crawlers

Modern site owners frequently want to allow search engine bots (like Googlebot and Bingbot) to index their content for organic search traffic, while restricting AI training crawlers (such as OpenAI's GPTBot or Common Crawl's CCBot).

# Allow Google Search Crawlers
User-agent: Googlebot
Allow: /

# Block AI Training Scrapers
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

Common Robots.txt Pitfalls

Disallowing Entire Site: A trailing slash on Disallow: / blocks search engines from crawling your entire website!
Assuming Disallow = Noindex: Disallowing a URL in robots.txt does NOT prevent Google from displaying the URL in search results if external links exist. Use a noindex meta tag instead.
Blocking CSS/JS Assets: Blocking /css/ or /js/ directories prevents Googlebot from rendering your mobile layout, causing ranking drops.

Related Tools

Combine robots directive auditing with XML sitemap validation using our Sitemap & Robots Analyzer, audit meta tags with the OpenGraph & Meta Tag Checker, or preview Google search listings with our SERP Snippet Simulator.

Frequently Asked Questions

Common questions about this tool.

Does Disallow in robots.txt prevent Google from indexing a URL? ▼

No. `Disallow` prevents Googlebot from downloading and crawling the page content, but Google can still index the URL if it receives inbound links from external websites. To prevent indexing completely, use a `<meta name="robots" content="noindex">` tag.

How do I block AI web scrapers like GPTBot or CCBot in robots.txt? ▼

Add specific `User-agent` directives followed by `Disallow: /`. For example: User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: /

Where should the Sitemap directive be placed in robots.txt? ▼

Place the `Sitemap: https://yourdomain.com/sitemap.xml` directive at the bottom or top of your robots.txt file. It should use an absolute HTTPS URL.

What is the wildcard (*) character in robots.txt? ▼

The asterisk wildcard (`*`) matches any sequence of characters in path rules. For example, `Disallow: /*.pdf$` blocks all PDF files across your domain.

How frequently does Google re-fetch robots.txt files? ▼

Googlebot typically caches `robots.txt` files for up to 24 hours. Updating rules in your file usually takes effect within a single day.