Robots.txt Tester

What does Robots.txt Validator do?

The Robots.txt Tester fetches the file from a domain root or accepts pasted content, then parses user-agent groups, `Allow`, `Disallow`, and `Sitemap` directives. Enter a crawler name and URL path to get an ALLOWED or BLOCKED result with the rule that decided it. The diagnostic list also catches malformed lines before you publish them.

Put the file in the right place: Crawlers request `/robots.txt` at the root of an origin. A file at `https://example.com/folder/robots.txt` does not control that site. Protocol and host matter too: the file on `https://www.example.com/robots.txt` does not automatically govern `http://example.com`. Test the exact production origin after a host or HTTPS migration.

Read groups before rules: A `User-agent` line starts the group that receives the following `Allow` and `Disallow` rules. A rule placed before any user-agent has no group to apply to, and the tester reports it. `User-agent: *` is the wildcard fallback. A named group such as `User-agent: Googlebot` can produce a different result for Googlebot than the wildcard group produces for another crawler.

Use the winning match, not the first line: PageChecks compares rules that match the test path and uses the longest literal match. If two matches have equal specificity, `Allow` wins. With `Disallow: /private/` and `Allow: /private/public/`, the longer allow rule opens `/private/public/report`. A separate `Disallow: /private/public/drafts/` closes the more specific drafts directory again. The result panel shows the matched directive so you can trace that decision.

Know what an empty rule means: `Disallow: /` blocks every path for the selected group. `Disallow:` with no value blocks nothing. `Allow: /` explicitly permits the whole site for a crawler that supports Allow. One missing slash can reverse the intended policy, so test the home page, a protected directory, and one expected exception.

Use wildcards precisely: `` matches a sequence of characters, and a trailing `$` anchors the match to the end of the path. `Disallow: /.pdf$` targets paths ending in `.pdf`. Without `$`, the same pattern can also match a URL whose path continues after `.pdf`. Broad wildcard rules deserve path tests because a short pattern can cover more URLs than its author expected.

Separate crawling from indexing: Robots.txt controls whether a compliant bot may fetch a URL. It is not an index-removal mechanism. A blocked URL can still appear in search if an engine discovers it through links, and the crawler cannot see a page-level `noindex` while fetch access is blocked. Allow crawling and return a meta robots or `X-Robots-Tag: noindex` directive when removal from the index is the goal.

Test search and AI crawlers independently: `Googlebot`, `GPTBot`, `ClaudeBot`, and `PerplexityBot` are different user-agent tokens. A named block for one does not establish the result for the others. Enter each token and test the same important paths, especially documentation, articles, product pages, and assets required to understand those pages. The tester reports the applicable group and match for each run.

Do not block render assets by accident: Rules such as `Disallow: /assets/` or `Disallow: /static/` can prevent a crawler from retrieving CSS, JavaScript, or images used to render a page. Test a representative asset path as well as the HTML route. Restrict private content with authentication; robots.txt is public and does not stop a user from opening a disallowed URL.

Declare sitemap locations as absolute URLs: A line such as `Sitemap: https://example.com/sitemap.xml` can sit outside user-agent groups and may point to a sitemap index. Multiple Sitemap lines are valid. The tester lists discovered declarations and warns when none exists, but the sitemap must still return a usable XML response at the stated URL.

Review diagnostics in context: The parser reports missing colons, unknown directives, and rules without a preceding user-agent. An unknown directive may be a typo such as `Dissallow`, while an extension unsupported by this checker may need documentation from the crawler that uses it. Fix definite syntax errors, then run path tests for the bots that matter.

When should you use Robots.txt Validator?

Robots.txt Validator is most useful when you need a direct answer on a live URL or draft before you change templates, ship content, or rerun a wider audit.

How to Use This Tool

1. 1 Enter the exact production origin to fetch `/robots.txt`, or paste the proposed file content before deployment.
2. 2 Review diagnostics for missing colons, unknown directives, rules outside a user-agent group, and missing sitemap declarations.
3. 3 Choose a user-agent such as Googlebot, GPTBot, ClaudeBot, or PerplexityBot and enter a path beginning with `/`.
4. 4 Read the ALLOWED or BLOCKED result and the exact `Allow` or `Disallow` rule selected by longest-match precedence.
5. 5 Test one public page, one protected path, one exception path, and any CSS or JavaScript directory covered by broad rules.
6. 6 Publish the corrected file at the origin root, fetch the live version again, and repeat the same bot and path tests.