Crawler rule simulator

Robots.txt Tester

Validate a proposed robots.txt file and see exactly whether each URL path is allowed or blocked for a specific crawler—including the group and source line that decides it.

Private by design: the tester does not transmit your rules, files, or paths, and it does not fetch remote robots.txt files. The site shell’s standard analytics still runs on public pages.

Add your robots.txt

Paste UTF-8 text or upload a local .txt file. Maximum: 500 KiB.
0 bytes

Choose crawler and paths

Enter one exact product token using letters, hyphens, or underscores. Matching is case-insensitive.
One per line; up to 100. Full HTTP(S) URLs are reduced to path + query. Fragments are ignored.

How to test robots.txt rules safely

Use the proposed file, not just the live one

  1. Paste the exact robots.txt text you plan to publish, or load it from a local text file.
  2. Choose the crawler and add representative public, private, asset, pagination, and query-string URLs.
  3. Review the deciding source line and every validation finding before uploading the file to the origin root.

How the verdict is chosen

The tester matches one exact user-agent product token, without treating Googlebot as Googlebot-News, and combines repeated groups for that token. If none exists, repeated * groups are the fallback. Within the selected groups, the longest matching pattern wins; * spans characters, a final $ anchors the end, and Allow wins a length tie.

Interpret results correctly

“Blocked” means a compliant crawler is asked not to fetch that path. It does not guarantee removal from search results, and a blocked crawler may be unable to see a page-level noindex. “Allowed” means no selected rule blocks the path; it does not guarantee crawling or indexing.

Important boundaries

  • robots.txt is public guidance, not authentication or access control.
  • Rules belong at /robots.txt and apply only to the same protocol, host, and port.
  • Crawler extensions such as crawl-delay vary and are reported but do not alter access verdicts here.
  • For responsive local processing, one run is limited to 100 paths, 5,000 directives, 16 KiB per path rule, 250 stored findings (200 rendered), and five million matcher steps. The report identifies any limit reached.

Standards note: Matching follows the core Robots Exclusion Protocol in RFC 9309 plus the common * and $ pattern behavior documented by Google. This tool evaluates pasted text only; confirm the published file’s HTTP status and content separately.