Build a robots.txt you can explain.
Create crawler groups, catch risky rules, test a path against longest-match precedence, and export a clean UTF-8 file—without sending your policy anywhere.
Robots.txt builder workspace
Start with a safe shape
Presets replace the current draft; every field stays editable.
Crawler groups
Each group starts with one user-agent product token and the rules that follow it.
Sitemaps
Optional absolute URLs. Sitemap lines apply to the file, not one crawler group.
Use a complete https:// or http:// URL, such as https://example.com/sitemap.xml.
Non-compliant bots can ignore it, and blocked URLs can sometimes still appear in search. Protect sensitive pages with authentication; use a supported
noindex method when indexing is the concern.Three checks prevent most expensive mistakes.
Keep the file at the top level of the exact host it controls and review any full-site block twice.
Put it at /robots.txt
Serve UTF-8 plain text from the root, such as https://example.com/robots.txt. A file in a subdirectory does not control the whole site.
Test important paths
Allow and Disallow compete by their most specific match. Use the tester here, then verify critical changes with the crawler you care about.
Do not hide secrets
The file is public and voluntary. Use authentication for private content. Read the Robots Exclusion Protocol and Google's interpretation.