Loading content...
Generate a clean robots.txt with allow/disallow rules, crawl-delay, sitemap references, and multi-user-agent support. Copy or download.
User-agent: * Allow: /public Disallow: /admin Disallow: /private
Note: robots.txt is a courtesy, not a wall — sensitive data must be protected with auth, not just disallowed.
How Google actually reads robots.txt and the sitemap mistakes that cost rankings.
No spam. Unsubscribe anytime.
Discover more utility-driven tools designed to enhance your workflow and technical excellence.
Generate .gitignore files from 30+ templates — Node, Python, Go, Rust, Java, macOS, Windows, IDEs, and more. Copy or download instantly.
Generate production-ready Dockerfiles and .dockerignore for Node, Python, Go, Rust, Java, and Next.js — multi-stage, distroless-aware.
Searchable reference for every HTTP status code — 1xx to 5xx — with meanings, typical use cases, and developer-friendly explanations.