All Tools

Robots.txt Generator

Create a simple robots.txt file for crawler access rules and an optional sitemap location. Use it to configure crawler access accurately.

Use * for all crawlers or a specific bot like Googlebot.

One relative path per line.

One blocked path per line.

Optional absolute URL to your public XML sitemap.

Best Practices

  • Use User-agent: * unless targeting a specific bot.
  • Allow public indexing routes like blog and root domain.
  • Disallow private/admin endpoints to conserve crawl budget.
  • Add your canonical sitemap URL at the bottom.

Example Robots.txt

User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Sitemap: https://arjns.com/sitemap.xml

Safe Patterns

Keep public content open. Disallow low-value paths like internal search results, temporary drafts, and admin dashboards.

Limitations

Robots.txt does not hide sensitive pages. Use HTTP auth and noindex response headers for security.

Before Launch

Confirm the URL is accessible directly at /robots.txt and that the sitemap link responds with HTTP 200.

Related guides

Technical SEO checklist for new websitesA practical launch checklist for crawlability, metadata, URLs, internal links, and indexing.