Tools

Robots.txt Generator

Create a simple robots.txt file for crawler access rules and an optional sitemap location. Use it to describe crawl preferences, then validate the file before adding it to the root of your site.

Use * for all crawlers or a crawler token such as Googlebot.

Add one allowed path per line.

Add one blocked path per line.

Optional, but useful when a site has a public sitemap.

Use this tool when

You need a clean first draft for crawler access rules and a sitemap reference. Build the rules in this order:

  • Use User-agent: * unless a rule must target one crawler.
  • Allow the public routes that should be crawlable, especially the homepage, blog, and tools.
  • Disallow utility routes that should not waste crawler time.
  • Add the canonical sitemap URL after the access rules.
  • Publish the file at /robots.txt on the canonical domain.

Copy/share example

User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Sitemap: https://arjns.com/sitemap.xml

Safe patterns

Keep public pages open by default. Block routes that generate duplicate, private-looking, or low-value crawl paths, such as internal APIs, temporary previews, and utility folders.

What it cannot do

Robots.txt does not hide sensitive content, remove indexed URLs, or guarantee every crawler will obey the file. Use authentication, noindex directives, and server-side access controls for those jobs.

Before publishing

  • Check capitalization and path spelling.
  • Confirm the sitemap URL returns a live sitemap.
  • Make sure scheduled pages are excluded until their live date.
  • Revisit the file after adding new route groups.

Related guides

Technical SEO checklist for new websitesA practical launch checklist for crawlability, metadata, URLs, internal links, and indexing.