1. Home
  3. SEO Tools
  5. Robots.txt Generator

Robots.txt Generator

Create a custom robots.txt file with user-agent, allow, disallow, and sitemap directives for search engine crawlers.

This generator builds a valid robots.txt file for your site; copy it to your web root for crawler guidance.

Usage

  • Set a user-agent, allow public paths, and block private directories.
  • Add a sitemap URL to help crawlers find all indexed pages.
  • Copy the generated text to your site root as `robots.txt`.
FAQ

Frequently asked questions

What's the difference between allow and disallow directives?

Allow: permits crawlers to access specific pages or directories. Disallow: prevents crawlers from accessing pages. Use disallow for sensitive pages, duplicate content, or crawl-heavy areas. Use allow to override disallow rules.

How do I tell Google bots not to crawl certain pages?

Add "Disallow: /directory/" to robots.txt for that path. For individual pages, use "meta robots nofollow" in the HTML head. Disallow is faster for entire sections; meta tags are better for specific sensitive pages.

Where should I upload the robots.txt file on my server?

Upload robots.txt to your website root directory so it's accessible at yourdomain.com/robots.txt. All search engines check this location first. Ensure file permissions allow public read access.

Can robots.txt prevent indexing of sensitive pages?

Disallow blocks crawling but not indexing if pages are linked externally. For true prevention, combine robots.txt with "noindex" meta tags. For maximum security with sensitive data, don't link the pages at all.