Robots.txt Generator
Generate a robots.txt file to control how search engines crawl your website.
robots.txt
User-agent: * Allow: / Disallow: /admin/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml
Key Features
- Generate a valid robots.txt file with allow and disallow rules for any user-agent.
- Add your XML sitemap URL directly to the file for crawler discoverability.
- Block specific bots — including AI training crawlers — with targeted user-agent rules.
- Preview the final file and copy it with one click.
- Supports crawl-delay directives for controlling server load from bots.
- Validates syntax to prevent rules that silently fail in production.
Frequently Asked Questions
What is a robots.txt file?
robots.txt is a plain-text file placed at the root of your domain that instructs search engine crawlers which pages or directories they are allowed or disallowed from accessing.
Does blocking a page in robots.txt remove it from Google?
Not necessarily. Blocking crawling prevents Google from reading the page content, but the URL can still appear in search results if other sites link to it. Use a noindex meta tag to fully remove a page.
Where do I put my robots.txt file?
It must be placed at the root of your domain — for example, https://example.com/robots.txt. A robots.txt in a subdirectory is not recognised by most crawlers.
Related Tools
Meta Tag Generator
Generate SEO meta tags, Open Graph and Twitter cards for your site
Open Graph Preview
Preview how your site looks when shared on social media
Google SERP Preview
Preview how your page appears in Google search results
UTM Link Builder
Create UTM-tagged URLs for tracking marketing campaigns