Robots.txt Generator
Create a clean, Google-aligned robots.txt file for your website with crawler rules, sitemap URLs, Allow/Disallow directives and a built-in local robots.txt tester.
1 Quick Presets
Start with a common configuration, then customize the rules for your website.
2 Website & Sitemap
Enter your website and sitemap information. The robots.txt file belongs at the root of the host it controls, such as /robots.txt.
3 Default Crawler Rules
These rules control crawlers that match the selected User-agent.
4 Common SEO & WordPress Rules
Select common areas that you may want to keep out of crawler requests. Review every rule before publishing.
5 Advanced Crawler-Specific Rules
Add special rules for specific crawlers. Leave this section empty if you only need a general User-agent rule.
6 Additional Options
Generated robots.txt
Review the generated file carefully before placing it at your site's root URL: https://yourdomain.com/robots.txt
7 Robots.txt Tester
Test a URL path against the generated rules locally. This tester does not crawl your website and does not send your URL anywhere.
What Is a Robots.txt File?
A robots.txt file is a plain-text file placed at the root of a website. It provides instructions to automated crawlers about which URLs or directories they may or may not crawl.
For example, the robots.txt file for: https://example.com/ should normally be available at: https://example.com/robots.txt.
Basic Robots.txt Example
Robots.txt Best Practices
- Keep the file as a valid UTF-8 plain-text file.
- Place robots.txt at the root of the host it controls.
- Use specific paths instead of blocking the entire website unless that is intentional.
- Keep important CSS, JavaScript and image resources crawlable when Google needs them for rendering.
- Use absolute URLs for Sitemap directives.
- Review Allow and Disallow conflicts carefully.
- Remember that robots.txt is not a security mechanism.
- Do not assume Disallow means a URL will disappear from Google Search.
Robots.txt vs Noindex
Robots.txt primarily controls crawling. It is not a guaranteed URL-removal mechanism. If you need search-engine indexing controls for a page, use appropriate page-level robots directives such as noindex where applicable.
Robots.txt and Google AI Search
Google's current guidance does not require a special robots.txt format, special AI file, or special markup to appear in Google Search's AI features. Normal SEO fundamentals continue to matter for Google's generative search experiences.
Do not add invented directives such as AI-crawl, GEO-allow, or similar unsupported fields. Unsupported robots.txt fields should not be treated as Google Search controls.
Understanding the 500 KiB Limit
Google currently enforces a robots.txt size limit of 500 KiB. Content after that maximum is ignored. This generator checks the generated file size so you can catch oversized files before publishing them.
Frequently Asked Questions
Upload it to the root of the host it controls. For example, https://example.com/robots.txt. The rules apply to the protocol, host and port where the file is hosted.
No. Disallow primarily controls crawling. A URL can potentially remain known to search engines even when its content cannot be crawled.
Google does not support the Crawl-delay directive. Some other crawlers may support it, which is why this generator labels it as crawler-dependent.
Google uses the most specific matching rule based on the matching path length. When conflicting rules have the same specificity, Google uses the least restrictive rule.
Google's current documentation says there is no need to create llms.txt or other special AI files for Google Search's AI features.
Google currently enforces a maximum robots.txt size of 500 KiB. Content beyond that limit is ignored.