Robots.txt Tester
Free tool · Crawl rules check
Robots.txt tester
Test your website's robots.txt rules to see whether specific URLs are allowed or blocked for search engine crawlers. Check how Allow and Disallow directives apply to a chosen user agent and identify the rule responsible.
What This Robots.txt Tester Checks
- Allow and Disallow rules: Shows which rule allows or blocks the URL you test.
- User-agent rules: Tests your robots.txt against the exact user agent you type in, such as Googlebot or GPTBot.
- Wildcard and anchor rules: Supports '*' wildcards and a trailing '$' anchor in Disallow and Allow paths.
- URL access: Checks whether the specific path you enter is allowed or blocked. Leave it blank to test the homepage.
- Rule matching: Uses the longest matching pattern to decide the result, with ties going to Allow, and shows every rule group along with which one matches your user agent.
What This Robots.txt Tester Does Not Check
- Does not guarantee that a crawler will actually visit your website
- Does not verify whether a crawler's identity is genuine
- Does not check whether a page is indexed by Google
- Does not check meta robots or X-Robots-Tag directives
- Does not check whether your page can be rendered correctly
Why Test Your Robots.txt File?
An incorrect robots.txt rule can prevent search engine crawlers from accessing pages you want indexed.
Use the tester to:
- Check important URLs before and after robots.txt changes
- Find rules that unintentionally block pages
- Test specific user agents
- Verify Allow and Disallow directives
- Troubleshoot crawling issues
How to Test Robots.txt Rules
1. Enter Your Website
Enter your website's domain. The tool reads its public robots.txt file.
2. Enter a User Agent
Type the crawler you want to test, such as Googlebot, Bingbot, or GPTBot. It defaults to Googlebot.
3. Enter the URL Path
Add the specific path you want to check, like /some/page. Leave it blank to test the homepage.
4. Run the Test
The tool evaluates the robots.txt rules against the path and user agent using longest-match-wins, with ties going to Allow.
5. Review the Result
See whether the URL is allowed or blocked, which rule decided it, and every rule group with the one matching your user agent marked.
Robots.txt vs. Meta Robots
robots.txt controls crawler access to URLs, while meta robots directives provide instructions about how a page should be handled after it has been accessed.
For example, a Disallow rule in robots.txt can prevent a crawler from accessing a URL, while a noindex directive can tell a crawler not to include an accessible page in search results.
Frequently Asked Questions
What is a Robots.txt Tester?
A Robots.txt Tester checks your robots.txt rules to determine whether a specific URL is allowed or blocked for a chosen user agent.
How do I test my robots.txt file?
Enter your website, type the user agent you want to test, add the URL path, and run the check. The tool shows whether the URL is allowed or blocked and which rule decided it.
What does Disallow mean in robots.txt?
Disallow tells a compliant crawler not to access the specified path.
What does Allow mean in robots.txt?
Allow can specify paths that a crawler is permitted to access, including cases where a more specific rule needs to override a broader restriction.
Does robots.txt prevent a page from being indexed?
Not necessarily. Robots.txt controls crawling, not indexing. A URL can potentially remain known to search engines even when crawling is restricted.
Can I test any user agent, or only ones from a list?
Any user agent. Type in the exact crawler name you want to test; the tool is not limited to a fixed list.