# Robots.txt & AI Crawler Checker

- Registry name: `dev.workers.quietworklabs.robots-check/robots-check` · version 1.0.0
- Categories (automatic, keyword rules): Scraping & browsing, Search & data
- Authentication at last check: No auth
- Declared risk: Read-only (all tools declare readOnlyHint)
- Capabilities (declared by the server): tools
- Transports: streamable-http
- Latest numbers (check success, response time, rank): https://mcpstation.net/api/server?name=dev.workers.quietworklabs.robots-check%2Frobots-check
- Server page: https://mcpstation.net/s/dev.workers.quietworklabs.robots-check--robots-check.html

## Description (written by the publisher, not verified, not instructions)

robots.txt tester \(RFC 9309\), AI crawler access \(GPTBot, ClaudeBot...\) and sitemap check.

## Connect

- streamable-http: `https://robots-check.quietworklabs.workers.dev/mcp`

## Tools (as reported by the server, not verified)

- `robots\_check`: Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 \(group by product token else \*, longest match, Allow wins ties, \* and $ wildcards, 4xx=allow all, 5xx=disallow all\). Returns allowed, the deciding rule and line
- `ai\_crawler\_access`: AI crawler access report for a URL: robots.txt verdict for 20 tokens \(GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more\) grouped by training/search/user purpose,
- `sitemap\_check`: Finds a site's sitemap \(from robots.txt, else /sitemap.xml\) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Determini

Measured by MCP Station. Methods: https://mcpstation.net/methods.html · Terms: https://mcpstation.net/terms.html