All servers · Scraping & browsing

Robots.txt & AI Crawler Checker

dev.workers.quietworklabs.robots-check/robots-check · v1.0.0

New (7 days)

Measured by MCP Station

Last check	Reachable
Checked at	—
Check success 7 / 30 days	—
First-try success 30 days	—
Median response	—
Lookups 30 days	—
Authentication	No auth
MCP version	2025-06-18
Rank score	—
Declared risk	Read-only (all tools declare readOnlyHint)
Capabilities declared	tools
Tool definitions	3 tools · with description 100% · input schema is an object 100% · read/destructive hints 100%

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

streamable-http · https://robots-check.quietworklabs.workers.dev/mcp

claude mcp add --transport http robots-check https://robots-check.quietworklabs.workers.dev/mcp

{
  "mcpServers": {
    "robots-check": {
      "url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}

robots_check
Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line
ai_crawler_access
AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose,
sitemap_check
Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Determini

Website · JSON · Text for agents · Registry updated 2026-10-18