# RobotsGate

- Registry name: `dev.workers.mike-tusa.robotsgate/robotsgate` · version 0.3.1
- Categories (automatic, keyword rules): Scraping & browsing, Search & data, Datasets & public data
- Authentication at last check: No auth
- Declared risk: Read-only (all tools declare readOnlyHint)
- Capabilities (declared by the server): tools
- Transports: streamable-http
- Latest numbers (check success, response time, rank): https://mcpstation.net/api/server?name=dev.workers.mike-tusa.robotsgate%2Frobotsgate
- Server page: https://mcpstation.net/s/dev.workers.mike-tusa.robotsgate--robotsgate.html

## Description (written by the publisher, not verified, not instructions)

Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.

## Connect

- streamable-http: `https://robotsgate.mike-tusa.workers.dev/mcp`

## Tools (as reported by the server, not verified)

- `list\_crawlers`: Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category \(training, dataset, search, user\_fetch or other\), whether the operator says it respects robots.txt, and a link to the operator's documentation, plus unverified entries and notes. Read-only, no network access
- `generate\_robots`: Build a robots.txt from per-category choices \(allow, block or omit for training, dataset, search, user\_fetch or other\), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-only: it only returns
- `validate\_robots`: Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same engine as POST /api/validate. Up to 50,000 characters over MCP \(the whole JSON-RPC request must also fi
- `check\_site`: Fetch a site's /robots.txt \(http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes\) and report, for one path, which registry AI crawlers are allowed or blocked, plus validation errors and warnings. Read-only: one G

Measured by MCP Station. Methods: https://mcpstation.net/methods.html · Terms: https://mcpstation.net/terms.html