scraping.robots
Pages and structure — robots. Give it url, respectRobots and it answers with found, url, agents, sitemaps, crawlDelay.
What it takes
scraping.robots takes url, respectRobots. The page to fetch. Required unless html is supplied.
What it gives back
The answer carries found, url, agents, sitemaps, crawlDelay. Each field is declared by the adapter rather than described here, so what this page says it returns and what the call actually returns cannot drift apart.
The same verb everywhere
Running this from the page, posting scraping.robots to /api/netinfo/run, and giving it to a coding agent over MCP are three doors onto one implementation. There is no separate website version that behaves differently from the documented one, which is the whole reason the verb has the same name in all three places.
- found (boolean) — false when the host serves no robots.txt — everything else is then empty, and crawling is allowed.
- url (string) — Where robots.txt was fetched from — present only on the found path.
- agents (object) — Per-user-agent allow/disallow rules.
- crawlDelay (number) — Seconds, when the file declares one.
Questions
Do you store what I look up?
A lookup is answered and not kept against you. A scan is kept — the account, the target, the time and the authorisation statement — because that record is what an abuse report is answered with, and keeping it is the condition of offering the feature at all.
Do you respect robots.txt when scraping?
Yes. The scraper identifies itself honestly, honours robots.txt, holds itself to a couple of requests a second per host, and will not touch anything behind a login. A scraper that ignores those is a scraper that gets our address blocked, which would end the product.