Scraping

Fetch a public page and pull the structure back out of it. 11 of the 20 verbs this service exposes are in here, and each has its own page with the arguments it takes and what it gives back.

What is in Scraping

scraping.fetchHtml, scraping.headers, scraping.extractLinks, scraping.pageMetadata, scraping.extractText, scraping.extractTables, scraping.extractContacts, scraping.extractJsonLd, scraping.discoverFeeds, scraping.sitemap, scraping.robots. Which tab a verb appears under follows the question somebody arrives with; which pack it lives in follows where the code is. The two are allowed to disagree, and here they do.

Generated, so it cannot go stale

This tab is written from the adapters that actually answer these calls. A verb added to a pack appears here on the next build and a verb removed disappears from it, so the list and the engine cannot disagree. The build refuses outright to ship a tab with no verbs behind it, because a tab that opens onto an empty state is worse than a tab that is not there.

Questions

Do you store what I look up?

A lookup is answered and not kept against you. A scan is kept — the account, the target, the time and the authorisation statement — because that record is what an abuse report is answered with, and keeping it is the condition of offering the feature at all.

Do you respect robots.txt when scraping?

Yes. The scraper identifies itself honestly, honours robots.txt, holds itself to a couple of requests a second per host, and will not touch anything behind a login. A scraper that ignores those is a scraper that gets our address blocked, which would end the product.