{"id":1009169,"recorded_at":"2026-10-29T09:15:41.369912Z","observed_at":"2026-10-29T09:15:41.369863Z","subject_kind":"endpoint","subject":"5ac4792b-f812-4ee5-8b5d-1a0471cd3473","kind":"mcp.tools","method":"probe.mcp/1","body":{"count":8,"tools":[{"description":"Does robots.txt allow fetching this exact URL? Fetches the live robots.txt and applies the longest-match rule for a user agent (default: * plus GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot). Returns allowed or not, the rule that decided it, and any crawl delay. A courtesy check, not a legal opinion.","inputSchema":{"properties":{"url":{"description":"The full URL you want to fetch","type":"string"},"user_agent":{"description":"optional, e.g. GPTBot","type":"string"}},"required":["url"],"type":"object"},"name":"can_fetch"},{"description":"Can I scrape this site, and how? Returns a verdict (open, guarded, hostile, blocked), a hostility score, a recommended approach with steps and risks, bot protections, robots.txt and AI-crawler policy, machine-readable surfaces (sitemap, llms.txt, JSON-LD), and eu_ai_training: the signals the EDPB draft Guidelines 03/2026 on web scraping for generative AI count against collecting personal data (robots.txt, ai.txt, CAPTCHAs, login walls), as none, some or strong. Signals, not legal advice. Uses a weekly-probed dataset of several hundred sites; for any other domain it probes live from a datacenter IP.","inputSchema":{"properties":{"domain":{"description":"A domain such as reddit.com","type":"string"}},"required":["domain"],"type":"object"},"name":"check_site"},{"description":"What should I build with this site, and how? Give a domain and, optionally, a goal in plain words (for example \"price alerts\", \"sales leads\", \"new jobs\", \"change log\", \"knowledge base\"). Returns the best-matching workflow for that site: what it does, its cadence, a trigger, collect, shape, store, act pipeline, the fields to keep, and step-by-step build recipes for n8n, Make, Zapier and Apify (or just one with platform). Workflows marked found_on_this_site come from what the weekly probe saw on the site itself. Also returns the other workflows, the golden data, and the guardrails.","inputSchema":{"properties":{"domain":{"description":"A domain such as amazon.com","type":"string"},"goal":{"description":"optional, what you want to achieve","type":"string"},"platform":{"description":"optional, only return this platform (n8n also returns n8n_workflow, an importable file):'s recipe","enum":["n8n","make","zapier","apify"],"type":"string"}},"required":["domain"],"type":"object"},"name":"get_playbook"},{"description":"List the sites in the dataset, optionally by category or verdict. Use it to find easy targets or to see what is covered.","inputSchema":{"properties":{"category":{"description":"e.g. shop, jobs, news, property, travel, gov","type":"string"},"limit":{"description":"default 50, max 200","type":"integer"},"verdict":{"enum":["open","guarded","hostile","blocked"],"type":"string"}},"type":"object"},"name":"list_sites"},{"description":"Find sites in the dataset by a word: a name, host, category (jobs, shop, news, gov), country or verdict. Use it to discover what is covered before calling check_site.","inputSchema":{"properties":{"limit":{"description":"default 25, max 100","type":"integer"},"query":{"type":"string"}},"required":["query"],"type":"object"},"name":"search_sites"},{"description":"Headline numbers across the dataset: verdict mix, how many sites block AI crawlers (and which), protection vendors, by category and country.","inputSchema":{"properties":{},"type":"object"},"name":"summary"},{"description":"Draw a page's anatomy as an Excalidraw diagram: its regions, the ranked ways to get its data, proof rows, the best route and what the page is made of. Returns a download link for the .excalidraw file (give it to the user; it opens at excalidraw.com or in Obsidian) and a file_name. With include_scene, also the scene itself (a few hundred KB).","inputSchema":{"properties":{"include_scene":{"description":"also return the full Excalidraw scene JSON (large). Default false: just the download link","type":"boolean"},"url":{"description":"Full page URL, e.g. https://example.com/products","type":"string"}},"required":["url"],"type":"object"},"name":"xray_diagram"},{"description":"How do I extract the data from THIS page? Fetches one public page and returns a ranked extraction plan: embedded JSON, JSON-LD, repeated records with CSS selectors and field names, tables, feeds, hidden API calls. Each source has a reliability, proof rows pulled from the real page, and the plan notes pagination and blockers. Set include_code for a starter Python script. Works on the raw HTML only, so JavaScript-built pages report a blocker instead of data. Use check_site first to learn whether the site allows scraping at all.","inputSchema":{"properties":{"include_code":{"description":"also return a starter Python script","type":"boolean"},"url":{"description":"Full page URL, e.g. https://example.com/products","type":"string"}},"required":["url"],"type":"object"},"name":"xray_page"}]},"body_sha256":"4c1d787d651d1c3e882bcafd0960ce7ff528286020bf7d30802211127342e9ee","prev_id":null,"leaf":"0x781a29536df6709914137ebf528ba081c0499726ad471129256d87c047ca9040","leaf_text":"{\"body\":\"4c1d787d651d1c3e882bcafd0960ce7ff528286020bf7d30802211127342e9ee\",\"id\":1009169,\"kind\":\"mcp.tools\",\"method\":\"probe.mcp/1\",\"observed_us\":1793265341369863,\"prev\":null,\"subject\":\"5ac4792b-f812-4ee5-8b5d-1a0471cd3473\",\"subject_kind\":\"endpoint\"}","day":{"day":"2026-10-29","leaves":242798,"root":"0xc050555c6b023f2aefb6536efb9f0760b92c30a0d06c572b7daf63db259ea17c","chained":"0x4cde8bae2042512e1bec11650628f0293556311cc4ea74d99ffddf02b671a7d6","anchor_chain":"eip155:5042002","anchor_tx":"0x9e85c340732f416e431b111fae72f7624870125f7f379af2732eb6c2ba8cf6a1","anchor_block":69753317},"proof":["0x7819f3f2a306e918983305f5f2c515266ef5b0c5cc8a2fa8e2864e37831b72c4","0x524591f5ca7eb9eeb4de564e9461e8bfa55f7611843beb0c1802bd193508419c","0xbc34e056ceb2b00f7834b6eaf5ae4f5d87aee9584148d47c749568d8f29a75cc","0xc71e55a7d82a477aaf8db327abb3e43a8968b2c52ab8acf65b331f303a0d118e","0x82048844eaf329539381c8e4c92cad0138b75235556d74ba30812c5019857d8a","0xf960157b8f10958878742085fee6c2b69d1af83dd1ea74216641e254fbe91d7e","0xcd6cafb2417ce86d5a1b55258fe62f1dacd93a5d701a0b860c4feee2eaabfceb","0x4094af0e38aef69f41f9ef6ca424709da2270550f513e28e67f1350f254f90f9","0x5884b16a3eec301617b07990c42bb4a093e30ecd46fa7784cedfb74fd5613db1","0x07e0c84d917943facd277b591b19a4104398984df43cbbe2fa056e0a04c8067e","0xeb4c89f10ecf7036c1a5612800e96a9fdbf56f9ae92f4d338413d367ff69ebbf","0x63a11cd9e65d16993c0e702ce6d7bda551e5a72e62115aaf790a2e3519c7fac6","0xc0f950bf9c872c684cf701ee03b280e31f9fd7a44bed4df783964fd0df78d0c2","0xa07c03f1ff4a33c8a0105ab761d762ba211920a4e09af3e5ae84b51a3c4cf51b","0x60b5befbd81f025904c8a3936b7ba86e3245ad5b37c615d01d42aa3227e3ed2e","0xe2d603cd6f987214550319ad4fa39952b6098bc5c1b20893b6b80e8d82a79934","0x91f12a2db1e2c1be07cbe4de3981a37a33051a36d3c3eee5a260804add6d1d75","0xd56e61af23709ed9f460471e3c08400c0142ee83e1fd9057e46350d12edf9c07"]}