{"id":1850496,"recorded_at":"2026-11-02T00:08:16.818855Z","observed_at":"2026-11-02T00:08:16.818795Z","subject_kind":"endpoint","subject":"5ac4792b-f812-4ee5-8b5d-1a0471cd3473","kind":"mcp.tools","method":"probe.mcp/1","body":{"count":8,"tools":[{"description":"Does robots.txt allow fetching this exact URL? Fetches the live robots.txt and applies the longest-match rule for a user agent (default: * plus GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot). Returns allowed or not, the rule that decided it, and any crawl delay. A courtesy check, not a legal opinion.","inputSchema":{"properties":{"url":{"description":"The full URL you want to fetch","type":"string"},"user_agent":{"description":"optional, e.g. GPTBot","type":"string"}},"required":["url"],"type":"object"},"name":"can_fetch"},{"description":"Can I scrape this site, and how? Returns a verdict (open, guarded, hostile, blocked), a hostility score, a recommended approach with steps and risks, bot protections, robots.txt and AI-crawler policy, machine-readable surfaces (sitemap, llms.txt, JSON-LD), and eu_ai_training: the signals the EDPB draft Guidelines 03/2026 on web scraping for generative AI count against collecting personal data (robots.txt blocks on AI training crawlers, a Content-Signal of ai-train=no, ai.txt, CAPTCHAs, login walls), as none, some or strong. Signals, not legal advice. Uses a weekly-probed dataset of several hundred sites; for any other domain it probes live from a datacenter IP.","inputSchema":{"properties":{"domain":{"description":"A domain such as reddit.com","type":"string"}},"required":["domain"],"type":"object"},"name":"check_site"},{"description":"What should I build with this site, and how? Give a domain and, optionally, a goal in plain words (for example \"price alerts\", \"sales leads\", \"new jobs\", \"change log\", \"knowledge base\"). Returns the best-matching workflow for that site: what it does, its cadence, a trigger, collect, shape, store, act pipeline, the fields to keep, and step-by-step build recipes for n8n, Make, Zapier and Apify (or just one with platform). Workflows marked found_on_this_site come from what the weekly probe saw on the site itself. Also returns the other workflows, the golden data, and the guardrails.","inputSchema":{"properties":{"domain":{"description":"A domain such as amazon.com","type":"string"},"goal":{"description":"optional, what you want to achieve","type":"string"},"platform":{"description":"optional, only return this platform (n8n also returns n8n_workflow, an importable file):'s recipe","enum":["n8n","make","zapier","apify"],"type":"string"}},"required":["domain"],"type":"object"},"name":"get_playbook"},{"description":"List the sites in the dataset, optionally by category or verdict. Use it to find easy targets or to see what is covered.","inputSchema":{"properties":{"category":{"description":"e.g. shop, jobs, news, property, travel, gov","type":"string"},"limit":{"description":"default 50, max 200","type":"integer"},"verdict":{"enum":["open","guarded","hostile","blocked"],"type":"string"}},"type":"object"},"name":"list_sites"},{"description":"Find sites in the dataset by a word: a name, host, category (jobs, shop, news, gov), country or verdict. Use it to discover what is covered before calling check_site.","inputSchema":{"properties":{"limit":{"description":"default 25, max 100","type":"integer"},"query":{"type":"string"}},"required":["query"],"type":"object"},"name":"search_sites"},{"description":"Headline numbers across the dataset: verdict mix, how many sites block AI crawlers (and which), protection vendors, by category and country.","inputSchema":{"properties":{},"type":"object"},"name":"summary"},{"description":"Draw a page's anatomy as an Excalidraw diagram: its regions, the ranked ways to get its data, proof rows, the best route and what the page is made of. Returns a download link for the .excalidraw file (give it to the user; it opens at excalidraw.com or in Obsidian) and a file_name. With include_scene, also the scene itself (a few hundred KB).","inputSchema":{"properties":{"include_scene":{"description":"also return the full Excalidraw scene JSON (large). Default false: just the download link","type":"boolean"},"url":{"description":"Full page URL, e.g. https://example.com/products","type":"string"}},"required":["url"],"type":"object"},"name":"xray_diagram"},{"description":"How do I extract the data from THIS page? Fetches one public page and returns a ranked extraction plan: embedded JSON, JSON-LD, repeated records with CSS selectors and field names, tables, feeds, hidden API calls. Each source has a reliability, proof rows pulled from the real page, and the plan notes pagination and blockers. Set include_code for a starter Python script. Works on the raw HTML only, so JavaScript-built pages report a blocker instead of data. Use check_site first to learn whether the site allows scraping at all.","inputSchema":{"properties":{"include_code":{"description":"also return a starter Python script","type":"boolean"},"url":{"description":"Full page URL, e.g. https://example.com/products","type":"string"}},"required":["url"],"type":"object"},"name":"xray_page"}]},"body_sha256":"a6c34dccc6e5b6ca76724e7c90cd0e988cf80cb87857597a2c60a7561bc19a2a","prev_id":1009169,"leaf":"0xd04fa87f8da543118af8bbfffde07940987e6acbcc33ffc24f53403eef17a624","leaf_text":"{\"body\":\"a6c34dccc6e5b6ca76724e7c90cd0e988cf80cb87857597a2c60a7561bc19a2a\",\"id\":1850496,\"kind\":\"mcp.tools\",\"method\":\"probe.mcp/1\",\"observed_us\":1793578096818795,\"prev\":1009169,\"subject\":\"5ac4792b-f812-4ee5-8b5d-1a0471cd3473\",\"subject_kind\":\"endpoint\"}","day":{"day":"2026-11-02","leaves":444186,"root":"0x78fecaabbe319e56f5c4b80fc02e2052e198ad24781da8832a475ff65f2002d1","chained":"0xdc35957db5d1371ca382b9152878ffece28ec7c5cb4f98fd71ddb367f8b999b2","anchor_chain":"eip155:5042002","anchor_tx":"0x56b814612eb2e3e544153df3e91e1f7199106d1e430c7d8c6b653feeb9ebe9af","anchor_block":70432815},"proof":["0xd04fa0ed7d85acf2e4cf49d069f5b2873abdbf15e5d3b83800c71a97af529f64","0xf68d1c67dd2bd58fc3c7721a9a7acbafcc587b5d2253c7afde1294167b32ba3a","0x4277e59726b5219b1f68404056d005a2075865588c95de4a01738eefd6356927","0x78ff171024483bc1d53fa1827646f54f8ca60912e4edd2c1d7b2aaa07c19942b","0xd08705c84fef4c4c1d27c98e4512071c5b683c4d1d15ea592d79c4955c0de75b","0x94a482d84ca805b7dd66454a8d1d39b83b11acb83bb0eff78c79b9d329ffc7c0","0x3bb1a21919ccf0194fc7c75de2aff2d5bb118667cb2671e5a45d824298c27bd8","0x4952b84f43da402048402a7a8b8fa08bc27baf50d4996309818a0bf730bdf43d","0xa27eb11850f2699c8facb748b5e30d4888bdad78ecbb3029b3a59f63958af055","0x772a7d62e7f68f5fefc0c14f50afbed7127ff6e05fdec047ee9172692eb4a101","0xa3e59ea5049f0fb500b94c99db5bf2f100811da0dccd9ed6404f70897aa1e5fd","0x958b4a44f39e6f2ea70e91d6ebf20620245d2b968ca9c28f1b19f03aa4f5e55f","0xc68903bc934de8372c6b911a71856ad742aaac2d4b2de9ce90e84a085da4373d","0x0a363925e7ecbf308dcd4c5e94385984a43aafeb977e419f632e8dc2019e0553","0xa9134f4ed9cec34e13ba243b6a51db73753dbbb3b48c63bd82b22694813d2b02","0x7ddefa7d79e7640c3fed21ee4472689415182c41a0258ab04a81d60495f4c40a","0xdb8bd933619a267f9790faac9839143d202be46af284371cc369cdf683f897ba","0x98cbbf2c68ab4c0b4db782aeb88bea1f30b02178cf8b5d3d24a931d19492ea6d","0x252cce55f1a3d5391c16f82d7f62494429a5f69b03486d4cfcfb7b2c96d6c884"]}