{"id":567003,"recorded_at":"2026-10-27T22:03:04.246552Z","observed_at":"2026-10-27T22:03:04.246393Z","subject_kind":"endpoint","subject":"b6510ef5-2f81-4b02-abbd-ea0dfbd14084","kind":"mcp.tools","method":"probe.mcp/1","body":{"count":3,"tools":[{"description":"AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, plus page meta robots, X-Robots-Tag (noindex/noai) and llms.txt presence. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.","inputSchema":{"properties":{"url":{"description":"Site or page URL","examples":["https://www.theguardian.com/"],"type":"string"}},"required":["url"],"type":"object"},"name":"ai_crawler_access"},{"description":"Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line, crawl-delay and sitemaps. Deterministic, no AI. Price: $0.001 USDC per call via x402 (Base); the first 5 calls per day per client are free.","inputSchema":{"properties":{"url":{"description":"Page URL to test","examples":["https://www.nytimes.com/section/technology"],"type":"string"},"user_agent":{"description":"Crawler product token, e.g. GPTBot or MyAgent. Default *","examples":["GPTBot"],"type":"string"}},"required":["url"],"type":"object"},"name":"robots_check"},{"description":"Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.","inputSchema":{"properties":{"url":{"description":"Site URL (discovers the sitemap) or a sitemap URL","examples":["https://www.python.org/"],"type":"string"}},"required":["url"],"type":"object"},"name":"sitemap_check"}]},"body_sha256":"d8b38efa0bd6914b555c50bfc83b3ed8f434833c30eb2b78b6444b3127e41634","prev_id":null,"leaf":"0x19301fbd6c086b6a3acb7570da07c200a5eaea4b66a14a246a7cfda73523d844","leaf_text":"{\"body\":\"d8b38efa0bd6914b555c50bfc83b3ed8f434833c30eb2b78b6444b3127e41634\",\"id\":567003,\"kind\":\"mcp.tools\",\"method\":\"probe.mcp/1\",\"observed_us\":1793138584246393,\"prev\":null,\"subject\":\"b6510ef5-2f81-4b02-abbd-ea0dfbd14084\",\"subject_kind\":\"endpoint\"}","day":{"day":"2026-10-27","leaves":273461,"root":"0xdd77e5e99da32c3f154180d63e3c3e6fe11e896a2c6896412ef6b5c4bcf7e795","chained":"0x23bb4dfd09f55c15ac3962f53578e02c6114d5bfd9bb451cf8794fb6e2817ef4","anchor_chain":"eip155:5042002","anchor_tx":"0xb2ebd83aafeb765a0ce66e76aefb133ed06606c98038fd56191e2bcd1dcb77b4","anchor_block":69414147},"proof":["0x193009e813c2737c8d9b521b549f70a3f2fb054f70a3620ca39954a21f66d79c","0x78750c224e8cf7bf425e8a547b6cbd3615a653838cc2e3d2512f15afc44f03b0","0xc4ee44bc5b7f52cd939215ac5dfc84647184d589a8111aa4258c34062469442c","0xfa727ce0682032ec3f80e32c1739689fe6b44c16f5562c3ec3dd8bfbe7d6640c","0x579dcf1ef2c826af28832b0e25d65fea72dcf08115b1477e2c5640cf4b208895","0x18f591d440f5925a8de0baf37a83cec3ecd6a4b30ece75c25a915abc2d7b4317","0x31583c7bc3ed0e27f7a723c25adb3beb5e8d82c6417e01eb632f5ca75c45c6d1","0xfde76c7df9daf6cd7ba732ca13ca943f0e0915a1ab218367f6cce28084dfa97f","0xa1df58585e78d9acfef007ddf18bcfc263e9621ecaba176698a7741078d6001f","0x92ad4d25cf40a90a064b6296c16b622df60247955858f9880c77f181288069c4","0xee7752a111eb14027f818d33fc8806527d41f25841a61d0dfd8ec487e0c8660b","0x864874d7fe156494acebcd974fb0ea64b6d7d6ef442b90a07ce8adbc57f7d75b","0xca0b37bb3b043ccf0f24c50fccd9c2f22b7e5290df49886008fc2ff55380fdd1","0x2064e3a2b01d0d12d5079bf720a0d0125e9703c55e92de54ad40c302469ec047","0x6913cb48392acb8dc5c940dcdb6e6fc889d544f85dff11da249551a5f027e688","0x518b93854db03dc9fc938883abf30da6eaae5ae0d4336903968fd32a7df8c71b","0x22c1fb9487ee0b9732fa017a0cc321eb27a60c08450d112f755089796e12cb1c","0xa20d48798a34aba12eb9c7297722982d910f06c148f5e27ac59307422964a585","0x9fc354c0ea3efc3f1b6548bb77ea902a50f2c9647647ea61592efce2bd795699"]}