{"id":124035,"recorded_at":"2026-10-26T14:40:24.029938Z","observed_at":"2026-10-26T14:40:24.029899Z","subject_kind":"source_record","subject":"x402_bazaar:https://web.openverbs.com/v1/robots","kind":"listing","method":"baseline/1","body":{"endpoints":[["x402","https://web.openverbs.com/v1/robots"]],"name":"Check robots.txt crawl rules","owner_address":"0xb3f4de103f34b99f0da19b4f16d6b53a14c2194a","raw":{"accepts":[{"amount":"6000","asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","extra":{"name":"USD Coin","version":"2"},"maxTimeoutSeconds":300,"network":"eip155:8453","payTo":"0xb3f4de103F34b99f0Da19B4F16D6B53a14C2194A","scheme":"exact"}],"description":"Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards.","extensions":{"bazaar":{"info":{"input":{"body":{"paths":["example"],"url":"https://example.com","userAgent":"example"},"bodyType":"json","method":"POST","type":"http"}},"schema":{"$schema":"https://json-schema.org/draft/2020-12/schema","properties":{"input":{"additionalProperties":false,"properties":{"body":{"additionalProperties":false,"properties":{"paths":{"description":"Optional extra paths (or same-origin URLs) to check in the same call, e.g. [\"/admin\", \"/search?q=x\"]. Up to 50.","items":{"maxLength":2048,"minLength":1,"type":"string"},"maxItems":50,"type":"array"},"url":{"description":"Public http(s) URL to check. Its origin's /robots.txt is fetched and its path is the first path evaluated.","format":"uri","maxLength":2048,"type":"string"},"userAgent":{"description":"User-agent token to evaluate rules for, e.g. \"Googlebot\". Defaults to \"*\".","maxLength":200,"minLength":1,"type":"string"}},"required":["url"],"type":"object"},"bodyType":{"enum":["json","form-data","text"],"type":"string"},"method":{"enum":["POST"],"type":"string"},"type":{"const":"http","type":"string"}},"required":["type","method","bodyType","body"],"type":"object"}},"required":["input"],"type":"object"}}},"lastUpdated":"2026-10-25T21:58:55.756Z","quality":{"l30DaysTotalCalls":7,"l30DaysUniquePayers":6,"lastCalledAt":"2026-10-25T21:58:55.5Z"},"resource":"https://web.openverbs.com/v1/robots","serviceName":"Check robots.txt crawl rules","tags":["robots.txt parser","robots txt checker","is url crawlable","crawl permission api","user-agent allow disallow"],"type":"http","x402Version":2},"source":"x402_bazaar","summary":"Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards."},"body_sha256":"4869a5db2bc19e3c0cd57cf62cb689ee3de854e299f31658acbdda73ad45ebee","prev_id":null,"leaf":"0x7831eaff5d01ff43b1d5cbd09d6ce5241116a86cd694a268e67dfee7c875cf25","leaf_text":"{\"body\":\"4869a5db2bc19e3c0cd57cf62cb689ee3de854e299f31658acbdda73ad45ebee\",\"id\":124035,\"kind\":\"listing\",\"method\":\"baseline/1\",\"observed_us\":1793025624029899,\"prev\":null,\"subject\":\"x402_bazaar:https://web.openverbs.com/v1/robots\",\"subject_kind\":\"source_record\"}","day":{"day":"2026-10-26","leaves":296695,"root":"0xf43e8dcc3d451a4788b26be904c55e8c4c0fa75c9666796c51db38d37e2c0df8","chained":"0x702d5ac508f08b0d1accf36af25822d8e5660367494edd17a642fb61e158a57a","anchor_chain":"eip155:5042002","anchor_tx":"0x495162f8936e19f0039918feebe6c307f327cd6bf8c37219620b1970041fa9ae","anchor_block":69251483},"proof":["0x78321de3cd9deab5623b3ba66d706f232199f462e3b0d1e0734ced98b557fc83","0x866aba0d5d676c209967ca788448e24ede2efd723d4fc6a97522fc22aa6b2361","0xb0b7ef3c63c0071d4bf02a564b3425b38e5277c951bb7f5581385ceec77fb23e","0x42ad0eabb099386bffc1383fb706bad5d69860f20047e84138f689fee9f8de50","0xfc7ed8551b05e8af79dd8e446eb5935ab1e9d072cffa2c1f9305ff7bf29aab05","0x04d08554db9df25c02dfd5d2b5bb26700c27d4f59cc074949d974960439a0ab7","0x18f7d63543ccc6f36b6ed2c6ab425ff9f12d9e8153261a7e10c042c416044316","0x79074def72dae8c5e7b2f05bfb5c8994e602649db5e77fdd9fe314515455c5f1","0xe7a567a06402dbc17c02e19cb76a342a6e2e155568ea808558236b065d1478f7","0x7289d13e9c5129911e8dd0a94c4ccdb3088dca627d8df693e7d7c3b58bbbd576","0x8fe7ebf89d10ac2e699de3f80f5dda3e72eef4ae079794ab7d1035929ad723cc","0x8fdfac9874f184de58b63efbdab0fc27674958d741146d02f58b04d3c559cabf","0xdef3cab0b4070d8ad7f4f91288da4881d19789f131b1b353621df79df04a0617","0xc4fe2177880ea1b0e82b6fe4c270090fd0fbe5af3a42a8cbf6a6ba2fa35625f6","0x15039087bf60ae9b06b756ec3cbc299760e59923a438bbd91e34d105186db0ed","0x2b6e23e3624a1b66e1d2298cfbce99a93ce6b13d24a58112a71802d79c5f7a76","0xd21942ae3214eff88e08ab04884fd4b11958b51af9a0741b7668e2d07a2c56c9","0x9a9024edb5069d88a324facb310a5ce8e495214bbaa968f077bede437390194f","0x9b9ce5b74b9bab5121612ef6b64920a8d453cbf311096a423eef9702747e868a"]}