Crawler's Eye - AI readability for any site

Open any page and see what an AI assistant actually gets. Most tools read a site's robots.txt file and tell you what it says. Crawler's Eye also checks whether the site honours it, by asking for the page as a crawler and comparing the answer. In our own audit of more than twenty thousand online stores, 250 of them allowed crawlers in robots.txt and blocked them anyway. Every tool that only reads robots.txt told those owners they were fine. WHAT IT CHECKS For the site: which AI crawlers the robots.txt rules allow, which rule is blocking the rest, and whether the server actually serves a crawler or answers with a challenge page. For the page: what a crawler receives, compared with what your browser built after JavaScript ran. Is there readable text without JavaScript, how much of the visible text was actually sent, and are links and headings present in the served HTML. A page can look complete to a person and arrive close to empty. For a product page: whether the price on screen also exists in machine-readable markup, including the case that usually goes unnoticed, where the markup price disagrees with the displayed one. And twenty-five ordinary on-page checks covering metadata, headings, structured data and the signals that let a machine identify the business behind the site. All of that is free, and always will be. WITH A LICENCE How to fix each failing check, with a snippet you can paste, and a readiness score out of 100 for any site, compared against the median for its platform and category. The comparison comes from a public audit of 20,232 online stores. WHAT IT DOES NOT DO The crawler test checks the user-agent. A content delivery network that fingerprints TLS handshakes or filters by address range will pass this test and still block a real crawler. When it reports a block, believe it. When it reports none, read that as "not blocked by user-agent" rather than a clean bill of health. It also cannot tell you whether a particular assistant mentions you today. That needs a set of prompts run across each assistant with the answers recorded over time, which is a different exercise. PRIVACY It runs only when you click it. On that click it reads the page you are viewing and asks our server for the public score of the domain in your address bar. We receive that domain and nothing else. Not your browsing history, not other tabs, not the page contents, and nothing from sites you have not clicked on. The research behind it is public: tikbo.in/ecommerce-geo-index