AyzeoBot and AyzeoLinkCheck

AyzeoLinkCheck

User agent

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 AyzeoLinkCheck/1.0

What it does

AI assistants often link to pages when they answer questions about a brand. Ayzeo customers track those answers for their own brand, and AyzeoLinkCheck checks whether the links in them that point at the customer's own website still open. It records whether each link loaded, redirected or returned an error, and the customer sees the result in the URL Health report of their Ayzeo project.

It starts only from addresses on the website of the Ayzeo customer it works for, and follows a redirect only where that site sends it. It does not crawl a site, follow links it finds on pages, run JavaScript, submit forms or log in.

What it requests

• Pages of the site that AI answers linked to, or that AI models opened or looked at as sources while answering. A HEAD request first. When that answer is not conclusive or says the page is missing, or to read the page title, a GET limited to the start of the page with a Range header.
• A made-up address in each site section it checks, ending in ayzeo-404-probe- followed by random characters. It is expected to return 404: it shows how the site answers a page that does not exist, which is how the checker tells a real page from an error page that answers with status 200. When the site redirects that address instead, the checker also requests up to three pages of the same section that are known to work, to tell a redirect that only missing pages get from one that every request gets (a country, cookie or age page).
• A few candidate pages for a broken link, such as its path with one segment removed or a page from the site's sitemap with a matching name, to find the working page the link most likely meant. The customer gets these as suggestions to verify; nothing on your site is changed.
• robots.txt, and, when needed, the sitemap it names (or /sitemap.xml when it names none).

How often and how fast

• On average no more than 2 requests per second to any one host name, and no more than 4 requests per second across all host names of one company's domain (www.example.com and shop.example.com count together). These limits are shared by every Ayzeo customer who tracks the same site.
• A link is retried at most once, and only after a timeout or a connection error.
• An HTTP 429 answer pauses it for that host, for the time given in Retry-After (up to one hour), or for one minute when no time is given.
• Each Ayzeo project that tracks the site runs its own checks on the schedule below, so a site tracked by several projects can see the same link, and a made-up address in the same section, requested by each of them.
• Working links are rechecked at most once a week, and only while AI answers keep showing them (the comparison requests described above aside). A link that returned 404, 410 or a soft 404 is rechecked on every check, which runs after each scan of the project tracking it, for as long as AI answers keep showing it: the second check confirms it before it is reported as broken, and the later ones notice when it is fixed. Links that could not be reached are rechecked after one, three and then seven days.

robots.txt

AyzeoLinkCheck reads the robots.txt of each host at the start of each check and follows the first group that applies to it, in this order: AyzeoLinkCheck, then AyzeoBot, then *. A Crawl-delay in that group caps its rate on that host at one request per delay. A delay longer than three minutes is too slow for a check to finish, so AyzeoLinkCheck then checks no links on that host and tells the Ayzeo customer why.

When robots.txt returns a 4xx status other than 429, the site is treated as open, as RFC 9309 describes. When it cannot be read because of a server error, a 429 or a timeout, AyzeoLinkCheck checks no links on that host until its next check.

Why the user agent starts like a browser

Some firewalls refuse any user agent that contains the word "bot", which would make every link on those sites look unreachable. The user agent therefore starts like a desktop browser and ends with the product token AyzeoLinkCheck/1.0. Match on that token to recognise it.

AyzeoBot

User agent

Mozilla/5.0 (compatible; AyzeoBot/1.0; +https://ayzeo.com/bot)

AyzeoBot fetches the files an Ayzeo site audit reads, such as robots.txt, llms.txt, the XML sitemap and an AI instructions page. It runs when someone starts an Ayzeo audit of the site, usually the site's owner or an Ayzeo customer who tracks it, and on the scheduled audits of sites that Ayzeo customers track.

A site audit also loads the audited page itself, and checks whether the links on it open, the way a regular web browser does. Those requests do not carry the AyzeoBot token.

The request rate and robots.txt rules described above apply to AyzeoLinkCheck.

Allow or block

Allow it in a firewall

If your firewall or bot protection answers these requests with a challenge or an HTTP 403, add a rule that matches the product token in the User-Agent header: AyzeoLinkCheck for link checks, AyzeoBot for the files a site audit reads. Until then, the Ayzeo customer who tracks your site sees those links as unreachable. Unreachable links are never reported as broken.

Block it with robots.txt

To stop link checks on your whole site:

User-agent: AyzeoLinkCheck
Disallow: /

AyzeoLinkCheck does not request a path robots.txt disallows for it. It does follow a redirect your site sends, and then only reads that page if robots.txt allows it. A robots.txt rule never makes a link count as broken. A link it has not checked before is shown to the Ayzeo customer as not checked because of robots.txt. A link it checked before the rule keeps the status measured then, with a note that robots.txt now stops the check. To block a single section, disallow that path instead.