Ayzeo's user agents

AyzeoLinkCheck

User agent

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 AyzeoLinkCheck/1.0

What it does

AI assistants often link to pages when they answer questions about a brand. Ayzeo customers track those answers for their own brand, and AyzeoLinkCheck checks whether the links in them that point at the customer's own website still open. It records whether each link loaded, redirected or returned an error, and the customer sees the result in the URL Health report of their Ayzeo project.

It starts only from addresses on the website of the Ayzeo customer it works for, and follows a redirect only where that site sends it. It does not crawl a site, follow links it finds on pages, run JavaScript, submit forms or log in.

What it requests

• Pages of the site that AI answers linked to, or that AI models opened or looked at as sources while answering. A HEAD request first. When that answer is not conclusive or says the page is missing, or to read the page title, a GET limited to the start of the page with a Range header.
• A made-up address in each site section it checks, ending in ayzeo-404-probe- followed by random characters. It is expected to return 404: it shows how the site answers a page that does not exist, which is how the checker tells a real page from an error page that answers with status 200. When the site redirects that address instead, the checker also requests up to three pages of the same section that are known to work, to tell a redirect that only missing pages get from one that every request gets (a country, cookie or age page).
• A few candidate pages for a broken link, such as its path with one segment removed or a page from the site's sitemap with a matching name, to find the working page the link most likely meant. The customer gets these as suggestions to verify; nothing on your site is changed.
• robots.txt, and, when needed, the sitemap it names (or /sitemap.xml when it names none).

How often and how fast

• On average no more than 2 requests per second to any one host name, and no more than 4 requests per second across all host names of one company's domain (www.example.com and shop.example.com count together). These limits are shared by every Ayzeo customer who tracks the same site.
• A link is retried at most once, and only after a timeout or a connection error.
• An HTTP 429 answer pauses it for that host, for the time given in Retry-After (up to one hour), or for one minute when no time is given.
• Each Ayzeo project that tracks the site runs its own checks on the schedule below, so a site tracked by several projects can see the same link, and a made-up address in the same section, requested by each of them.
• Working links are rechecked at most once a week, and only while AI answers keep showing them (the comparison requests described above aside). A link that returned 404, 410 or a soft 404 is rechecked on every check, which runs after each scan of the project tracking it, for as long as AI answers keep showing it: the second check confirms it before it is reported as broken, and the later ones notice when it is fixed. Links that could not be reached are rechecked after one, three and then seven days.

robots.txt

AyzeoLinkCheck reads the robots.txt of each host at the start of each check and follows the first group that applies to it, in this order: AyzeoLinkCheck, then AyzeoBot, then *. A Crawl-delay in that group caps its rate on that host at one request per delay. A delay longer than three minutes is too slow for a check to finish, so AyzeoLinkCheck then checks no links on that host and tells the Ayzeo customer why.

When robots.txt returns a 4xx status other than 429, the site is treated as open, as RFC 9309 describes. When it cannot be read because of a server error, a 429 or a timeout, AyzeoLinkCheck checks no links on that host until its next check.

Why the user agent starts like a browser

Some firewalls refuse any user agent that contains the word "bot", which would make every link on those sites look unreachable. The user agent therefore starts like a desktop browser and ends with the product token AyzeoLinkCheck/1.0. Match on that token to recognise it.

AyzeoPageTitle

User agent

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 AyzeoPageTitle/1.0

What it does

When AI assistants answer the questions Ayzeo customers track, they open web pages to read them. Ayzeo customers see which pages those were, and AyzeoPageTitle reads each page's title so that the list names every page, not only its address. It keeps the page address, its title and whether the read worked, nothing else from the page.

It starts only from addresses that AI assistants opened while answering, and follows a redirect only where that site sends it. It does not crawl a site, follow links it finds on pages, run JavaScript, submit forms or log in.

What it requests

• The page an AI assistant opened: one GET limited to the first 64 KB of the page with a Range header, from which it reads the <title>. Only when that part ends before the page's head does, one more GET of up to 512 KB.
• robots.txt of the host, before it reads a page there. Each Ayzeo server keeps a copy for up to an hour, so a host sees at most a few requests for it an hour.

How often and how fast

• A page is read once, and again at most once every 30 days while AI assistants keep opening it. One read serves every Ayzeo customer whose answers opened the page.
• A page that could not be read is tried again after one, three and then seven days, then at most once a month.
• On its own it asks for no more than one request per second to any one host name, and it shares the limits of AyzeoLinkCheck: together, on average no more than 2 requests per second to any one host name, and no more than 4 requests per second across all host names of one company's domain.
• When a host refuses three requests in a row or does not answer, it stops reading that host for the run and tries those pages again a day later.
• An HTTP 429 answer pauses it for that host, for the time given in Retry-After (up to one hour), or for one minute when no time is given.

robots.txt

AyzeoPageTitle follows the first group in robots.txt that applies to it, in this order: AyzeoPageTitle, then AyzeoBot, then *. It does not request a path that group disallows. It does follow a redirect your site sends, and then only reads that page if its host's robots.txt allows it. A Crawl-delay in that group caps its rate on that host at one request per delay; with a delay longer than 30 seconds it requests no pages on that host.

When robots.txt returns a 4xx status other than 429, the site is treated as open, as RFC 9309 describes. When it cannot be read because of a server error, a 429 or a timeout, AyzeoPageTitle requests no page on that host and tries again later, after one, three and then seven days.

Why the user agent starts like a browser

For the same reason as AyzeoLinkCheck: some firewalls refuse any user agent that contains the word "bot". It ends with the product token AyzeoPageTitle/1.0. Match on that token to recognise it.

AyzeoBot

User agent

Mozilla/5.0 (compatible; AyzeoBot/1.0; +https://ayzeo.com/bot)

AyzeoBot fetches the files an Ayzeo site audit reads, such as robots.txt, llms.txt, the XML sitemap and an AI instructions page. It runs when someone starts an Ayzeo audit of the site, usually the site's owner or an Ayzeo customer who tracks it, and on the scheduled audits of sites that Ayzeo customers track.

A site audit also loads the audited page itself, and checks whether the links on it open, the way a regular web browser does. Those requests do not carry the AyzeoBot token.

The request rate and robots.txt rules described above apply to AyzeoLinkCheck and AyzeoPageTitle.

Allow or block

Allow it in a firewall

If your firewall or bot protection answers these requests with a challenge or an HTTP 403, add a rule that matches the product token in the User-Agent header: AyzeoLinkCheck for link checks, AyzeoPageTitle for page titles, AyzeoBot for the files a site audit reads. Until then, link checks show those links as unreachable, never as broken, and the titles of your pages are not read.

Block it with robots.txt

To stop link checks on your whole site:

User-agent: AyzeoLinkCheck
Disallow: /

AyzeoLinkCheck does not request a path robots.txt disallows for it. It does follow a redirect your site sends, and then only reads that page if robots.txt allows it. A robots.txt rule never makes a link count as broken. A link it has not checked before is shown to the Ayzeo customer as not checked because of robots.txt. A link it checked before the rule keeps the status measured then, with a note that robots.txt now stops the check. To block a single section, disallow that path instead.

To stop page title reads on your whole site:

User-agent: AyzeoPageTitle
Disallow: /

AyzeoPageTitle does not read pages your robots.txt closes to it. A title it read before you added the rule is removed the next time that page is due, as long as AI assistants keep opening it. Ayzeo customers may still see a title that an AI assistant itself showed for the page.