Legal
KongMetricsBot
About 3 min read
Contents
KongMetricsBot is the crawler of Kong Metrics. It reads public pages that Google Search Console already reports for a website whose owner connected that property to Kong Metrics. It reads the page title, main heading and main text to tell if two pages serve the same search intent and to suggest internal links between pages of the same site. It also reads the robots meta tag to skip noindex pages. It reads some more files and pages for AI readiness reports and for the free AI access check (see below).
How it behaves
- User-Agent:
KongMetricsBot/1.0 (+https://kongmetrics.com/bot) - It obeys robots.txt rules for
User-agent: KongMetricsBot. To block it, addUser-agent: KongMetricsBotandDisallow: /to your robots.txt. - It does not follow links, log in or submit forms. It runs JavaScript only for the AI readiness report (see below). It follows at most 3 redirects on the same host.
- Successful page reads are cached for about 14 days. Temporary fetch errors (timeout, unreachable host, server error) can be retried the next day. It sends at most 8 parallel requests per host.
- Once a week it reads the robots.txt file of each connected site (one address, HTTPS only) to show the site owner a robots.txt report. It does not fetch other hosts that the file names.
- Once a week it also reads the llms.txt file and up to 20 pages of each connected site (the home page and the pages with the most Google Search clicks) to show the site owner an AI readiness report. For this report it may also load these pages in a headless browser that runs the page's JavaScript, with the same user agent. The browser does not load images, fonts or media, keep cookies, log in or submit forms.
- When someone uses the free AI access check on kongmetrics.com, it reads the robots.txt file, the llms.txt file and the home page of the website that person entered, once. It does not run JavaScript for this check.
Sitemap reader
Kong Metrics also reads the sitemap files that a site owner submitted in Google Search Console. It uses them to list the pages it checks with Google URL Inspection.
- User-Agent:
KongMetrics/1.0 (index-scanner) - It reads only the sitemaps listed in Search Console and the files that a sitemap index names. It does not use robots.txt to find sitemaps, because the site owner chose them in Search Console.
- It follows at most 3 redirects within the same site and reads gzip files.
- It reads each sitemap at most twice a day.
If your bot protection blocks it
Some bot protection services, for example Cloudflare, block the sitemap reader. Kong Metrics then cannot list your pages. Allow the user agent KongMetrics on sitemap URLs.
- In Cloudflare: Security → WAF → Custom rules → Create rule. Expression:
(http.request.uri.path contains "sitemap" and http.user_agent contains "KongMetrics"). Action: Skip, with all components selected. Deploy it and move it to the top. - A rule cannot skip Bot Fight Mode. If it is on, turn it off or contact us.
- The rule opens only your sitemap files, which are public. The rest of your site keeps its protection.
Contact
Questions: [email protected]
Last updated: 12 November 2026