This tool may not be available yet. If you do not see it in Optimize, it is not available yet.
What it shows
Once a week, Kong Metrics reads https://<host>/robots.txt with its own crawler, KongMetricsBot. It checks one exact host with HTTPS. Then it shows:
- the fetch result,
- findings about the file,
- Googlebot verdicts for the home page and some key paths,
- the Sitemap lines in the file.
You cannot start a check manually.
When to use it
- After a change to robots.txt, to make sure that Googlebot can still crawl important paths.
- When pages are missing from Google and you want to exclude a crawl block.
How to use it
- Open Optimize → Robots.txt Checker.
- Check Checked address, Last checked, and Next check.
- Read Fetch result. If the file was not fetched, the other panels can be empty.
- Read Findings. They are grouped by severity: High priority, Warning, and Information.
- In Googlebot, read the verdict for each path and its Deciding rule.
- Read Sitemap lines.
For a Domain property, the Owner can select the checked host in Site Settings → Robots.txt. The new host is used at the next weekly check.
How to read the result
| Panel | What it tells you |
|---|---|
| Fetch result | If Kong Metrics could read the file. For example: Fetched, No robots.txt, Server error, or No answer in time. |
| Findings | Problems in the file. A finding on one line does not make the whole file invalid. |
| Googlebot | Allowed, Blocked, or Not evaluated for the home page, the property path, and up to 3 sitemap URLs. Deciding rule is the line that decides the verdict. |
| Sitemap lines | The Sitemap addresses in the file. Kong Metrics reads these lines. It does not fetch the sitemaps here. |
| Recent checks | Weekly results of the last 30 days. It shows when there is more than one check. |
Example
All data is fictional. The file at https://example.com/robots.txt contains:
User-agent: *
Disallow: /cart/
Crawl-delay: 10
Sitemap: https://example.com/sitemap.xml
The checker shows a finding for Crawl-delay, because Google ignores this line. The Googlebot verdict for the home page is Allowed. The Sitemap line has the same host, so it is correct.
Limits
- Allowed means that Googlebot can crawl the path. It does not mean that Google indexes the page.
- Kong Metrics reads only the first 500 KiB of the file. Google also ignores content after 500 KiB.
- “No findings” means that the file has none of the problems that Kong Metrics checks for. It does not mean that the site has no technical problems.
- To fix a problem, change the file on your website. Kong Metrics does not change your file.
If you see no result
- “First check runs on (weekday).”: the first weekly check has not run yet.
- “No Sitemap lines. This is not an error.”: the file has no Sitemap line.