Menu

Robots.txt Checker

Read the weekly check of your robots.txt file - fetch status, syntax findings, Googlebot rules, and Sitemap lines.

Updated

On this page

This tool may not be available yet. If you do not see it in Optimize, it is not available yet.

What it shows

Once a week, Kong Metrics reads https://<host>/robots.txt with its own crawler, KongMetricsBot. It checks one exact host with HTTPS. Then it shows:

  • the fetch result,
  • findings about the file,
  • Googlebot verdicts for the home page and some key paths,
  • the Sitemap lines in the file.

You cannot start a check manually.

When to use it

  • After a change to robots.txt, to make sure that Googlebot can still crawl important paths.
  • When pages are missing from Google and you want to exclude a crawl block.

How to use it

  1. Open Optimize → Robots.txt Checker.
  2. Check Checked address, Last checked, and Next check.
  3. Read Fetch result. If the file was not fetched, the other panels can be empty.
  4. Read Findings. They are grouped by severity: High priority, Warning, and Information.
  5. In Googlebot, read the verdict for each path and its Deciding rule.
  6. Read Sitemap lines.

For a Domain property, the Owner can select the checked host in Site Settings → Robots.txt. The new host is used at the next weekly check.

How to read the result

PanelWhat it tells you
Fetch resultIf Kong Metrics could read the file. For example: Fetched, No robots.txt, Server error, or No answer in time.
FindingsProblems in the file. A finding on one line does not make the whole file invalid.
GooglebotAllowed, Blocked, or Not evaluated for the home page, the property path, and up to 3 sitemap URLs. Deciding rule is the line that decides the verdict.
Sitemap linesThe Sitemap addresses in the file. Kong Metrics reads these lines. It does not fetch the sitemaps here.
Recent checksWeekly results of the last 30 days. It shows when there is more than one check.

Example

All data is fictional. The file at https://example.com/robots.txt contains:

User-agent: *
Disallow: /cart/
Crawl-delay: 10

Sitemap: https://example.com/sitemap.xml

The checker shows a finding for Crawl-delay, because Google ignores this line. The Googlebot verdict for the home page is Allowed. The Sitemap line has the same host, so it is correct.

Limits

  • Allowed means that Googlebot can crawl the path. It does not mean that Google indexes the page.
  • Kong Metrics reads only the first 500 KiB of the file. Google also ignores content after 500 KiB.
  • “No findings” means that the file has none of the problems that Kong Metrics checks for. It does not mean that the site has no technical problems.
  • To fix a problem, change the file on your website. Kong Metrics does not change your file.

If you see no result

  • “First check runs on (weekday).”: the first weekly check has not run yet.
  • “No Sitemap lines. This is not an error.”: the file has no Sitemap line.

Choose which optional cookies we may use on kongmetrics.com and in the Kong Metrics App. You can change this at any time from "Cookie preferences" in the footer. Cookie Policy