Robots & Meta Analyzer: Overview
This tool fetches robots.txt, any sitemaps it references, and the target page's
HTML <head> — then reports discoverability and SEO signals in one place.
What it reports
- robots.txt — presence,
User-agentgroupings,Disallowrules, declared sitemaps. - sitemap.xml — validity, URL count (sampled),
lastmodmetadata. <head>meta —title, description, canonical,<meta name="robots">, Open Graph and Twitter tags.
What it does not do
- It does not crawl the full sitemap — only the first N URLs, to stay polite.
- It does not render JavaScript — pages that set canonical tags from the client will show as missing.
- It does not audit schema.org / JSON-LD structured data in depth.