Sitemaps and robots.txt: telling crawlers what to find

8 min read

A page sits on a site with no link pointing to it from anywhere else on the domain. Not from the homepage, not from a category page, not from anything. Weeks pass. It never appears in search. Nobody wrote a broken tag or misconfigured anything. Google simply never found a path to it. Crawling works by following links from page to page, and no page on the site links to this one.

On another site, a staging path is blocked with Disallow: /staging/ in robots.txt. The intent is to keep it out of search entirely. Disallow works exactly as written: Google never crawls that path again. But another site linked to the URL once, so it already knew the URL existed before the block went in. Not crawling a page and not listing it are two different things. Google lists the URL anyway, with no title and no snippet, because it never needed to fetch the page to know the URL exists.

Both problems trace back to two files that sit at the domain root: sitemap.xml and robots.txt. One hands Google a list of pages so it does not have to find them by chance. The other tells it which paths to leave alone. Neither one indexes or deindexes a page by itself. They only control what gets crawled in the first place, a step that happens before indexing even starts. It is the same layer a build never checks.

What a sitemap is

A sitemap is an XML file, conventionally at /sitemap.xml, that lists every URL Google should consider for indexing. Each entry can include the URL and a last modified date. Google uses lastmod as a signal for when to recrawl a page. Update it honestly when content actually changes, and the recrawl tends to get prioritised sooner.

A minimal sitemap:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-08-01</lastmod>
  </url>
</urlset>

The spec also defines two elements that no longer do anything: changefreq and priority.

<url>
  <loc>https://example.com/</loc>
  <lastmod>2026-08-01</lastmod>
  <changefreq>daily</changefreq>
  <priority>1.0</priority>
</url>

Google has stated for years that it ignores both. changefreq suggested how often a page changes; priority suggested relative importance on a 0.0 to 1.0 scale. Neither affects crawl behaviour or ranking. They are accepted without error if present, and accomplish nothing. Omit them.

Size limits and sitemap indexes

A single sitemap file is capped at 50,000 URLs and 50MB uncompressed. A site with more pages than that needs multiple sitemap files, referenced from one sitemap index file.

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-pages.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-posts.xml</loc>
  </sitemap>
</sitemapindex>

Submit the index file to Search Console rather than each individual sitemap. Google crawls the index, finds every referenced file, and processes them independently. Splitting by content type, as above, also makes it easier to spot which section of a large site has a crawl problem when the Search Console report comes back with errors.

Image and video sitemaps

A regular sitemap entry can also carry image and video metadata directly, using extra namespaces on top of the standard format. This gives Google more to work with than what it would find by parsing the page itself.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
        xmlns:image="http://www.google.com/schemas/sitemap-image/1.1"
        xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">
  <url>
    <loc>https://example.com/gallery/</loc>
    <image:image>
      <image:loc>https://example.com/photo.jpg</image:loc>
      <image:caption>A photo from the gallery</image:caption>
    </image:image>
    <video:video>
      <video:thumbnail_loc>https://example.com/thumb.jpg</video:thumbnail_loc>
      <video:title>Gallery walkthrough</video:title>
      <video:description>A short tour of the gallery.</video:description>
      <video:content_loc>https://example.com/video.mp4</video:content_loc>
    </video:video>
  </url>
</urlset>

A page can list up to 1,000 images per <url> entry. Video entries require at least thumbnail_loc, title, and description, plus one of two ways to point at the actual video: content_loc for a direct link to the video file, or player_loc for an embeddable player URL when the file itself is not directly downloadable.

<!-- Direct file, hosted on your own domain -->
<video:content_loc>https://example.com/video.mp4</video:content_loc>

<!-- Embeddable player, e.g. a YouTube or Vimeo embed URL -->
<video:player_loc>https://www.youtube.com/embed/VIDEO_ID</video:player_loc>

Use one or the other, not both, unless the video genuinely exists at both locations. Both extensions matter most on sites where the image or video is the content itself: a photography portfolio, a product catalogue, a video archive.

Submitting it to Google

Add the sitemap URL to Google Search Console under Sitemaps. Also declare it in robots.txt with a Sitemap: line. Both matter for different reasons. The Search Console submission gets a new site processed faster, since it is a direct request rather than something Google has to stumble onto. The robots.txt declaration means any crawler, not only Google, can find it without a manual submission step of its own.

Google and Bing both retired their sitemap ping endpoints, /ping?sitemap=, in 2023. Submitting through Search Console or Bing Webmaster Tools directly is now the only supported path. Any tutorial or script that pings those endpoints is calling a URL that no longer does anything.

What robots.txt does

robots.txt is a plain text file at the domain root that tells crawlers which paths to skip entirely. It uses Allow and Disallow rules grouped under User-agent directives. User-agent: * applies to every crawler that does not have its own more specific block.

User-agent: *
Disallow: /admin/
Disallow: /api/
Allow: /

Sitemap: https://example.com/sitemap.xml

Different crawlers can be given different rules by targeting them individually. A more specific User-agent block overrides the wildcard block for that crawler only:

User-agent: *
Disallow: /internal-search/

User-agent: Googlebot-Image
Disallow: /

User-agent: GPTBot
Disallow: /

The example above does three different things in one file: internal search result pages are hidden from every crawler, Google Images specifically is kept away from indexing images, and OpenAI's crawler is shut out entirely.

Two pattern characters extend beyond exact path matches. * matches any sequence of characters, and $ anchors the end of the URL:

  • Disallow: /*.pdf$: blocks every URL ending in .pdf, regardless of the path in front of it.
  • Disallow: /*?sort=: blocks any URL containing a sort query parameter, useful for keeping sorted or filtered list views out of the crawl budget.
  • Disallow: /search/*: blocks every path beginning with /search/, with or without the trailing $, since the wildcard already matches anything after it.

Crawl-delay is a directive that requests a minimum number of seconds between crawler requests. Google has ignored it since at least 2019 and manages crawl rate through Search Console settings instead. Bing and Yandex still honour it. Include it only if a non-Google crawler is putting real load on the server.

The file itself is capped at 500KB. Anything beyond that gets truncated, and only what fits gets processed. Google also caches a fetched robots.txt for up to 24 hours, so a change does not take effect on the next request, it takes effect on the next crawl.

What robots.txt does not do

Most of the trouble with robots.txt comes from expecting it to do things it was never designed to do. This is also where it overlaps with, and gets confused for, the robots meta tag covered in more depth in the meta tags reference. robots.txt controls whether a page is crawled. The meta tag controls what happens after a page is crawled. They are not interchangeable, and the most common failure is using one where the other was needed.

  • It does not prevent indexing. A disallowed URL can still appear in search results if another site links to it. Google indexes the URL itself without ever crawling the content behind it. The result is a bare link with no title and no snippet.
    Disallow: /staging/

# Google still lists it:
example.com/staging/old-page
"No information is available for this page"
  • It does not keep anything private. robots.txt is a public file, readable by anyone at example.com/robots.txt. Listing a sensitive path there announces exactly where it lives to every visitor, not only to well-behaved crawlers.
  • It does not remove existing results. Disallowing a URL that is already indexed does not pull it from results. It only stops Google from recrawling and refreshing what it already has stored.

Common mistakes

  • Blocking the whole site. A stray slash turns a scoped rule into a site-wide one.
    # Intended: block one folder
Disallow: /admin

# Typed instead: blocks everything
Disallow: /
  • Blocking CSS or JS. Googlebot renders the page like a browser to understand it. A rule that blocks the assets directory blocks the styles and scripts needed to render the page correctly, and Google indexes what looks like a broken, unstyled page.
    # Blocks Googlebot from seeing the site's own stylesheet and scripts
Disallow: /assets/
  • No sitemap on a large site. Google can still find pages through links alone, but the discovery is slower and less complete once a site passes a few hundred pages, and pages several links deep from the homepage may never get crawled at all.
  • Noindex pages listed in the sitemap. The sitemap is a list of pages meant to be indexed. Including a page that also carries a noindex meta tag sends Google two contradictory instructions and wastes part of the crawl budget confirming the contradiction.
    <!-- Page has noindex in its head, but is still listed in sitemap.xml -->
<meta name="robots" content="noindex" />

Checking what Google actually read

Google Search Console, under Settings, shows the robots.txt file Google fetched most recently and when it was last checked, not necessarily the current state of the file on the server. The Sitemaps report under the same section shows how many URLs were submitted, how many were indexed, and any parsing errors, per sitemap file.

A change to either file does not take effect the instant it is deployed. Request a recrawl of the sitemap by resubmitting it in Search Console, and expect the 24-hour robots.txt cache to run its course before assuming a rule change has taken hold.

What the build process will not tell you

No build step validates robots.txt syntax. A stray character in the wrong position can silently block entire sections of a site, and the build succeeds regardless, because the file is syntactically valid text even when it says the wrong thing. Next.js generates robots.txt from a robots.ts file in the app directory, but only if that file exists. Nothing scaffolds it by default.

The sitemap has the same blind spot. Nothing in the workflow confirms it lists the right pages, excludes the wrong ones, or keeps its dates current. The file can be well-formed XML and still be quietly wrong for months. Search Console is where that finally becomes visible, and only because someone opened it.