Meta tags: what each one tells a crawler and what to leave out

12 min read

A page was meant to stay out of search results. The robots meta tag is set to noindex. The URL is also blocked in robots.txt. Google never reads it. The page appears in results anyway, listed as a URL known from external links, with no content snippet and no cached version.

On another page, the snippet shown in search results is not the description tag text. The tag is present, well-formed, and under 160 characters. Google rewrote it. The query the page ranks for does not match the description closely enough, so Google substituted whatever paragraph it found more relevant to that search. The description exists in the source. The snippet just comes from somewhere else.

Both are meta tag problems. Both are invisible from the browser. Meta tags sit in the document head, seen by no visitor, and form the direct interface between a page and every automated system that reads it: crawlers, mobile browsers, social platform scrapers, operating system link previews. Getting them wrong produces effects that only surface in a crawl report, weeks later. This is the same invisible layer every other technical SEO problem lives in.

description

<meta name="description" content="How duplicate content silently splits ranking signals, and what a canonical tag does about it." />

Provides the text shown under a page title in search results. Google uses it when it matches the search query and rewrites it from page content when it does not. An absent tag produces the same outcome as a non-matching one: Google picks its own text.

  • Keep under 160 characters. Google truncates at around 155 to 160 depending on viewport and device.
  • Write for the query, not the page. Use the words someone searching for this topic would type, not the words the page uses to describe itself.
  • Unique per page. Identical descriptions across many pages signal thin or duplicated content.

robots

Controls what Google does with a page after crawling it. robots.txt decides whether Google visits a page at all. The robots meta tag is read after the crawl and tells Google what to do with the content it found. The two are separate controls, not two ways to accomplish the same thing.

<meta name="robots" content="noindex, nofollow" />

All directives for the content attribute, comma-separated, in any order:

  • index / noindex: whether to include this page in search results. Default is index.
  • follow / nofollow: whether to follow links on this page and pass ranking signals through them. Default is follow.
  • noarchive: prevents Google from showing a cached version of the page in results.
  • nosnippet: no text snippet or video preview in results. Title only.
  • max-snippet: N: limits snippet text to N characters. max-snippet: 0 is equivalent to nosnippet.
  • max-image-preview: none | standard | large: controls how large an image preview can appear in results.
  • noimageindex: images on this page are not indexed separately in Google Images.
  • unavailable_after: [date]: removes this page from results after a specific date. Use ISO 8601 format.
  • none: shorthand for noindex, nofollow.
  • all: shorthand for index, follow. Redundant since it is the default, but explicit.

To target a specific crawler rather than all crawlers, replace robots with the crawler name. Google reads both robots and googlebot and applies the most restrictive combination:

<meta name="googlebot" content="noindex" />
<meta name="bingbot" content="nofollow" />

X-Robots-Tag

The same directives as the robots meta tag, delivered as an HTTP response header instead of HTML. Necessary for file types that have no <head>: PDFs, images, video files. The meta tag and the X-Robots-Tag header can coexist on the same page. Google reads both and applies the most restrictive combination.

X-Robots-Tag: noindex
X-Robots-Tag: googlebot: noindex, nofollow

On a static site without server-side logic, set it through a _headers file, a CDN rule, or hosting platform configuration depending on where the site is deployed.

charset

<meta charset="UTF-8" />

Declares the character encoding for the document. Place it as the first tag in <head>, within the first 1024 bytes. A browser that encounters a character it cannot decode with the declared encoding renders it as a garbled symbol. Crawlers, parsers, and accessibility tools then read those symbols as the actual content.

UTF-8 encodes every character in every living language. There is no reason to use any other value.

viewport

<meta name="viewport" content="width=device-width, initial-scale=1" />

Tells the browser how to scale the page on mobile. Without it, mobile browsers render at desktop width and scale down, which breaks layout and collapses Core Web Vitals scores. Google uses mobile-first indexing, meaning the mobile render is what gets indexed. A page without the viewport tag sends Google a broken mobile version of itself.

All properties available in the content attribute:

  • width: viewport width. device-width matches the screen. A fixed number (e.g. 320) sets a specific pixel width, rarely needed.
  • height: viewport height. device-height or a fixed pixel value. Almost never required. Width alone is sufficient for layout.
  • initial-scale: zoom level when the page first loads. 1 means no zoom. Accepts values from 0.1 to 10.
  • minimum-scale: how far the user can zoom out. Default is 0.25 in most browsers. Accepts 0.1 to 10.
  • maximum-scale: how far the user can zoom in. Default is 5. Setting to 1 effectively disables zoom-in.
  • user-scalable: yes or no. Whether the user can zoom at all. no prevents all zooming and fails WCAG 1.4.4. Do not use it.
  • viewport-fit: auto (default), contain, or cover. Controls whether the viewport extends into the notch and home indicator area on newer iPhones. cover allows content to fill the full screen including safe areas.

format-detection

<meta name="format-detection" content="telephone=no" />

Safari on iOS auto-detects phone numbers, email addresses, dates, and street addresses in page text and converts them into tappable links. This rewrites visible content in ways the CSS cannot anticipate and can break layout. format-detection disables specific auto-detections.

<meta name="format-detection" content="telephone=no, date=no, email=no, address=no" />

Not a ranking factor. Affects mobile layout stability, which feeds into CLS in Core Web Vitals.

color-scheme

<meta name="color-scheme" content="light dark" />

Tells the browser which color schemes the page supports before CSS loads. Without it, a page that supports dark mode may flash white on dark-mode devices while the stylesheet downloads, because the browser defaults to a light background. With the tag set, the browser matches the user's system preference from the first painted pixel.

<meta name="color-scheme" content="light dark" />   <!-- both supported, use system preference -->
<meta name="color-scheme" content="only light" />  <!-- force light regardless of preference -->
<meta name="color-scheme" content="dark" />        <!-- only dark mode supported -->

theme-color

<meta name="theme-color" content="#0a0a0a" />

Sets the browser chrome color on mobile: the address bar, tab strip, and status bar in Chrome on Android. Not a ranking factor. Appears in some mobile search result previews. Can be made theme-aware using the media attribute:

<meta name="theme-color" content="#fafafa" media="(prefers-color-scheme: light)" />
<meta name="theme-color" content="#0a0a0a" media="(prefers-color-scheme: dark)" />

referrer

<meta name="referrer" content="strict-origin-when-cross-origin" />

Controls how much referrer information is sent when a user follows a link off the page. Not a ranking factor. Affects analytics accuracy for every site your visitors click through to. When referrer data is stripped, traffic from your pages appears as direct in the destination site's analytics rather than as referral traffic from your domain.

  • no-referrer: no referrer header sent. Traffic from your site appears as direct in every destination's analytics.
  • no-referrer-when-downgrade: full URL on same-protocol requests, nothing when downgrading HTTPS to HTTP. Was the browser default before 2020.
  • origin: domain only on every request. Path and query string are never shared.
  • origin-when-cross-origin: full URL on same-origin requests, domain only on cross-origin.
  • same-origin: full URL on same-origin requests, nothing on cross-origin. External destinations receive no referrer at all.
  • strict-origin: domain only on same-protocol requests, nothing when downgrading HTTPS to HTTP.
  • strict-origin-when-cross-origin: full URL on same-origin, domain only on cross-origin same-protocol, nothing when downgrading. The recommended default and the current browser default.
  • unsafe-url: full URL on every request regardless of protocol. Leaks paths and query strings to every external destination, including over unencrypted connections.

http-equiv

http-equiv meta tags simulate HTTP response headers inside HTML. They apply when the hosting environment does not allow direct header configuration.

content-type

<meta http-equiv="content-type" content="text/html; charset=UTF-8" />

The legacy way to declare character encoding before <meta charset> existed. Both do the same thing. If both are present, <meta charset> takes precedence. Use <meta charset="UTF-8"> instead. This form still appears in HTML generated by older tools and templating systems.

refresh

Refreshes the page after N seconds, or redirects to a different URL after a delay.

<!-- Reload the page every 30 seconds -->
<meta http-equiv="refresh" content="30" />

<!-- Redirect after 0 seconds -->
<meta http-equiv="refresh" content="0; url=https://example.com/new-page/" />

A zero-delay redirect works, but it passes reduced ranking signals to the destination compared to a 301 HTTP redirect. Use http-equiv="refresh" only when a server-level redirect is not possible.

default-style

<meta http-equiv="default-style" content="main-stylesheet" />

Specifies which linked stylesheet should be the preferred default when multiple alternate stylesheets are declared. The value must match the title attribute on the corresponding <link> element. Rarely used in practice.

cache-control, pragma, expires

<meta http-equiv="cache-control" content="no-cache" />
<meta http-equiv="pragma" content="no-cache" />
<meta http-equiv="expires" content="0" />

Attempt to control caching via HTML. Modern browsers and CDNs ignore these in favour of actual HTTP cache headers. They do not prevent caching in any current browser or proxy. They still appear in legacy codebases and have no effect.

x-ua-compatible

<meta http-equiv="x-ua-compatible" content="IE=edge" />

Instructed Internet Explorer to render using its most current engine rather than a compatibility mode. Internet Explorer reached end-of-life in June 2022. This tag has no effect in any current browser. It still appears in HTML boilerplates and CMS templates written before 2022 and not since updated. It does nothing and can be removed.

content-security-policy

<meta http-equiv="content-security-policy" content="default-src 'self'" />

Sets a Content Security Policy via HTML rather than an HTTP header. The meta tag version is weaker: it cannot protect against the HTML itself being tampered with before the policy is parsed, which the HTTP header prevents. On static sites where header configuration is not available, the meta tag is the only option. It restricts which scripts, styles, and resources the page is allowed to load.

Home screen and web app tags

These control how a page behaves when saved to a mobile home screen. They have no effect on search ranking.

<!-- iOS: hide browser chrome when launched from home screen -->
<meta name="apple-mobile-web-app-capable" content="yes" />

<!-- iOS: title under the home screen icon (falls back to <title> if absent) -->
<meta name="apple-mobile-web-app-title" content="App Name" />

<!-- iOS: status bar appearance: default | black | black-translucent -->
<meta name="apple-mobile-web-app-status-bar-style" content="default" />

<!-- Android/Chrome: same as apple-mobile-web-app-capable for Chrome -->
<meta name="mobile-web-app-capable" content="yes" />

<!-- All platforms: web app name shown in browser UI (separate from the page <title>) -->
<meta name="application-name" content="App Name" />

apple-mobile-web-app-capable and mobile-web-app-capable are the meta tag equivalents of the web app manifest display: standalone property. Set both for consistent behaviour across iOS and Android. application-name stays constant across pages where the <title> changes, giving the browser a stable name for the installed app.

Tags that no longer matter

These appear in code written before a specific cutoff and in tutorials that copied from that era. None have any effect on search ranking or crawl behaviour in any current search engine.

  • keywords: ignored by Google since 2009, ignored by Bing, ignored by every major crawler. Used in the early web to declare topic keywords for search engines. All major engines dropped it after keyword stuffing turned it into a spam signal.
    <meta name="keywords" content="seo, meta tags, html, search" />
  • author: not used for ranking. May be read by RSS aggregators or academic scrapers. No effect on indexing or display in search results.
    <meta name="author" content="Nitish Kumar" />
  • generator: identifies the software that produced the HTML. No SEO value. Sometimes used by security researchers to fingerprint software versions against known vulnerability lists. Usually removed in production.
    <meta name="generator" content="CMS Platform 6.0" />
  • revisit-after: a suggested crawl frequency. Every major crawler ignores it. Google determines its own crawl schedule based on content freshness and inbound links, not this tag.
    <meta name="revisit-after" content="7 days" />
  • rating: a content rating tag for marking pages as safe, general, or adult. SafeSearch uses content analysis rather than this tag. No major engine uses it.
    <meta name="rating" content="general" />
  • copyright, language, abstract, distribution, coverage: all appear in legacy boilerplates. None affect indexing, ranking, or display in any current search engine.

Precedence rules

robots.txt and the robots meta tag look like two ways to do the same thing. They are not. robots.txt is the door: it controls whether Google enters a page at all. The robots meta tag is the instruction inside: it only gets read if Google enters. Block the door, and Google never reaches the instruction.

A page blocked in robots.txt and tagged noindex looks doubly protected. It is not. Google hits the Disallow, stops, and never reads the noindex. If Google already knew about the URL from an external link, the page stays in results regardless. The noindex sitting inside did nothing.

When directives conflict, Google applies a fixed hierarchy:

  • robots.txt Disallow: page not crawled. The meta robots tag is never reached. noindex set here does nothing because Google cannot read it.
  • Meta robots noindex: page crawled and then excluded from the index. The page must be crawlable for this to work.
  • HTTP header X-Robots-Tag and meta robots tag together: Google reads both and applies the most restrictive combination.
  • Crawler-specific tag (e.g. googlebot) overrides the generic robots tag for that crawler only. The most restrictive combination of both applies when both are present.

To deindex a page: remove the Disallow, keep the page crawlable, and add noindex. Google enters, reads it, and removes the page from results.

What the build process will not tell you

No build step validates the robots tag against robots.txt for contradictions. No warning fires when the description is missing. No error appears when the viewport tag is absent. The page deploys, renders correctly in the browser, and the crawl and indexing problems sit invisible until Google Search Console surfaces them.

In Search Console, URL Inspection shows the exact meta tags Google read on its last crawl, not the current state of the source. After any change that affects crawl or indexing behaviour, request indexing. The source file shows what you wrote. The crawled page view shows what Google read. Those two are not always the same document.