How to Use Robot Meta Tags and X-Robots Tag Specifications on Your Website Set up to Show Your Content in Google Properly
You’ve probably heard before that your website must communicate with two audiences to perform optimally. These two audiences are users on your website to communicate the value of your products/service, and it must be set up properly for search engines, like Google, to present your content properly.
If you’re happy with the content on your website, but you’re unsure if it shows up properly in Google or other search engines, keep reading to find out what you need to do to periodize the indexing commands using robot meta tags and X-robots tags on your website.
What are Robots Meta Tag Data-nosnippet and X-Robots-Tag Specifications?
These tags are settings on your website. They are your starting point to help you index your content properly. Keep reading for a brief description on how to use these settings to make page-level and text-level changes to your website, or review this article for full details.
You can specify page-level settings by including a meta tag on HTML pages or in an HTTP header. You can also specify text-level settings with the data-nosnippet attribute on HTML elements within a page.
How to Use the Robots Meta Tag
The robots meta tag makes it possible to use a granular, page-specific approach to controlling how an individual page should be indexed and presented to users when they perform a Google search. Place the robots meta tag in the <head> section of a page and type the following instructions below so this page does not appear in a Google search:
<!DOCTYPE html>
<html><head>
<meta name=”robots” content=”noindex” />
(…)
</head>
<body>(…)</body>
</html>
After setting up the robots meta tag, it will instruct search engines not to show the page in search results. The value of the name attribute (robots) specifies that these instructions apply to all crawlers. A crawler is an Internet bot that systematically browses websites for the purpose of web indexing.
To address a specific crawler, you can replace the robot’s value of the name attribute with the crawler’s name. Specific crawlers are also called user-agents, and a crawler uses its user-agent to request a page.
Google’s standard web crawler is named “Googlebot.” If you want to stop Googlebot from indexing your page, make the following changes to your website:
- <meta name=”googlebot” content=”noindex” />
After inputting these commands, this tag will instruct Google not to show this page in search results. It’s also important to know that the name and the content attributes are not case sensitive.
More In-Depth Information About Search Engines
Search engines may have multiple crawlers for different properties or purposes. Please reference this article for a complete list of Google’s crawlers.
For example, if you wanted to show a page in Google’s web search results, but not in Google News, you would use the following meta tag:
- <meta name=”googlebot-news” content=”noindex” />
To specify multiple crawlers individually, you would use multiple robots meta tags:
- <meta name=”googlebot” content=”noindex”>
- <meta name=”googlebot-news” content=”nosnippet”>
Are Meta Robot Tags or Robots txt More Effective?
Robot .txt files are the best choice for this purpose because they give administrators more control than meta robot tags do. If set up properly, you are able to block/hide an entire folder or section of a website, such as products, categories or archives.
However, meta robot tags are better at blocking and hiding single posts, pages or PDFs in Google or other search engine results.
Meta robot tags and robot .txt perform important functions related to web indexing, and you may need to use a combination of both settings to present your website effectively online.