Indexability Checking

If someone changed the robots.txt to deny all, how quickly would you notice? * Change Me!

Indexability refers to whether a page on your website is eligible to be indexed by search engines or not. This is very important because if a page can not be indexed, then it has no chance of ranking and generating traffic.

What can prevent indexing of my pages?

There are a few ways that you can prevent a page from being indexed and unfortunately, pretty much all of them can happen by accident. This may be someone who has checked the wrong box in the CMS or a developer who has deleted some code without realising how important it is.

The most common methods for preventing indexing are:

• META Robots tag: this can tell search engines to not index a page by using a directive called “noindex”. When search engines crawl a page and see this tag, they will usually not index a page. You can see this tag by viewing the HTML of a page.

• The X-Robots tag: this serves the same purpose as the tag above, but it is set within the HTTP response of a page which means that you can’t see it by viewing the HTML of a page. You need to use a tool to check the HTTP response.

Both of these tags can be set on a single page or some platforms will let you set the tags on a group of pages that follow the same template.

What happens if a noindex tag gets added to an indexed page?

If you have a page that is already indexed, but someone accidentally adds a META Robots or X-Robots noindex tag, then search engines will find it next time they crawl the page. They will interpret the addition of this tag as meaning you do not want the page to be indexed and available in organic search anymore.

Search engines may not instantly remove the page from their index, especially if the page has been indexed for a while, but if they come back again and the tag is still there, they may do.

So, after a few days or a week or two, your page may lose it’s organic search rankings and you will lose traffic as a result.

The big problem here is that it could take a few weeks before you even notice this, so you really need to constantly monitor your pages for indexability so that if a change is made, you can spot it and fix it before you lose traffic.