Are the robots ignoring you? Beware of your robots.txt file

SEO, Techy Stuff! No Comments | Date Published: 31 January 2013
Share on Twitter

A robot spider - is your robots.txt file causing you problems?A client come to me last week complaining that they were not seeing any movement with the work that was being done on the SEO side for their site. After an initial discussion on setting expectations for how long SEO work can take, it transpired that it was brand terms that were not ranking, and this rang warning bells!

SEO work can indeed take a long time to take effect. It will depend on a variety of issues such as:

  • How competitive the keywords are that you are targeting – ie how many other webpages out there are talking about your keywords
  • How old your site is and your reputation
  • Your link profile – ie what kinds of links you have pointing to your site

For a hotel client that I worked with last year, we did a lot of work in May, and real improvements were not seen until September (see the Rosevine case study).

However for terms including your brand, you should expect to see good rankings fairly quickly, especially for bigger companies.

So, I took a look around, and found that the website’s robots.txt file was causing the problems. For those of you that don’t know what the robots.txt file is, it’s a set of instructions that you can give the search engine spiders with regards to what you would like them NOT to index when crawling your site. The default is to index everything, but there may be occasions when you would like particular content NOT to be indexed. If you want to understand more about how the robots.txt file works, check out this site.

It turns out that in error, my client’s robots.txt file was set to ignore everything. Ooops. No wander the pages were not showing in searches.

This is how it looked:

Useragent: *
Disallow: /

The trailing slash after the Disallow means ignore everything after a /

The file should have looked like this:

Useragent: *

So if you can’t see any pages indexed for your site, or you see something like the following showing under the link to your site in Google, check your robots.txt file and make sure you have not inadvertently set it to ignore everything!

Meta description image - "A description for this result is not available because of this site's robot.txt - learn more"

By the way, in case you are not sure, the way you can check your own robots.txt file is to add /robots.txt after your website URL – eg In our robots.txt file we are asking the search engines to ignore anything that appears in the folders wp-admin, wp-includes and wordpress.

If you found this article helpful and want to be sure you know when the next article is posted, sign up to my RSS feed or follow me on twitter @wendychamier. And if you’d like to share the article or link to it then that would be great!

If you have a riddle on your site you would like help in solving, please contact us or check out the website audit services that we can offer.

By Wendy Chamier

Share on Twitter

Leave a Reply


two × = 2