Is Your Site Not Being Indexed By Google? Check Your Robots.txt File
This blog entry was posted on March 2, 2012.
Is Google not indexing your website? You may have optimized your website and still you get no search traffic? Your robots.txt file may be to blame.
I was recently asked to check over a website that had zero pages indexed with Google, but 945 pages indexed by Bing. The domain had a website on it continuously for over ten years. A new website was built a few years back. I was asked to make recommendations to fix the problem.
The website had a number of minor SEO problems, but the big issue was the robots.txt file. The file showed:
User-agent: *
Disallow: /
The “User-agent: *” means this applies to all robots or search engine spiders. The “Disallow: /” tells the spiders that they should not visit any pages on this site.
My recommendation was to rewrite the robots.txt file to only disallow sections of the site not to be indexed.
Apparently the web developer that created the new website had added the robots.txt file and this resulted in a loss of indexed pages and a loss of rankings.
The robots.txt file is a simple text file that defines rules for spidering a site. It allows a site owner to specify which robots should crawl the site and which areas of their website should be left unspidered. A robots.txt file should only used to disallow areas of the website that you do not want spidered. An example might be:
User-agent: *
Disallow: /privatedir/
Disallow: /cgi-bin/
The above robots.txt file would instruct all bots and crawlers to not-spider the directories “privatedir” and “cgi-bin”. The remainder of the website should be crawled and indexed.
I do find it interesting that Bing did end up indexing the website even though their Bot was advised not to. Don’t get me wrong, the site owner is delighted that Bing did index their site. This just shows that not all crawlers will follow the directions of the robots.txt.
If You Like this posting please +1 it!
Filed under: SEO Strategies


Placing a +1 button on your web page or blog gives your visitors a way to make a social recommendations that could be used to influence search results. Every logged in Google user will be able see these votes and if they are in your social circle, they will also see who voted.