|
Robots.txt Rules & SEO
Do you need a Robots.txt file? YES! Will it help me with SEO? Yes and no
When a search engine 'crawls' your site, it looks for the Robots.txt file. This file should be on the main level of your directly - where your index page resides.
What the ROBOTS.txt file does is list the files and folders that you do NOT want the search engines to crawl. This could be a private directory or just a couple files. <continued below>
|
 |
 |
 |
 |
|
 |
 |
 |
|
Some uses would be if you have a 'testing' folder on your server where you try out new designs - if the content is the same as on your live site, you don't want the search engines to crawl through there and index duplicate pages. You would want to tell the search engines not to spider those pages.
The Robots.txt language looks like this:
User-agent: *
Disallow: file.htm
Disallow: /folder
Now, there's more to it than just that:
Recently, the major search engines decided to adopt the same language in the Robots.txt file that will point the search engines to the location of your site map. This is a good thing, and you should update your Robots.txt file now, if you haven't already done so.
The line in the Robots.txt file should look like this:
Sitemap: http://www.DOMAIN.com/YOURSITEMAP.htm
If you don't have a sitemap, get one and read why here. :: Want to learn more about Robots.txt and what it means to you? Click here.
Home :: Getting Started with SEO :: Keyword Basics :: Meta Tag Basics :: Robots.txt
