About the robots.txt file

[Halfdeck]

"<META NAME="ROBOTS" CONTENT="ALL">"

is useless and unnecessary.

-----------------------------

Here's the difference between META ROBOTS tag and robots.txt.

Robots.txt tells Google not to crawl a URL. "Crawl" meaning don't retrieve content from the page.

META ROBOTS noindex tells Google don't show this page in Google's index. For that META tag to do its magic, a URL must be crawled (otherwise Google can't see the META tag).

As far is Google is concerned, there's something else to worry about. URLs blocked by robots.txt disallow accumulate PageRank. If you're having indexing issues, robots.txt is not a great way to handle it. META ROBOTS=noindex also leads to blocked URLs accumulating PageRank. On the other hand, META RObOTS=noindex,nofollow may be treated by Google like a dangling page.