robots.txt

Robots checker

http://www.robotstxt.org/wc/robots.html

http://www.rietta.com/robogen/

http://www.lilengine.com/search-engine-promotion/robots.txt/

User-Agent: Slurp (Inktomi's spider)
Disallow: internet-marketing-gg.html
Disallow: internet-marketing-al.html
Disallow: advertising-secrets-gg.html
Disallow: advertising-secrets-al.html

# All robots will spider the domain
User-agent: *
Disallow:

Good to know:

• Use: robots.txt NOT Robots.txt as filename.

Robots.txt | mod_rewrite | .htaccess | Meta tags