Bangla Radio


Bookmark and Share

Friday, June 11, 2010

Differences between robot.txt vs .htaccess

The robots.txt file consists of directives to search engine spiders (robots) as to what files and folders you want or do not want to be indexed. However, this will not necessarily prevent spiders from following links into those folders and there are some spiders that do not respect the robots.txt file (all of the major search engines do but there are still quite a few unscrupulous bots to worry about). Additionally, the use of robots.txt directives does not prevent human visitors from accessing those folders and directories if they know they are there (or if they're just hacking their way in via guesses, e.g., looking for index.html or index.php files).

Depending on how you set it up, the .htaccess file, in contrast, actually blocks access to certain files or folders. This applies to both human visitors and bots.


Blogger said...

Submit your website or blog now for appearing in Google and 300+ search engines!

Over 200,000 sites indexed!


Post a Comment

Submit your Web Site to Search Engine

Google PR Check

Check Page Rank of your Web site pages instantly:

This page rank checking tool is powered by Page Rank Checker service

Keyword Based Search Engine Checker

Your domain:
Search on:  
Limit search to:
Keywords to search:

Powered by iWEBTOOL

Backlink Checker Tools

Your domain:

Powered by iWEBTOOL

Search Engine Spider Simulator

Enter URL to Spider

Google Penalty Checker

Google Penalty Checker

Powered by Li'l Engine's Google Penalty Checker