Jump to content
Sign in to follow this  
tenny7395

What is Robots.txt file

Recommended Posts

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

Share this post


Link to post
Share on other sites

A robots.txt file consists of lines which contain two fields: one line with a user-agent name (search engine crawlers) and one or several lines starting with the directive 

   Disallow: 

Robots.txt has to be created in the UNIX text format. 

Share this post


Link to post
Share on other sites

Robots .txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots .txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users. The REP also includes directives like meta robots, as well as page-, sub-directory-, or site-wide instructions for how search engines should treat links (such as “follow” or “no-follow”).

Share this post


Link to post
Share on other sites

Robots.txt file consists of instructions to the web crawlers while crawling. Instructions are like which links to follow, which pages to not follow etc. This text file is create by webmasters and also this is this file is the part of the robots exclusion protocol (REP).

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×