Jump to content

Recommended Posts

The robots. txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. Let's say a search engine is about to visit a site.

Share this post


Link to post
Share on other sites

Hi Friends,

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.
 

Share this post


Link to post
Share on other sites

The robotstxt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots  which pages on your site to crawl. It also tells web robots which pages not to crawl. 

Share this post


Link to post
Share on other sites

The robots. txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. Let's say a search engine is about to visit a site.

Share this post


Link to post
Share on other sites

Robots.txt is one way of telling the Search Engine Bots about the web pages on your website which you do not want them to visit.

Robots.txt is useful for preventing the indexation of the parts of any online content that website owners do not want to display.

Share this post


Link to post
Share on other sites

Robots.txt is a text file webmasters create to instruct web robots ( search engine robots ) which pages on your website to crawl or not to crawl.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...