Jump to content
Now you can advertise your business on Top Gold Forum.
Sign in to follow this  
shoppingswag

What is the limit of a robot.txt file?

Recommended Posts

A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb. In terms of limitations (not sure if you meant that as well), there's the fact that therobots.txt file is merely a optional directive, not a mandate.

Share this post


Link to post
Share on other sites

There is a limit to robots.txt file in terms of file size 500 KB as Vipin points out. A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
Sign in to follow this  

×
×
  • Create New...