Jump to content
Advertise your business on Top Gold Forum today for as low as $100 per month. Contact us.
Sign in to follow this  
online mcafeeactivate

Is it necessary to use Robots.txt file?

Recommended Posts

Hello friends- Is it necessary to use Robots.txt file? It can be affect on website ranking in search engine.

Share this post


Link to post
Share on other sites

A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be ‘crawled’. For instance, if you didn’t want your images to be listed by Google and other search engines, you’d block them using your robots.txt file.

Share this post


Link to post
Share on other sites

Yes robots.txt is an integral part of your website. It is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.


Cheap VPS | $1 VPS Hosting

Cheap Dedicated Servers | Server starting at $29/MO

Share this post


Link to post
Share on other sites

Yes, absolutely. Because Robosts.txt generator tells the search engines which pages to access and index on your website and which pages not to. But it still depends on the type of website.

Share this post


Link to post
Share on other sites

Is it necessary to use Robots.txt file Because Robosts.txt generator tells the search engines which pages to access and index on your website and which pages not to. But it still depends on the type of website.

Share this post


Link to post
Share on other sites

Yes, your site will work fine. However you will not be able to block search engines from indexing your whole accessible site. This may include login pages, direct URLs to uploaded images, and other undesirable search results.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this  

×