Ad

Use Of Robot.txt

- 1 answer

Why will I want a Robot.txt file on my website. I know how to use it, I want to know why it is used?

Ad

Answer

Surely you can do a bit more research on Google to find the answer but as an example:

Imagine Google finds your new site published online and scans all your pages to publish in its search index. You may not want google to publish every page such as a sub page because you don’t want users to access it directly from google but rather through a main entry page.

Ad
source: stackoverflow.com
Ad