How To Prevent A URL While Indexing In To Google?

- 1 answer

I have a development server where i have 100 websites(folders) placed but am pointing each and every folder with another domain, but still I can access the folder using server ip or domain right? How can i stop indexing the url which is opening by the server's ip or domain?


Server is: Website(folder): mywebsite

I can access this like

i have another domain called and pointing to mywebsite folder which is present in then I can access same folder using, I need to stop indexing Can any one help me in this? Thanks in advance.



In every website you've added to the google repository you can add a file called Robots.txt. This file includes all your rules about:

  1. Which pages and URLs will be indexed and followed.
  2. Which URLs denies the crawler access as hidden points.
  3. And all other rules you can read about in this documentation.

For example you can block your favorite URLs in each domain(or subdomain) like this:

User-agent: *
Disallow: /~joe/junk.html
Disallow: /~joe/foo.html

Don't remember the robots.txt file must be placed in root directory of the each website to be detected by crawlers. Regards