Ad

How To Prevent A URL While Indexing In To Google?

- 1 answer

I have a development server where i have 100 websites(folders) placed but am pointing each and every folder with another domain, but still I can access the folder using server ip or domain right? How can i stop indexing the url which is opening by the server's ip or domain?

Eg:

Server is: myserver.com Website(folder): mywebsite

I can access this like myserver.com/mywebsite

i have another domain called newwebsite.com and pointing to mywebsite folder which is present in myserver.com then I can access same folder using newwebsite.com, I need to stop indexing myserver.com/mywebsite.. Can any one help me in this? Thanks in advance.

Ad

Answer

In every website you've added to the google repository you can add a file called Robots.txt. This file includes all your rules about:

  1. Which pages and URLs will be indexed and followed.
  2. Which URLs denies the crawler access as hidden points.
  3. And all other rules you can read about in this documentation.
documentation.

For example you can block your favorite URLs in each domain(or subdomain) like this:

User-agent: *
Disallow: /~joe/junk.html
Disallow: /~joe/foo.html

Don't remember the robots.txt file must be placed in root directory of the each website to be detected by crawlers. Regards

Ad
source: stackoverflow.com
Ad