How To Prevent A URL While Indexing In To Google?
I have a development server where i have 100 websites(folders) placed but am pointing each and every folder with another domain, but still I can access the folder using server ip or domain right? How can i stop indexing the url which is opening by the server's ip or domain?
Eg:
Server is: myserver.com Website(folder): mywebsite
I can access this like myserver.com/mywebsite
i have another domain called newwebsite.com and pointing to mywebsite folder which is present in myserver.com then I can access same folder using newwebsite.com, I need to stop indexing myserver.com/mywebsite.. Can any one help me in this? Thanks in advance.
Answer
In every website you've added to the google repository you can add a file called Robots.txt. This file includes all your rules about:
- Which pages and URLs will be indexed and followed.
- Which URLs denies the crawler access as hidden points.
- And all other rules you can read about in this documentation.
For example you can block your favorite URLs in each domain(or subdomain) like this:
User-agent: *
Disallow: /~joe/junk.html
Disallow: /~joe/foo.html
Don't remember the robots.txt file must be placed in root directory of the each website to be detected by crawlers. Regards
Related Questions
- → reindex an object so keys are sequential
- → Javascript loop indexing issue
- → Google indexing uses content instead of description
- → angular.js how to tell google to index my site?
- → No hreflang return tag detected by Google
- → Error 404 on Google Webmaster After Changing the CMS
- → Serve different page to google bot crawler
- → robots.txt: How to disallow all subdirectories but allow parent folder
- → How to prevent a URL while indexing in to google?
- → AngularJS and Google SEO Indexing
- → Google indexation
- → What is Google reaction to indexing redirect URLs?
- → Preventing search engines from indexing all posts