Ad

Bing.com Indexes URLs I've Never Submitted

- 1 answer

Sometimes, before launching new web projects, i put the site / app under a subdomain like new.domain.com or beta.domain.com.

These URLs are only meant for my clients. So they don't get submitted at search engines and there aren't any public links to them.

However, I noticed in a few occasions, these subdomains get indexed by Bing anyway. How is this possible ?

Does Bing crawl generic subdomain names like new, old, archive, beta, ... ? Or do URLs sent in mails, get scraped in Office 365 (which my clients use) and get indexed ?

Ad

Answer

This is possible the user has installed a toolbar from that search engine.

The best way to prevent from this, is adding no-index tag for all pages in sub domain and even you can block using robot.txt.

Ad
source: stackoverflow.com
Ad