Ad

Best Robots.txt Practice To Hide Secret Folder

- 1 answer

I have a secret folder in my website and I don't want search engines to know about it. I didn't put the folder name in the Disallow rule of robots.txt because writing this folder name in robots.txt means telling my visitors about that secret folder.

My question is, will search engines be able to know about this folder / crawl it even if I don't have any links published to this folder?

Ad

Answer

The only truly reliable way to hide a directory from everyone is to put it behind a password. If you absolutely cannot put it behind a password, one band-aid solution is to name the folder something like:

http://example.com/secret-aic7bsufbi2jbqnduq2g7yf/

and then block just the first part of the name, like this:

Disallow: /secret-

This will effectively block the directory without revealing its full name. It will prevent any crawler that obeys robots.txt from crawling the directory, but it won't make the directory easy for hostile crawlers to find. Just don't mistake this for actual security. This will keep the major search engines out. There are no guarantees beyond that. Again, the only truly reliable way to keep everyone out of a secret directory is to put the directory behind a password.

Ad
source: stackoverflow.com
Ad