Best Robots.txt Practice To Hide Secret Folder
I have a secret folder in my website and I don't want search engines to know about it. I didn't put the folder name in the Disallow rule of robots.txt because writing this folder name in robots.txt means telling my visitors about that secret folder.
My question is, will search engines be able to know about this folder / crawl it even if I don't have any links published to this folder?
The only truly reliable way to hide a directory from everyone is to put it behind a password. If you absolutely cannot put it behind a password, one band-aid solution is to name the folder something like:
and then block just the first part of the name, like this:
This will effectively block the directory without revealing its full name. It will prevent any crawler that obeys robots.txt from crawling the directory, but it won't make the directory easy for hostile crawlers to find. Just don't mistake this for actual security. This will keep the major search engines out. There are no guarantees beyond that. Again, the only truly reliable way to keep everyone out of a secret directory is to put the directory behind a password.
- → How do I call the value from another backed page form and use it on a component in OctoberCms
- → Inline JS callback syntax Vs Internal JS callback syntax
- → Prevent form action from being followed by robots
- → How to remove parameters from the root URL if it does I18n
- → SEO Canonical URL in Greek characters
- → Htaccess negation
- → Wrapping anchor elements with text
- → Adding schema.org to site
- → dynamic php title depends on page id and using of "if (isset"
- → Building sitemap for 2 wordpress install under 1 domain
- → Bigcommerce Repeating url
- → 301 Redirection from no-www to www in wordpress
- → Wrapper inside or outside HTML5 semantics main, which is best practice?