Ad
Robots.txt: How To Disallow All Subdirectories But Allow Parent Folder
I'd like to disallow all subdirectories in my folder /search but allow indexing the search folder itself (I have content on /search).
Testing this does not work:
User-Agent: *
Allow: /search/
Disallow: /search/*
Ad
Answer
Your code appears correct. Try with a slight adjustment to Allow:
User-Agent: *
Disallow: /search/*
Allow: /search/$
Ad
source: stackoverflow.com
Related Questions
- → How do I call the value from another backed page form and use it on a component in OctoberCms
- → Inline JS callback syntax Vs Internal JS callback syntax
- → Prevent form action from being followed by robots
- → How to remove parameters from the root URL if it does I18n
- → SEO Canonical URL in Greek characters
- → Htaccess negation
- → Wrapping anchor elements with text
- → Adding schema.org to site
- → dynamic php title depends on page id and using of "if (isset"
- → Building sitemap for 2 wordpress install under 1 domain
- → Bigcommerce Repeating url
- → 301 Redirection from no-www to www in wordpress
- → Wrapper inside or outside HTML5 semantics main, which is best practice?
Ad