Robots.txt, Php.ini, Connect_to_database.php, .htaccess
I cannot seem to find an answer anywhere as to whether or not I should disallow configuration files like
/php.ini or hidden files like
/.htaccess? And what about stuff like
I have read on ProWebmasters and elsewhere on SO that we should not
disallow: /*.js$ or
/*.css$, but almost all of those answers are almost a decade old. Additionally, the purpose of
robots.txt is to determine what is indexed, not what is crawled, is it not? I mean, we would not want crawlers trying to index our
Even Google's own document regarding
robots.txt do not seem to cover this stuff. Is anybody aware of informative resources somewhere on the web relating to this stuff?
The files necessary to render the entire page in the web browsers and search spiders (ex. Google Bot) should be available in the robots.txt file. Allow files css, js, images jpg, jpeg, png, fonts.
Files loaded by the PHP (include() and require(), ex. connect_to_database.php) and configuration (php.ini) should be inaccessible to the public and blocked in the .htaccess file.
- → .htaccess - not routing to public folder
- → Laravel 4.2 application occurs unnecessary redirection after uploading live server
- → Htaccess negation
- → Building sitemap for 2 wordpress install under 1 domain
- → htaccess not working properly for sub url having more slashes
- → How to give access execute specific php file in server?
- → 301 Redirection from no-www to www in wordpress
- → Cyclical redirection at the start laravel 5 project
- → Can't route in Laravel framework
- → codeigniter seo url with htaccess
- → PHP Seo Friendly URL
- → Convert URL to SEO Friendly
- → IP Canonicalization (WordPress)