Ad

Robots.txt, Php.ini, Connect_to_database.php, .htaccess

I cannot seem to find an answer anywhere as to whether or not I should disallow configuration files like /php.ini or hidden files like /.htaccess? And what about stuff like /includes or /includes/connect_to_database.php?

I have read on ProWebmasters and elsewhere on SO that we should not disallow: /*.js$ or /*.css$, but almost all of those answers are almost a decade old. Additionally, the purpose of robots.txt is to determine what is indexed, not what is crawled, is it not? I mean, we would not want crawlers trying to index our css and js files.

Even Google's own document regarding robots.txt do not seem to cover this stuff. Is anybody aware of informative resources somewhere on the web relating to this stuff?

Thanks

Ad

Answer

The files necessary to render the entire page in the web browsers and search spiders (ex. Google Bot) should be available in the robots.txt file. Allow files css, js, images jpg, jpeg, png, fonts.

Files loaded by the PHP (include() and require(), ex. connect_to_database.php) and configuration (php.ini) should be inaccessible to the public and blocked in the .htaccess file.

Ad
source: stackoverflow.com
Ad