Ad

Do I Absolutely Need A Robots.txt File For My Public-facing Web Sites?

- 1 answer

I'm reviewing a couple of my web sites to make sure my SEO bases are covered. There are no private pages on the site in question, and we want all pages to be indexed. Does including a robots.txt file anyway, even if it isn't needed, make my site look more reputable and important to Google?

Ad

Answer

I would just include a blank one regardless because without it your server is still spitting out a 404 message to the requester. Which is probably bigger then a blank robots.txt file.

Ad
source: stackoverflow.com
Ad