Ad

SEO Chaos From Changing Robots.txt File In Wordpress Site

- 1 answer

I recently edited the robots.txt file in my site using a wordpress plugin. However, since i did this, google seems to have removed my site from their search page. I'd appreciate if I could get an expert opinion on why this is so, and a possible solution. I'd initially done it to increase my search ranking by limiting the pages being accessed by google.

This is my robots.txt file in wordpress:

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Allow: /wp-content/uploads

Sitemap: http://www.instant-wine-cellar.co.uk/wp-content/themes/Wineconcepts/Sitemap.xml
Ad

Answer

This is a good robots.txt directive for WordPress. Add the Allow: /wp-content/uploads if you want uploads to be indexed, but that doesn't make sense, as all your images, pdfs, etc., are included in your posts and pages and are indexed there.

User-agent: *
Allow: /
Disallow: /*?s=
Disallow: /wp-admin/*
Disallow: /wp-content/*
Disallow: /wp-includes/*
Disallow: /wp-content/cache
Disallow: /wp-content/themes/*
Disallow: /trackback
Disallow: /comments
Disallow: /category/
Disallow: */trackback
Disallow: */comments

But the most critcal bit of info is in your page source:

<meta name='robots' content='noindex,nofollow' />

That means you have privacy set in Dashboard/Settings/Privacy and that's blocking all search bots even before they get to robots.txt.

Once you get a good robots.txt file and change the Wordpress privacy setting, so to Google webmaster tools and turn up your crawl rate to have Google hit the site faster.

Ad
source: stackoverflow.com
Ad