SEO Chaos From Changing Robots.txt File In Wordpress Site
I recently edited the robots.txt file in my site using a wordpress plugin. However, since i did this, google seems to have removed my site from their search page. I'd appreciate if I could get an expert opinion on why this is so, and a possible solution. I'd initially done it to increase my search ranking by limiting the pages being accessed by google.
This is my robots.txt file in wordpress:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads Sitemap: http://www.instant-wine-cellar.co.uk/wp-content/themes/Wineconcepts/Sitemap.xml
This is a good robots.txt directive for WordPress. Add the
Allow: /wp-content/uploads if you want uploads to be indexed, but that doesn't make sense, as all your images, pdfs, etc., are included in your posts and pages and are indexed there.
User-agent: * Allow: / Disallow: /*?s= Disallow: /wp-admin/* Disallow: /wp-content/* Disallow: /wp-includes/* Disallow: /wp-content/cache Disallow: /wp-content/themes/* Disallow: /trackback Disallow: /comments Disallow: /category/ Disallow: */trackback Disallow: */comments
But the most critcal bit of info is in your page source:
<meta name='robots' content='noindex,nofollow' />
That means you have privacy set in Dashboard/Settings/Privacy and that's blocking all search bots even before they get to robots.txt.
Once you get a good robots.txt file and change the Wordpress privacy setting, so to Google webmaster tools and turn up your crawl rate to have Google hit the site faster.
- → CORS missmatch because of http
- → Building sitemap for 2 wordpress install under 1 domain
- → How to remove empty elements after class?(jQuery)
- → Get width of an element and apply to another relative to the first one?
- → How to remove caption p class from wordpress?
- → 301 Redirection from no-www to www in wordpress
- → Laravel 5 routing using prefix
- → WordPress - Header position Top, Left &Right
- → how to add rel=nofollow to some specific external links in wordpress
- → octobercms install error: database is not empty?
- → How to edit the index page of wordpress theme?
- → How to select a Post Type (Wordpress) to pass a filter in head?
- → What sort of URL structure should be used to display AMP HTML vs vanilla HTML