Googlebot Questions

Ad

Googlebot not respecting Robots.txt

For some reason when i check on google webmaster tool's "analyze robots.txt" to see which urls are blocked by our robots.txt file, it's not what
Ad

Usage of 'Allow' in robots.txt

Recently i saw a site's robots.txt as follows: user-agent: * allow: /login allow: /register i could find only allow entries

Serve different page to google bot crawler

I have an spa with a lot of images in it. i want to expose those images to search engines. so i want to create "special" pages that will only be seen by the bot. the pages
Ad

Blog Categories

Ad