Why Google Robots.txt Tester Has Error And It's Not Valid
Please help me to figure out
That is a valid robots.txt - but you've got a UTF-8 BOM (\xef\xbb\xbf) at the beginning of the text file. That's why there's a red dot next to 'User' in the first line. This mark tells browsers and text editors to interpret the file as UTF-8 whereas the robots.txt is expected to use only ASCII characters.
Convert your text file to ASCII and the errors will go away. Or copy everything after the red dot and try pasting it in again.
I tested this on the live version, here's the result translated from byte form:
\xef\xbb\xbfUser-agent: *\r\nDisallow: /en/news/iranology/\r\nDisallow: /en/tours-services/tour-the-soul-of-iran\r\nDisallow: /en/tours-services/tour-a-whistle-stop-tour\r\nDisallow: /en/to
You can clearly see the BOM at the beginning. Browsers and text editors will ignore it but it may mess with a crawlers ability to parse the robots.txt. You can test the live version using this python script:
import urllib.request text = urllib.request.urlopen('http://www.best-iran-trip.com/robots.txt') print(repr(text.read()))
If you're able to install Notepad++, it actually has an encoding menu that lets you save it in any format.
- → How do I call the value from another backed page form and use it on a component in OctoberCms
- → Inline JS callback syntax Vs Internal JS callback syntax
- → Prevent form action from being followed by robots
- → How to remove parameters from the root URL if it does I18n
- → SEO Canonical URL in Greek characters
- → Htaccess negation
- → Wrapping anchor elements with text
- → Adding schema.org to site
- → dynamic php title depends on page id and using of "if (isset"
- → Building sitemap for 2 wordpress install under 1 domain
- → Bigcommerce Repeating url
- → 301 Redirection from no-www to www in wordpress
- → Wrapper inside or outside HTML5 semantics main, which is best practice?