Ad

Robots.txt Cannot Be Located

- 1 answer

I'm currently running a website (http://www.agents-world.com/) served by a node.js from heroku host. The problem is the robots.txt cannot be located whatever I do. I was wondering if heroku was caching something or more generally not deployment that file.

I've made changes to the app so I know Heroku deployed the latest version.

I would expect to see the content of the robots.txt file here http://www.agents-world.com/robots.txt

Any thoughts ?

Ad

Answer

It looks like you have no route pointing to robots.txt.

You have to serve this specific file, using for example an Express route:

app.get("/robots.txt", function(req, res){
    res.sendFile(path.resolve("PATH_TO_ROBOTS.TXT"));
});
Ad
source: stackoverflow.com
Ad