Robots.txt And Disalowing Absolute Path URL
I am using Heroku pipes. So when I push my application it is pushed to staging app
and if everything is correct I promote that app to prodcution. There is no new build process. It is the same app that was build the first time for staging.
The thing is that this causes a problem with duplicate content. Sites are clones of each other. Exactly the same. I would like to exclude the staging app from Google indexing and search engine.
One way that I thought off was with robots.txt file.
For this to work I should write it like this
User-agent: * Disallow: https://appname.herokuapp.com/
using the absolute path because this file will be on the server in staging and production application and I only wanna remove staging app from Google indexing and not touch the production one.
Is this the right way to do it?
Disallow field can’t take full URL references. Your robots.txt would block URLs like these:
Disallow value always represents the beginning of the URL’s path.
To block all URLs under
https://appname.herokuapp.com/, you would need:
So you have to use different robots.txt files for
If you don’t mind bots crawling
https://appname.herokuapp.com/, you could make use of
noindex instead. But this would also require different behaviour for both sites. An alternative that doesn’t require different behaviour could be to make use of
canonical. This conveys to crawlers which URL is preferred for indexing.
<!-- on https://appname.herokuapp.com/foobar --> <link rel="canonical" target="_blank" rel="nofollow noreferrer" target="_blank" rel="nofollow noreferrer" href="https://appname.com/foobar" />
<!-- on https://appname.com/foobar --> <link rel="canonical" target="_blank" rel="nofollow noreferrer" target="_blank" rel="nofollow noreferrer" href="https://appname.com/foobar" />
- → Angular 2 Universal Heroku Deploy
- → Shopify After APP Installation in Store / Show Iframe to every Client without Login
- → After Variant Save, Update the Price in StoreFront Shopify
- → How to Display Heroku hosted Ruby on Rails Shopify App Page on Shopify Storefront
- → No css/js or images found on heroku deployment octobercms
- → Heroku Shopify App after installation on shop “The page you were looking for doesn't exist.”
- → robots.txt and disalowing absolute path URL
- → How do i make my react web appp to appear on google search?
- → Heroku Server Timeout for Shopify App
- → Hapi server on heroku fails to bind port
- → Shopify App Proxy - How to include my asset files in the request response | React / JS / Heroku
- → shopify_app gem webhooks not working on Heroku environment
- → Display Rails 6 show view as embed shopify page