Ad

Avoid Page Redirect For Bots

- 1 answer

When showing the website for the first time, I need the user to confirm he is 18 or more (winery website). My client wants a specific page for that (not a popin nor a popup), so using cookies presence, I redirect to that specific page. You can't access any page if you didn't confirmed.

This is a problem for SEO, because bots (like GoogleBot) can't index correctly the website.

How could I avoid that redirection specifically for the bots? Or how to identify the user as a human or a bot?

I'm working with PHP and I've tried to make a list of known bots and compare it to HTTP_USER_AGENT, to prevent the redirection. But I don't think this is a permanent solution (what about if bots change their names, or if news bots are involved?).

$crawlers = array(
        'Googlebot',
        'Baiduspider',
        'ia_archiver',
        'R6_FeedFetcher',
        'NetcraftSurveyAgent',
        'Sogou web spider',
        'bingbot',
        'Yahoo! Slurp',
        'facebookexternalhit',
        'PrintfulBot',
        'msnbot',
        'Twitterbot',
        'UnwindFetchor',
        'urlresolver',
        'Butterfly',
        'TweetmemeBot'
    );
$implCrawlers = implode('|', $crawlers);
$isBot = (strpos($_SERVER['HTTP_USER_AGENT'], $implCrawlers ) !== false) ? true : false;
if (!$isBot) {
...
}
Ad

Answer

You can go for this package https://github.com/JayBizzle/Crawler-Detect they update their repo as crawlers changes their names.

Ad
source: stackoverflow.com
Ad