Ad

Finding Right Selector For Pagination With Scrapy

I'm trying to extract data from this forum:

https://schwangerschaft.gofeminin.de/forum/all

I get the data from the first page. I use the css selector 'li.selected > a::attr(href)' Unfortunately I can not get all other data from other pages.

What is the right path for xpath or css selectors for pagination?

Python:

import scrapy

class ForumSpider(scrapy.Spider):
    name = "pregnancy"

    def start_requests(self):
        url = 'https://schwangerschaft.gofeminin.de/forum/all'
        yield scrapy.Request(url, self.parse)


    def parse(self, response):
        for thread in response.css('div.af-thread-item'):
            yield{
                'threadTitle': thread.css('span.thread-title::text').extract_first(),
                'username': thread.css('div.user-name::text').extract_first()
            }
        next_page = response.css('li.selected > a::attr(href)').extract_first()
        if next_page is not None:
            yield scrapy.Request(response.urljoin(next_page))

HTML:

<nav class="af-pagination " role="navigation"><ul><li class="selected">
<a target="_blank" rel="nofollow noreferrer" href="https://schwangerschaft.gofeminin.de/forum/all">1</a></li><li>
<a target="_blank" rel="nofollow noreferrer" href="https://schwangerschaft.gofeminin.de/forum/all/p2">2</a></li><li>
<a target="_blank" rel="nofollow noreferrer" href="https://schwangerschaft.gofeminin.de/forum/all/p3">3</a></li><li>
<a target="_blank" rel="nofollow noreferrer" href="https://schwangerschaft.gofeminin.de/forum/all/p4">4</a></li><li>
<a target="_blank" rel="nofollow noreferrer" href="https://schwangerschaft.gofeminin.de/forum/all/p5">5</a></li><li>
<a target="_blank" rel="nofollow noreferrer" href="https://schwangerschaft.gofeminin.de/forum/all/p6">6</a></li><li>
<a target="_blank" rel="nofollow noreferrer" href="https://schwangerschaft.gofeminin.de/forum/all/p7">7</a></li><li>
<a target="_blank" rel="nofollow noreferrer" href="https://schwangerschaft.gofeminin.de/forum/all/p8">8</a></li><li>
...

link of next page: https://schwangerschaft.gofeminin.de/forum/all/p2

Ad

Answer

Try response.css('link[rel=next]::attr(href)').get(), this should work.

Ad
source: stackoverflow.com
Ad