Scrapy FormRequest
I'm having trouble with scrapy FormRequest. I am trying to get all reviews from this page (infinite scrolling) : https://www.temporel-voyance.com/voyance/planning/consultations-voyants-en-privee/angele/1041
When I scroll, I can see a post request with data form : {xyz":"3"}
But when I try it with scrapy command line, I am not able to get the correct response.
from scrapy import FormRequest
fetch("https://www.temporel-voyance.com/voyance/planning/consultations-voyants-en-privee/angele/1041")
req = FormRequest(response.url, formdata={"xyz":"3"})
fetch(req)
I had a look to this page : https://blog.scrapinghub.com/2016/06/22/scrapy-tips-from-the-pros-june-2016 but I am still stuck.
Can someone help me ?
Thank you very much ! Mylha
Answer
If you take a better look at the request being made, you'll see that it's actually to a different url, https://www.temporel-voyance.com/voyants/temoignages?voyant_id=1041
If you make your form request to that url, you get the updated data:
>>> req = scrapy.FormRequest('https://www.temporel-voyance.com/voyants/temoignages?voyant_id=1041', formdata={'xyz': '3'})
>>> fetch(req)
2018-10-20 13:43:42 [scrapy.core.engine] DEBUG: Crawled (200) <POST https://www.temporel-voyance.com/voyants/temoignages?voyant_id=1041> (referer: None)
>>> response.css('.name_user b::text').getall()
['Chou', 'Jérôme', 'Sonia']
Related Questions
- → What are the pluses/minuses of different ways to configure GPIOs on the Beaglebone Black?
- → Django, code inside <script> tag doesn't work in a template
- → React - Django webpack config with dynamic 'output'
- → GAE Python app - Does URL matter for SEO?
- → Put a Rendered Django Template in Json along with some other items
- → session disappears when request is sent from fetch
- → Python Shopify API output formatted datetime string in django template
- → Can't turn off Javascript using Selenium
- → WebDriver click() vs JavaScript click()
- → Shopify app: adding a new shipping address via webhook
- → Shopify + Python library: how to create new shipping address
- → shopify python api: how do add new assets to published theme?
- → Access 'HTTP_X_SHOPIFY_SHOP_API_CALL_LIMIT' with Python Shopify Module