Ad

Error When Attempting To Crawl Site With Python

- 1 answer

I am trying to get prices from this website for a few monitors. Here is my Code:

    def noteBooksBillgerParser(url):
        headers = {
            'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36'}
        page = requests.get(url, headers=headers)
        while True:
            print ("test")
            sleep(3)
            try:
                doc = html.fromstring(page.content)

                XPATH_PRICE = '//div[@id="product_detail_price"]//content()'
                RAW_PRICE = doc.xpath(XPATH_PRICE)
                PRICE = ' '.join(''.join(RAW_PRICE).split()).strip() if RAW_PRICE else None

                data = {
                    'PRICE': PRICE,
                    'URL': url,
                }

                return data
            except Exception as e:
                print
                e


    def ReadIDs():
        # AsinList = csv.DictReader(open(os.path.join(os.path.dirname(__file__),"Asinfeed.csv")))
        IDList = ['vp248qg',
                  'vz239he',
                  'vs247hr+164581'
                  ]
        extracted_data = []
        for i in IDList:
            url = "https://www.notebooksbilliger.de/asus+" + i
            print("Processing: ", url)
            extracted_data.append(noteBooksBillgerParser(url))
            sleep(2)
        f = open('notebooksbilliger.json', 'w')
        json.dump(extracted_data, f, inde

nt=4)

I am pretty sure most of the code is working but i am not sure how to get the Price into my XPATH_PRICE variable. I think there could be a mistake.

Ad

Answer

It's not exactly getting "Price into my XPATH_PRICE variable", but getting it into your ROW_PRICE variable. If your:

RAW_PRICE = doc.xpath('//div[@id="product_detail_price"]')[0].values()[4]

your output will be (picking, at random IDList = vz239he):

156.99

The rest should then be processed as intended.

Ad
source: stackoverflow.com
Ad