Ad

Scrape A JS Rendered Site Without Chrome GUI?

I am trying to scrape a js rendered site using selenium and BeautifulSoup. The code works fine but I need to run it on a server which don't have any chrome. What shall I change in the code that it works without a GUI?

Below is the current code:

from bs4 import BeautifulSoup
import selenium.webdriver as webdriver
import json
from selenium.webdriver.common.keys import Keys

url = 'https://www.bigbasket.com/pc/fruits-vegetables/fresh-vegetables/?nc=nb'

chromepath = "/Users/Nitin/Desktop/Milkbasket/Scraping/chromedriver"
driver = driver = webdriver.Chrome(chromepath)
driver.get(url)

#rest of code for fetching prices
Ad

Answer

I would recommend you ditch the Selenium approach, and work on getting the information you need using using the built in urllib libraries, or if possible the requests library. The information for all of the products can be obtained from returned JSON data. For example:

import requests
import re

params = {
    "type" : "pc",
    "slug" : "fresh-vegetables",
    "tab_type" : '["all"]',
    "sorted_on" : "popularity",
    "listtype" : "pc",
}

session = requests.Session()

for page in range(1, 10):
    params['page'] = page
    req_vegetables = session.get("https://www.bigbasket.com/product/get-products", params=params)
    json_vegetables = req_vegetables.json()

    print(f'Page {page}')

    for product in json_vegetables['tab_info']['product_map']['all']['prods']:
        print(f"  {product['p_desc']} - {product['sp']} - {product['mrp']}")

This would give you the following output:

Page 1
  Onion - 21.00 - 26.25
  Potato - 27.00 - 33.75
  Tomato - Hybrid - 40.00 - 50.00
  Ladies Finger - 10.00 - 12.50
  Cauliflower - 35.00 - 43.75
  Palak - 30.00 - 37.50
  Potato Onion Tomato 1 kg Each - 88.00 - 110.00
  Carrot - Local - 59.00 - 73.75
  Capsicum - Green - 89.00 - 111.25
  Tomato - Local - 47.00 - 58.75
  Mushrooms - Button - 49.00 - 61.25
  Cucumber - 25.00 - 31.25
  Broccoli - 18.40 - 23.00
  Bottle Gourd - 17.00 - 21.25
  Cabbage - 32.00 - 40.00
  Cucumber - English - 23.00 - 28.75
  Tomato - Local, Organically Grown - 29.00 - 36.25
  Brinjal - Bottle Shape - 72.00 - 90.00
  Onion - Organically Grown - 23.00 - 28.75
  Methi - 19.00 - 23.75
Page 2
  Bitter Gourd / Karela - 59.20 - 74.00
  Beetroot - 40.00 - 50.00
  Fresho Palak - Without Root 250 Gm + Amul Malai Paneer 200 Gm - 94.20 - 102.00
  Capsicum - Red - 299.00 - 373.75 
  ... etc
Ad
source: stackoverflow.com
Ad