Ad

Select From Dropdown Using Selenium And Python

- 1 answer

With the help of Selenium and Python. I wanted to crawl a webpage which has a nested drop-down menu. I am posting only the nested section below:

<div class="dropDown active" data-dropdown-block="FOOTBALL_COMPSEASON" data-dropdown-default="All Seasons">
    <div class="label" id="dd-FOOTBALL_COMPSEASON">Filter by Season</div> 
    <div class="current" data-dropdown-current="FOOTBALL_COMPSEASON" role="button" tabindex="0" aria-expanded="false" aria-labelledby="dd-FOOTBALL_COMPSEASON" data-listen-keypress="true" data-listen-click="true">
        2018/19
    </div>
    <ul class="dropdownList" data-dropdown-list="FOOTBALL_COMPSEASON" role="listbox" aria-labelledby="dd-FOOTBALL_COMPSEASON" data-listen-keypress="true" data-listen-click="true">
        <li role="option" tabindex="0" data-option-name="All Seasons" data-option-id="-1" data-option-index="-1">
             All Seasons
        </li> 
        <li role="option" tabindex="0" data-option-name="2018/19" data-option-id="210" data-option-index="0">
            2018/19
         </li>
         <li role="option" tabindex="0" data-option-name="2017/18" data-option-id="79" data-option-index="1">
              2017/18
         </li>
         <li role="option" tabindex="0" data-option-name="2016/17" data-option-id="54" data-option-index="2">
             2016/17
         </li>
    </ul>
</div>

Here is the screenshot of how it looks:

So, I wanted to make the crawler click the drop-down and select 2017/18.

I first tried this:

driver.get(_url)
select_element = driver.find_elements_by_class_name("dropdownList")[1]

As the class dropdownList is used multiple times in the HTML and my desired element is in the second position, i.e. <ul class="dropdownList".... is the second time the class dropdown is used, so I used [1] to get the second child.

But then I get this error:

File "shots_2017_18.py", line 15, in shots_2017_18 select_element = driver.find_elements_by_class_name("dropdownList")1 IndexError: list index out of range

What should I change or do so that the crawler can select the 2017/18 item from the dropdown list and can crawl?

Ad

Answer

If you are able to click on drop down , using python and selenium. Then you can try this code :

UPDATE :

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC 
from selenium.webdriver.common.action_chains import ActionChains
import time


driver   = webdriver.Chrome(executable_path = r'C:/Users/user***/Downloads/chromedriver_win32/chromedriver.exe')
driver.maximize_window()

wait = WebDriverWait(driver,40)

driver.get("https://www.premierleague.com/stats/top/players/goals")  

wait.until(EC.visibility_of_element_located((By.ID, 'dd-FOOTBALL_COMPSEASON')))

time.sleep(5)
drop_down_click = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.current[data-dropdown-current='FOOTBALL_COMPSEASON']")))
drop_down_click.click()

options = driver.find_elements_by_css_selector("ul[data-dropdown-list='FOOTBALL_COMPSEASON'] li")

for option in options:
  if "2017/18" in option.text.strip():
    option.click()  

UPDATE1 :

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC 
from selenium.webdriver.common.action_chains import ActionChains
import time

driver   = webdriver.Chrome(executable_path = r'C:/Users/user***/Downloads/chromedriver_win32/chromedriver.exe')
driver.maximize_window()

wait = WebDriverWait(driver,40)

driver.get("https://www.premierleague.com/stats/top/players/total_scoring_att")


cookie_button = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.btn-primary.cookies-notice-accept")))
ActionChains(driver).move_to_element(cookie_button)
driver.execute_script('arguments[0].click();', cookie_button)
wait.until(EC.visibility_of_element_located((By.ID, 'dd-FOOTBALL_COMPSEASON')))

time.sleep(5)
drop_down_click = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.current[data-dropdown-current='FOOTBALL_COMPSEASON']")))
drop_down_click.click()

options = driver.find_elements_by_css_selector("ul[data-dropdown-list='FOOTBALL_COMPSEASON'] li")

for option in options:
  if "2017/18" in option.text.strip():
    option.click()  

Explanation :

An explicit wait is code you define to wait for a certain condition to occur before proceeding further in the code. The worst case of this is Thread.sleep(), which sets the condition to an exact time period to wait. There are some convenience methods provided that help you write code that will wait only as long as required. WebDriverWait in combination with ExpectedCondition is one way this can be accomplished.

More about explicit wait, can be found here

Ad
source: stackoverflow.com
Ad