I am trying to pull data from following page:
https://www.lifeinscouncil.org/industry%20information/ListOfFundNAVs
I have to select the name of the company and select a date from the calendar and click get data button. I am trying to achieve this using Selenium Web Driver using Chrome in Python, I am stuck how do i pass the date parameter to the page. it seems the page is postback after selection of date from the calendar. Date needs to be selected from the calendar else the data is not returned by the webpage. I have tried using requests Post method as well but am not able to get the NAV data.
I need to iterate this for a period of 5 years on daily (Trading Days) basis.
PS: I am bad at understanding DOM elements and have basic knowledge of Python and coding. by profession I am a data analyst.
Thanks in Advance. Kiran Jain
edit: adding current code below:
from selenium import webdriver
url='https://www.lifeinscouncil.org/industry%20information/ListOfFundNAVs'
opt = webdriver.ChromeOptions()
prefs = {"profile.managed_default_content_settings.images": 2}
opt.add_argument("--start-maximized")
# opt.add_argument("--headless")
opt.add_argument("--disable-notifications")
opt.add_experimental_option("prefs", prefs)
driver = webdriver.Chrome(options=opt)
driver.get('https://www.lifeinscouncil.org/industry%20information/ListOfFundNAVs');
insurer = driver.find_element_by_id("MainContent_drpselectinscompany")
nav_date=driver.find_element_by_id('MainContent_txtdateselect')
get_data_btn=driver.find_element_by_id('MainContent_btngetdetails')
options=insurer.find_elements_by_tag_name("option")
data=[]
row={'FirmName','SFIN','fundName','NAVDate','NAV'}
for option in options:
print('here')
print(option.get_attribute("value") + ' ' + option.text)
if(option.text!='--Select Insurer--'):
option.click()
driver.find_element_by_id("MainContent_imgbtncalender").click()#Calender Icon
driver.find_element_by_link_text("June").click()#Date
driver.find_element_by_link_text("25").click()#Date
get_data_btn=driver.find_element_by_id('MainContent_btngetdetails') #this is put here again because on clicking the date, the page is reloaded
get_data_btn.click()
print('clicked')
driver.quit()
See Question&Answers more detail:os