So I've asked this question earlier (Unable to get selenium (python) to download a csv file which doesnt have a link but only appears after i click the download button) and managed to get this far. I finally realised that the code wasn't working because it was in headless mode.
In my earlier post I also mentioned that I'd try to use requests to get the file but there doesn’t seem to be a link for the csv file in this case.
The code basically goes here https://www.macrotrends.net/1476/copper-prices-historical-chart-data, clicks the All Years button, then clicks the Download Historical Data button. and selenium tries to save the file after it clicks.
But like i said it only downloads the file when i'm in normal mode it doesn't seem to work in headless. Is there a reason for this? Is there a way to make it work in headless mode? I've been looking around but i cant find an answer.
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
start_time = time.time()
options = Options()
#options.add_argument("--headless")
options.add_argument("--disable-gpu")
options.add_argument("--disable-extensions")
options.add_experimental_option("prefs", {
"download.default_directory": r"'/home/Documents/testing/macrotrends'",
"download.prompt_for_download": False,
"download.directory_upgrade": True,
"safebrowsing.enabled": False
})
driver = webdriver.Chrome(executable_path=r'/home/chromedriver/chromedriver',options=options)
driver.get('https://www.macrotrends.net/1476/copper-prices-historical-chart-data')
time.sleep(5)
iframe = driver.find_element_by_xpath("//iframe[@id='chart_iframe']")
driver.switch_to.frame(iframe)
xpath = "//a[text()='All Years']"
driver.find_element_by_xpath(xpath).click()
xpath = "//button[@id='dataDownload']"
driver.find_element_by_xpath(xpath).click()
time.sleep(10)
driver.close()
print("--- %s seconds ---" % (time.time() - start_time))
Downloads are disabled by default when in headless mode. you can allow them by executing a developer tools command like this:
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options
options = Options()
options.headless = True
driver = Chrome(options=options)
params = {'behavior': 'allow', 'downloadPath': '/path/for/download'}
driver.execute_cdp_cmd('Page.setDownloadBehavior', params)
# downloads are now enabled for this driver instance