Search code examples
pythonselenium-webdriverweb-scrapingselenium-chromedriver

how to find id or name using selenium in a div container


I am trying to automate the login to the website, however, I can't for the life of me figure out how to find the elements of the login page. (https://data-exchange.amadeus.com/sfiler/Login.action)

 <form method="post" name="Login" action="Login.action" autocomplete="off">
        <div class="container">
            <input type="hidden" name="origUrl" value="" id="origUrl"/>
            <input type="hidden" name="request_locale" value="en_GB" id="request_locale"/>

            <!-- LOGIN FORM START -->
            <div class="row logo justify-content-md-center">

                <div class="col-12 col-md-12 col-lg-8 pb-4 logo-login">
                    <img alt="Login" src="themes/amadeus-adep/images/logo-login.png"/>
                </div>
            </div>

            <div class="row justify-content-md-center">
                <div class="col-12 col-md-12 col-lg-8 pb-4">
                    <div class="card">
                        <div class="card-body">

                            <div class="row">
                                <div class="col-6">
                                    <div class="form-group">
                                        
                                        
                                            <label for="loginForm_username">Username or email</label>
                                            <input type="text" name="loginForm.username" value="" id="loginForm_username" class="form-control"/>
                                        
                                    </div>
                                </div>
                                <div class="col-6">
                                    <div class="form-group">
                                        <label for="loginForm_password">Password</label>
                                        <input type="password" name="loginForm.password" id="loginForm_password" class="form-control" onkeypress="return submit_on_enter(document.Login,event);"/>
                                        
                                            <div class="d-flex justify-content-end" id="loginChangePassword">
                                                <a class="btn btn-sm btn-link text-right" href="ForgetPassword.action">Lost Password</a>
                                            </div>
                                        
                                    </div>
                                </div>
                            </div>
                            <div class="d-flex justify-content-end">
                                <a id="loginForm_button" class="btn btn-primary text-right" onclick="document.Login.submit();" href="#">
                                    <span class="far fa-sign-in"></span> Login
                                </a>
                                
                            </div>
                        </div>
                    </div>
                </div>
               
            </div>

            <br/>
        </div>

It should be pretty simple to find the element, however, I tried to find the name and id, but it is not showing up...

try:
    user_input = browser.find_element(By.NAME, 'loginForm.username')
    print('Username box found')
except Exception:
    print("Searching for 'NAME tag' took too much time!")
    try:
        user_input = browser.find_element(By.ID, 'loginForm_username')
    except Exception:
        print("Searching for 'ID tag' took too much time!")

I keep getting a timeout error.

Message: no such element: Unable to locate element:

I then tried to check if the items I was searching for was in an iframe, but has no luck doing that either....

frame = browser.find_element(By.CSS_SELECTOR,'div.tool_forms iframe')
browser.switch_to.frame(frame)
try:
    user_input = browser.find_element(By.NAME, 'loginForm.username')
    print('Username box found')
except Exception:
    print("Searching for 'NAME tag' took too much time!")
    try:
        user_input = browser.find_element(By.ID, 'loginForm_username')
    except Exception:
        print("Searching for 'ID tag' took too much time!")

I keep getting the error that the item can't be located.

This is the script I am using

#Amadeus data exchange portal binary download
#
# Installs
# python -m pip install selenium
# python -m pip install webdriver-manager
# python -m pip install fake_useragent
# python -m pip install bs4

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.expected_conditions import visibility_of_element_located
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from fake_useragent import UserAgent
from bs4 import BeautifulSoup
import time, os

#define variables
username = os.environ["AMADEUS_USER"]
password = os.environ["AMADEUS_PASS"]
url = 'https://data-exchange.amadeus.com/sfiler/Login.action'
url_int = 'https://data-exchange.amadeus.com/sfiler/Tree.action'
ua = UserAgent()
userAgent = ua.random
timeout = 900 # 15 mins
loginTitle = "Amadeus Data Exchange Portal - Login"
loggedInTitle = "Amadeus Data Exchange Portal"

#define chrome options
chrome_options = Options()
chrome_options.add_argument(f'user-agent={userAgent}')
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')

#initialize browser
browser = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=chrome_options)
browser.set_page_load_timeout(timeout) # 15 mins
browser.get(url)
browser.implicitly_wait(15)

search_timeout = 60 # 60 seconds
try:
    username_input_box = WebDriverWait(browser, search_timeout).until(EC.presence_of_element_located((By.NAME, "loginForm.username")))
    username_input_box.clear()
    print('Username box found')
except Exception:
    print("loading page took too much time!")
    username_input_box = WebDriverWait(browser, search_timeout).until(EC.presence_of_element_located((By.ID, "loginForm_username")))
    username_input_box.clear()
try:
    password_input_box = WebDriverWait(browser, search_timeout).until(EC.presence_of_element_located((By.ID, "loginForm_password")))
    password_input_box.clear()
except TimeoutError:
    print("loading page took too much time!")

try: 
    login_button = WebDriverWait(browser, search_timeout).until(EC.presence_of_element_located((By.ID, "loginForm_button")))
except TimeoutError:
    print("loading page took too much time!")

#fill login credentials
username_input_box.send_keys(username)
time.sleep(2)    #2 second time gap between filling username and password
password_input_box.send_keys(password)
time.sleep(2)

#hit the login button
login_button.click()

#Check that the login succeeded...
actualTitle = browser.getTitle();
print("The title of the web page is " + str(actualTitle) )
print("Verifying the page title has started")
if actualTitle.match(loggedInTitle):
    print("The page title has been successfully verified")
    print("User logged in successfully..!")
else: 
    print("Page titles do not match?")
    if actualTitle.match(loginTitle):
        print("the login title was found...") 
    browser.close()

#Get all download links
print('')
print('-----------------------------------------------------------------------------')
print('Searching for files to download...')
print('-----------------------------------------------------------------------------')
print('')
browser.get(url_int)
browser.implicitly_wait(30)
soup = BeautifulSoup(browser.page_source, 'html.parser')
for a in soup.find_all('a', href=True):
    print (" Found the URL: " + str(a) )

# automatically close the driver after 30 seconds
time.sleep(30)
browser.close()

Can someone please tell me what I am missing here?

Thanks!


Solution

  • I would use xpath as suggested in the comments. The generic way to use xpath would be:

    driver.find_element_by_xpath()
    
    //*/*[@Id='loginForm_username']
    

    With the '*' syntax it does not matter which element you are trying to find

    see: https://www.w3schools.com/xml/xpath_syntax.asp