How can I use Python Mechanize to fill out a web form that requires Javascript?

Question

I was trying to fill a form using mechanize. But the problem is the webpage needs javascript. So whenever I try to make an access to the page, the page redirects to an error page saying javascript needed. Is there a way to enable javascript when using mechanize browser?

Here is the code

import mechanize
import ssl

ssl._create_default_https_context = ssl._create_unverified_context
br = mechanize.Browser()
br.set_handle_robots(False)
br.open("https://192.168.10.3/connect/PortalMain")
for f in br.forms():
    print f

Also when I tried to extract the webpage using BeautifulSoup that 'works fine on my browser' I got the same problem. It redirects to a new page. (I tried disabling javascript on my browser and got the page which beautiful soup was showing me.)

Here is the code of BeautifulSoup if it helps

import ssl
import urllib2
from bs4 import BeautifulSoup

ssl._create_default_https_context = ssl._create_unverified_context
page = urllib2.urlopen("https://192.168.10.3/connect/PortalMain")
soup = BeautifulSoup(page,'html.parser')
print soup

Answer 1

You could just go ahead and use Selenium instead:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

usernameStr = 'putYourUsernameHere'
passwordStr = 'putYourPasswordHere'

browser = webdriver.Chrome()
browser.get('https://192.168.10.3/connect/PortalMain')

# fill in username and hit the next button (replace selectors!)
username = browser.find_element_by_id('Username')
username.send_keys(usernameStr)
password = browser.find_element_by_id('Password')
password.send_keys(passwordStr)
loginButton = browser.find_element_by_id('login')
loginButton.click()

This will use the Chrome web driver to open your browser and login, you can switch it to use any other driver Selenium supports e.g. Firefox.

Source: https://www.hongkiat.com/blog/automate-create-login-bot-python-selenium/

Remember you might need to make adjustments if the site is using a self-signed certificate.