Mechanical soup - form filling, working with a tricky HTML table
I am trying to use mechanical soup to automatically fill and submit a time sheet for me.
This is what the form looks like:
This is the relevant source code for that section of the page:
<FORM ACTION="bwpkteci.P_UpdateTimeInOut" METHOD="post">
<INPUT TYPE="hidden" NAME="JobsSeqNo" VALUE="208138">
<INPUT TYPE="hidden" NAME="LastDate" VALUE="0">
<INPUT TYPE="hidden" NAME="par_restart" VALUE="Y">
<INPUT TYPE="hidden" NAME="par_update" VALUE="Y">
<INPUT TYPE="hidden" NAME="par_submit" VALUE="Y">
<INPUT TYPE="hidden" NAME="par_recall" VALUE="N">
<INPUT TYPE="hidden" NAME="EarnCode" VALUE="RSA">
<INPUT TYPE="hidden" NAME="DateSelected" VALUE="20-FEB-2018">
<TABLE CLASS="dataentrytable" SUMMARY="This user enters the time of day for the hours worked into this table in order for the system to calculate the hours.">
<TR>
<TD CLASS="delabel" scope="row" >Date:</TD>
<TD CLASS="dedefault">Tuesday, Feb 20,2018</TD>
</TR>
<TR>
<TD CLASS="delabel" scope="row" >Earnings Code:</TD>
<TD CLASS="dedefault">Regular Student Aide</TD>
</TR>
</TABLE>
<TABLE CLASS="dataentrytable" SUMMARY="This is the detail table where the user enters the time of the day for the hours worked in order for the system to calculate the hours.">
<TR>
<TD CLASS="deheader" scope="col" ><LABEL for=shift_input_id><SPAN class="fieldlabeltext">Shift</SPAN></LABEL></TD>
<TD COLSPAN="2" CLASS="deheader" scope="col" ><LABEL for=timein_input_id><SPAN class="fieldlabeltext">Time In</SPAN></LABEL></TD>
<TD COLSPAN="2" CLASS="deheader" scope="col" ><LABEL for=timeout_input_id><SPAN class="fieldlabeltext">Time Out</SPAN></LABEL></TD>
<TD CLASS="deheader" scope="col" >Total Hours</TD>
</TR>
<INPUT TYPE="hidden" NAME="LineNumber" VALUE="1">
<TR>
<TD CLASS="dedefault"><INPUT TYPE="text" NAME="Shift" SIZE="2" MAXLENGTH="1" VALUE="1" ID="shift_input_id"></TD>
<TD CLASS="dedefault"><INPUT TYPE="text" NAME="TimeIn" SIZE="6" MAXLENGTH="5" ID="timein_input_id"></TD>
<TD CLASS="dedefault">
<SELECT NAME="TimeInAm" SIZE="1">
<OPTION VALUE="AM" SELECTED>AM
<OPTION VALUE="PM">PM
</SELECT>
</TD>
<TD CLASS="dedefault"><INPUT TYPE="text" NAME="TS_TimeOut" SIZE="6" MAXLENGTH="5" ID="timeout_input_id"></TD>
<TD CLASS="dedefault">
<SELECT NAME="TimeOutAm" SIZE="1">
<OPTION VALUE="AM" SELECTED>AM
<OPTION VALUE="PM">PM
</SELECT>
</TD>
<TD CLASS="dedefault"><p class="rightaligntext">0</p></TD>
</TR>
<TR>
<TD COLSPAN="5" CLASS="dedead"> </TD>
<TD CLASS="dedefault"><p class="rightaligntext">0</p></TD>
</TR>
</TABLE>
<P>
<TABLE CLASS="plaintable" SUMMARY="This layout table is used to align buttons.">
<TR>
<TD CLASS="pldefault">
<INPUT TYPE="submit" NAME="ButtonSelected" VALUE="Time Sheet">
<INPUT TYPE="submit" NAME="ButtonSelected" VALUE="Previous Day">
<INPUT TYPE="submit" NAME="ButtonSelected" VALUE="Next Day">
</TD>
</TR>
<TR>
<TD CLASS="pldefault">
<INPUT TYPE="submit" NAME="ButtonSelected" VALUE="Add New Line">
<INPUT TYPE="submit" NAME="ButtonSelected" VALUE="Save">
<INPUT TYPE="submit" NAME="ButtonSelected" VALUE="Copy">
<INPUT TYPE="submit" NAME="ButtonSelected" VALUE="Delete">
</TD>
</TR>
</TABLE>
</FORM>
I am using mechanical soup to fill the form out and submit it.
#navigate to the timesheet page
browser.open(timesheetURL)
browser.get_current_page()
#select the form
browser.select_form('form[action="bwpkteci.P_UpdateTimeInOut"]')
form = browser.get_current_form()
#add required controls and set values for testing
form.new_control('text', 'shift', '1')
form.new_control('text', 'TimeIn', '08:30')
form.new_control('select', 'TimeInAm', 1) #1 stands for PM
form.new_control('text', 'TS_TimeOut', '09:30')
form.new_control('select', 'TimeOutAm', 1) #1 stands for PM
form.new_control('submit', 'ButtonSelected', 'Save')
form.choose_submit('ButtonSelected')
browser.launch_browser()
browser.submit_selected()
browser.launch_browser()
I had to add new controls using new_control function because the TimeIn, TimeOut boxes did not show up when I checked by launching a browser.
This code does not work and I cannot figure out why. I thought maybe I was messing up the names or types of the controls that I was adding (maybe it didn't match with the actual input types and names of the form - I checked with a chrome extension and that too is not the case) as the save button would be clicked but the test values did not really register.
This is what the browser looks like before the save button is clicked:
It doesn't register!
Using MechancialSoup version 0.10.0, I was able to correctly parse your HTML snippet (as best I can tell). I don't see anything specifically that would cause this to fail for older version of MechanicalSoup, but perhaps try updating if nothing else works.
I used the following simple code:
import mechanicalsoup
browser = mechanicalsoup.StatefulBrowser()
browser.open_fake_page(text) #Here 'text' is the HTML snippet
form = browser.select_form('form[action="bwpkteci.P_UpdateTimeInOut"]')
form.print_summary()
browser.launch_browser()
The print_summary() method outputs all the form elements, which include the TimeIn and TimeOut boxes that were missing for you:
<input name="JobsSeqNo" type="hidden" value="208138"/>
<input name="LastDate" type="hidden" value="0"/>
<input name="par_restart" type="hidden" value="Y"/>
<input name="par_update" type="hidden" value="Y"/>
<input name="par_submit" type="hidden" value="Y"/>
<input name="par_recall" type="hidden" value="N"/>
<input name="EarnCode" type="hidden" value="RSA"/>
<input name="DateSelected" type="hidden" value="20-FEB-2018"/>
<input name="LineNumber" type="hidden" value="1"/>
<input id="shift_input_id" maxlength="1" name="Shift" size="2" type="text" value="1"/>
<input id="timein_input_id" maxlength="5" name="TimeIn" size="6" type="text"/>
<select name="TimeInAm" size="1">
<option selected="" value="AM">AM</option><option value="PM">PM</option></select>
<input id="timeout_input_id" maxlength="5" name="TS_TimeOut" size="6" type="text"/>
<select name="TimeOutAm" size="1">
<option selected="" value="AM">AM</option><option value="PM">PM</option></select>
<input name="ButtonSelected" type="submit" value="Time Sheet"/>
<input name="ButtonSelected" type="submit" value="Previous Day"/>
<input name="ButtonSelected" type="submit" value="Next Day"/>
<input name="ButtonSelected" type="submit" value="Add New Line"/>
<input name="ButtonSelected" type="submit" value="Save"/>
<input name="ButtonSelected" type="submit" value="Copy"/>
<input name="ButtonSelected" type="submit" value="Delete"/>
The elements missing for you were also displayed correctly when I use launch_browser():
