pythonpython-2.7file-uploadmechanizemechanize-python
Add file to a form using Mechanize (python package) does not work
I am using Mechanize for Python and webscraping a Ticket's Webserver (Where we host the breakdown/incidences for our company) and I wanted to automate it via a file upload, telling in that file new tickets to be added to the database.
Just at the end, when uploading the file, it seems to break while uploading. I tried to upload a non-compatible type of ticket both ways, first via Chromium, and it shooted an error. When I did the same with Mechanize the error didn't show, just as if the button "Submit" were clicked without any file.
I can see when I print br.form that the file is uploaded.. Don't know why it doesn't work.
CODE:
import cookielib
import urllib2
import mechanize
# Browser
br = mechanize.Browser()
# Enable cookie support for urllib2
cookiejar = cookielib.LWPCookieJar()
br.set_cookiejar( cookiejar )
# Broser options
br.set_handle_equiv( True )
br.set_handle_gzip( True )
br.set_handle_redirect( True )
br.set_handle_referer( True )
br.set_handle_robots( False )
br.set_handle_refresh( mechanize._http.HTTPRefreshProcessor(), max_time = 1 )
br.addheaders = [ ( 'User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1' ) ]
# authenticate
br.open('WEB')
br.select_form("loginForm")
br.form['userName'] = 'xxx'
br.form['password'] = 'xxx'
br.submit()
# Inside the webpage
nextPage = br.find_link(text_regex = "Configuración")
br.follow_link(nextPage)
nextPage = br.find_link(text_regex = "Importar Tickets")
br.follow_link(nextPage)
br.select_form("ImporterForm")
br.form['Field Separator'] = ['1'] #CSV
br.submit()
br.select_form("ImporterForm")
br.form.add_file(open("ticket_import_template.csv"), 'text/plain', "ticket_import_template.csv")
br.submit()
I explain the last part of the code: When the last form is selected, I have to change the type of file uploaded to CSV, and then submit() it (in the HTML code a onClick=submit() appears). Then when I select it again and do a print br.form it tells me the CSV radio button is selected, so that part it's OK.
Then I upload the file with br.form.add_file(open("ticket_import_template.csv"), 'text/plain', "ticket_import_template.csv"). When I do print br.form I can see the file is there: <FileControl(7.37.0.0.0.3.3.1.1.1.51=ticket_import_template.csv)>
I copy part of the HTML of the webpage in case it helps:
<form name="ImporterForm" enctype="multipart/form-data" method="post" action="/helpdesk/WebObjects/Helpdesk.woa/wo/148.7.37.0.0.0.3.3.19.1.1">
<table cellspacing="0" border="0" cellpadding="0" width="750">
<tr>
<td class="outline">
<table cellspacing="1" cellpadding="0" border="0" class="detailView" width="100%">
<tr>
<td class="prefHeader" colspan="2">
<div class="preferenceLabel">Importar Tickets</div>
<a target="_blank" class="preferenceHelpLink" href="http://www.solarwinds.com:80/onlinehelp?productType=WebHelpDesk&language=en_US&productVersion=12.2.0&viewType=ProductView&authenticationKey=ignore&helpKey=HelpDeskImportDataUsingTemplates">
<img alt="Help Icon" src="/helpdesk/help_button_icon.png?v=12_2_0_9011" />
</a>
</td>
</tr>
<tr>
<td class="data" colspan="2">
<table class="detailView" cellspacing="1" cellpadding="0" border="0">
<tr>
<td class="data" valign="top">
<img src="/helpdesk/widgets/info_icon_small.png?v=12_2_0_9011" />
</td>
<td class="data" valign="top">
The import file must be in tab-separated values (TSV), comma-separated values (CSV), or Microsoft Excel format.<br/>
<br/>
The first line in the file must be an exact copy of the template.
<br>
<br>
For new records, the following values must be provided:
<ul>
<li>
<b>
Open Date
</b>
</li>
<li>
<b>
Request Type (semicolon delimited)
</b>
</li>
<li>
<b>
Request Detail
</b>
</li>
</ul>
<br>
<i>Note: <strong>Import templates are not backwards compatible. Please make sure that the columns are in the same order as in template.</strong></i>
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td class="label" width="150">
Sincronización basada en
</td>
<td class="data">
<div style="float:left">
<select name="7.37.0.0.0.3.3.19.1.1.9.3.0.0.0.1.1.0"><option selected="selected" value="0">Ticket Number</option><option value="1">Tecnico asignado</option></select>
</div>
<div class="infoIcon" style="">
<a href="#" onClick="togglePopup('MDSDivHelpLink7_37_0_0_0_3_3_19_1_1_9_5', event); return false;" style="display: block;">
<img src="/helpdesk/info_button_icon16x16_t35.png?v=12_2_0_9011" alt="Info Icon"/>
</a>
<div id="MDSDivHelpLink7_37_0_0_0_3_3_19_1_1_9_5" class="helpPopup" onclick="event.cancelBubble = true;">
<strong class="label">Sincronización basada en<br></strong>
Specifies the column in the import file that will be used to synchronize the import data with existing records.
<br/>
<br/> If an import record contains an empty field for the sync column, the record will be added to the database rather than be used to replace an existing record.
</div>
</div>
</td>
</tr>
<tr>
<td class="label" width="150">
Clave del cliente
</td>
<td class="data">
<div style="float:left">
<select onChange="submit()" name="7.37.0.0.0.3.3.19.1.1.11.3.0.0.0.1.1.0"><option selected="selected" value="0">User Name</option><option value="1">E-Mail</option></select>
</div>
<div class="infoIcon" style="">
<a href="#" onClick="togglePopup('MDSDivHelpLink7_37_0_0_0_3_3_19_1_1_11_5', event); return false;" style="display: block;">
<img src="/helpdesk/info_button_icon16x16_t35.png?v=12_2_0_9011" alt="Info Icon"/>
</a>
<div id="MDSDivHelpLink7_37_0_0_0_3_3_19_1_1_11_5" class="helpPopup" onclick="event.cancelBubble = true;">
<strong class="label">Clave del cliente<br></strong>
Specifies which Cliente attribute the import file should use in columns that refer to Cliente records: <b>User Name</b> or <b>E-Mail</b>.
</div>
</div>
</td>
</tr>
<tr>
<td class="label">
Ignorar los campos en blanco
</td>
<td class="data">
<div style="float:left">
<input type="checkbox" name="7.37.0.0.0.3.3.19.1.1.15" value="7.37.0.0.0.3.3.19.1.1.15" checked="checked" />
</div>
<div class="infoIcon" style="">
<a href="#" onClick="togglePopup('MDSDivHelpLink7_37_0_0_0_3_3_19_1_1_17', event); return false;" style="display: block;">
<img src="/helpdesk/info_button_icon16x16_t35.png?v=12_2_0_9011" alt="Info Icon"/>
</a>
<div id="MDSDivHelpLink7_37_0_0_0_3_3_19_1_1_17" class="helpPopup" onclick="event.cancelBubble = true;">
<strong class="label">Ignorar los campos en blanco<br></strong>
Specifies whether fields that are blank in the import file should be ignored when importing. If not checked, blank fields in the import file will cause corresponding values in existing records to be cleared.
<br/>
<br/> If <b>Ignore Blank Fields</b> is unchecked, a field in the import file can still be forced to be ignored by setting it with an asterisk (*).
</div>
</div>
</td>
</tr>
<tr>
<td class="label">
Añadir nuevas entidades
</td>
<td class="data">
<div style="float:left">
<input type="checkbox" name="7.37.0.0.0.3.3.19.1.1.21.0.0" value="7.37.0.0.0.3.3.19.1.1.21.0.0" />
</div>
<div class="infoIcon" style="">
<a href="#" onClick="togglePopup('MDSDivHelpLink7_37_0_0_0_3_3_19_1_1_23', event); return false;" style="display: block;">
<img src="/helpdesk/info_button_icon16x16_t35.png?v=12_2_0_9011" alt="Info Icon"/>
</a>
<div id="MDSDivHelpLink7_37_0_0_0_3_3_19_1_1_23" class="helpPopup" onclick="event.cancelBubble = true;">
<strong class="label">Añadir nuevas entidades<br></strong>
Specifies whether entities referred to by import records should be added if they do not yet exist in the database.
<br/>
<br/> For example, consider an import file having a Localización column, with one of its rows having <i>North Campus</i> for its Localización value. If <b>Add New Entities</b> is unchecked, and <i>North Campus</i> has not yet been entered into the Web Help Desk as a Localización, then the importer will give an error. If <b>Add New Entities</b> is checked, the importer will create a new Localización named <i>North Campus</i>.
<br/>
<br/> Some entities require multiple fields and therefore cannot be created based on a single attribute. For example, Clientes are referenced in import files by e-mail or username, but a new Cliente cannot be created based solely on an e-mail or username. If a Cliente with a matching username or e-mail does not exist, the importer will give an error even if <b>Add New Entities</b> is checked.
<br/>
<br/> <font color="red"><b>Note: </b> This setting does not prevent the importer from creating new records of the base type being imported. It refers only to entities of other types that are referred to by the base records. (E.g., the data file for an Recurso import can add new Recursos regardless of this setting.) </font>
</div>
</div>
</td>
</tr>
<tr>
<td class="label">
Importación abortada en el primer error
</td>
<td class="data">
<div style="float:left">
<input type="checkbox" name="7.37.0.0.0.3.3.19.1.1.27" value="7.37.0.0.0.3.3.19.1.1.27" />
</div>
<div class="infoIcon" style="">
<a href="#" onClick="togglePopup('MDSDivHelpLink7_37_0_0_0_3_3_19_1_1_29', event); return false;" style="display: block;">
<img src="/helpdesk/info_button_icon16x16_t35.png?v=12_2_0_9011" alt="Info Icon"/>
</a>
<div id="MDSDivHelpLink7_37_0_0_0_3_3_19_1_1_29" class="helpPopup" onclick="event.cancelBubble = true;">
<strong class="label">Importación abortada en el primer error<br></strong>
If checked, the importer will stop immediately when an error is encountered. If unchecked, it will continue on with the next record.
</div>
</div>
</td>
</tr>
<tr>
<td class="label" width="125">
Tipo de fichero
</td>
<td class="data">
<div style="float:left">
<input onClick="submit()" type="radio" value="0" name="Field Separator" checked="checked" /> Fichero separado por espacio (TSV)
<input onClick="submit()" type="radio" value="1" name="Field Separator" /> Fichero separado por coma (CSV)
<input onClick="submit()" type="radio" value="2" name="Field Separator" /> Microsoft Excel
</div>
<div class="infoIcon" style="">
<a href="#" onClick="togglePopup('MDSDivHelpLink7_37_0_0_0_3_3_19_1_1_45', event); return false;" style="display: block;">
<img src="/helpdesk/info_button_icon16x16_t35.png?v=12_2_0_9011" alt="Info Icon"/>
</a>
<div id="MDSDivHelpLink7_37_0_0_0_3_3_19_1_1_45" class="helpPopup" onclick="event.cancelBubble = true;">
<strong class="label">Tipo de fichero<br></strong>
Specifies the format of the import file: tab-separated values (TSV) or comma-separated values (CSV).
<br/>
<br/> If the field separator (a comma or a tab) is to be included in a value, the value should be enclosed in quotes. If a quote character is to be included, it should be represented as two sequential quotes.
</div>
</div>
</td>
</tr>
<tr>
<td class="label">
Código de fichero
</td>
<td class="data">
<div style="float:left">
<select name="7.37.0.0.0.3.3.19.1.1.47.0.3.0.0.0.0.0.1.1.0"><option selected="selected" value="0">ASCII</option><option value="1">MS Excel Unicode (UTF-16)</option></select>
</div>
<div class="infoIcon" style="">
<a href="#" onClick="togglePopup('MDSDivHelpLink7_37_0_0_0_3_3_19_1_1_47_0_5', event); return false;" style="display: block;">
<img src="/helpdesk/info_button_icon16x16_t35.png?v=12_2_0_9011" alt="Info Icon"/>
</a>
<div id="MDSDivHelpLink7_37_0_0_0_3_3_19_1_1_47_0_5" class="helpPopup" onclick="event.cancelBubble = true;">
<strong class="label">Código de fichero<br></strong>
Specifies the method used to encode the import file: <b>ASCII</b> or <b>MS Excel Unicode (UTF-16)</b>. If your import file contains any Unicode characters, it must be encoded as UTF-16, the Unicode format used by Microsoft Excel.
<br/>
<br/> To save an Excel file in a format that can be imported, select <b>File > Save As...</b> and choose one of the following for the <b>Format</b> setting:<ul><li><b>CSV (Comma delimited)</b></li><li><b>Text (Tab delimited)</b></li><li><b>Unicode Text (UTF-16)</b></li></ul>
</div>
</div>
</td>
</tr>
<tr>
<td class="label" width="125">
Importar ficheros de datos
</td>
<td class="data">
<div style="float:left">
<table>
<tr>
<td>
<input type="file" name="7.37.0.0.0.3.3.19.1.1.51" />
</td>
<td>
<input title="Remove Upload File" title="Clear Upload Selection" name="7.37.0.0.0.3.3.19.1.1.53" border="0" type="image" src="/helpdesk/delete_line_item.gif?v=12_2_0_9011" />
</td>
</tr>
</table>
</div>
<div class="infoIcon" style="">
<a href="#" onClick="togglePopup('MDSDivHelpLink7_37_0_0_0_3_3_19_1_1_55', event); return false;" style="display: block;">
<img src="/helpdesk/info_button_icon16x16_t35.png?v=12_2_0_9011" alt="Info Icon"/>
</a>
<div id="MDSDivHelpLink7_37_0_0_0_3_3_19_1_1_55" class="helpPopup" onclick="event.cancelBubble = true;">
<strong class="label">Importar ficheros de datos<br></strong>
The data file to use for the import. The columns of the import file must match the order of columns given in the import template (obtained by clicking the <b>Download Template</b> button below).
<br/>
<br/> <font color="red"><b>IMPORTANT!</b> The first line of the import file must match the import template <b>exactly</b>. We recommend copying and pasting the template into your data file in order to be sure it is exactly the same. The importer will give an error if the first row of the import file does not match the template.</font>
</div>
</div>
</td>
</tr>
<tr>
<td colspan="2" class="saveOptions">
<table cellspacing="0" cellpadding="0" border="0" width="100%">
<tr>
<td align="left">
<script language="JavaScript">
// __MDSSubmitLink_sharedJavaScript__
function MDSSubmitLink_submitAction( event, formName, elementId, newWindow, closeNewWindow ) {
if( event.shiftKey ) {
fields = document.getElementsByName( "MDSForm__ShiftKeyPressed" );
for( i = 0; i < fields.length; ++i )
fields[i].value = "1";
}
if( event.altKey ) {
fields = document.getElementsByName( "MDSForm__AltKeyPressed" );
for( i = 0; i < fields.length; ++i )
fields[i].value = "1";
}
var inputObj = document.forms[formName].elements["MDSSubmitLink" + elementId];
inputObj.setAttribute( "name", elementId );
if ( newWindow) {
document.forms[formName].setAttribute("target", "popupWindow");
popupWindow = window.open('', 'popupWindow', 'scrollbars=auto,menubar=yes,height=768,width=1024,resizable=yes,toolbar=no,status=no');
}
document.forms[formName].submit();
return false;}
function nullUpdate() { return false;}</script>
<input type="hidden" name="MDSSubmitLink7.37.0.0.0.3.3.19.1.1.57.0.0" value="DUMMY"><a name="7.37.0.0.0.3.3.19.1.1.57.0.0" onClick="MDSSubmitLink_submitAction( event, 'ImporterForm','7.37.0.0.0.3.3.19.1.1.57.0.0', true);ImporterUpdateContainerUpdate();" ><div class="squareButton"><div class="squareButtonLeft"><img width="25" height="27" src="/helpdesk/buttons/square_icons/tsv.gif?v=12_2_0_9011" /></div><div class="squareButtonMiddle">Descargar plantilla</div><div class="squareButtonRight"></div></div></a>
</td>
<td>
<div class="buttonsRight">
<input type="hidden" name="MDSSubmitLink7.37.0.0.0.3.3.19.1.1.59.0.1.0" value="DUMMY"><a name="7.37.0.0.0.3.3.19.1.1.59.0.1.0" onClick="MDSSubmitLink_submitAction( event, 'ImporterForm','7.37.0.0.0.3.3.19.1.1.59.0.1.0', false);" class="aquaButtonLink" ><div class="aquaButton"><div class="aquaLeftSel"></div><div class="aquaMiddleSel">Importar</div><div class="aquaRightSel"></div></div></a>
</div>
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
</form>
I've searched over the web but I have no clue what's happening.
Thank you
EDIT:
Photo of the Form for you to have an idea of how is it:
Solution
I realized Mechanize doesn't support JavaScript. Tried with requests but it's so difficult.
Finally I decided to use Selenium. Doing some find_element_by_xpath("...") I got to the final form and click()ed the Submit button.
Thanks
- What does cls() function do inside a class method?
- Load Registered Component in Azure ML for Pipeline using Python sdk v2
- Can I add message to the tqdm progressbar?
- How can I preserve the previous value to find the row that is greater than it?
- What's a correct and good way to implement __hash__()?
- Python Harvesters Image Acquisition GigeCam
- Why do I get a recursion error when the depth of the expected recursion should be way less than 999?
- How to (intermittently) skip certain cells when running IPython notebook?
- How to give jupyter cell standard input in python?
- Connect EtherCAT Device with pysoem
- Regex to substitute the next two words after a matching point
- What is the time complexity of heapq.nlargest?
- How to check if a given number is a power of two?
- How to move Jupyter notebook cells up/down using keyboard shortcut?
- cannot override sys.excepthook
- Getting Python's unittest results in a tearDown() method
- Is redis in Python asynchronous?
- How to tell Click to always show option defaults
- flask sqlalchemy with circular imports with db models
- .join with a query
- Python's argparse to show program's version with prog and version string formatting
- classifiers in scikit-learn that handle nan/null
- How does improve detection of line in images by open cv?
- Performance Advantages to Iterators?
- Problem with selecting a specific web element with Playwright in Python
- Thread long running python task, doesn't work inside Docker
- How to add a column with JSON representation of rows in Polars DataFrame?
- Pandas insert empty row at 0th position
- What should HTTP 201 response body be when responding to a POST request with large data?
- Can Python's unittest test in parallel, like nose can?