Search code examples
pythonpostpython-requestsdeveloper-toolsmechanize-python

Alternative to using mechanize to fetch data?

Currently, I am using mechanize to fill up a form and send a POST request, then I am doing a regex search on the response to get the data ( a floating point number).

Is there any way I can do this by just sending a POST request? I know this is possible by using a combination of any browser's Developer Tools and the requests module to send the request but I have failed to find a comprehensive tutorial. I would also like some details about the steps involved.

Solution

  • First step: get the field name

    Inspect the HTML code and find the name attribute of the field. For example, the comment form on this page is (in Chrome, right-click and choose "inspect element"):

    <textarea name="comment" cols="68" rows="3" 
placeholder="Use comments to ask for more information or 
suggest improvements. Avoid answering questions in comments."
></textarea>

    The field name is comment.

    Step 2: assemble a dict of name: value for each field (including the hidden inputs)

    Lets call it data:

    data = {
   "comment": "this is a comment",
   "post_id": 1234
}

    Step 3: use the data argument of `requests.post'

    response = requests.post(url, data=data, cookies=cookies)

    More advanced stuff

    If your form has file fields you may have to check "More complicated POST requests" in the docs. Same goes for custom authentication, cookie handling, etc.