Passing a list as a url value to urlopen

Question

Motivation

Motivated by this problem - the OP was using urlopen() and accidentally passed a sys.argv list instead of a string as a url. This error message was thrown:

AttributeError: 'list' object has no attribute 'timeout'

Because of the way urlopen was written, the error message itself and the traceback is not very informative and may be difficult to understand especially for a Python newcomer:

Traceback (most recent call last):
  File "test.py", line 15, in <module>
    get_category_links(sys.argv)
  File "test.py", line 10, in get_category_links
    response = urlopen(url)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 420, in open
    req.timeout = timeout
AttributeError: 'list' object has no attribute 'timeout'

Problem

Here is the shortened code I'm working with:

try:
    from urllib.request import urlopen
except ImportError:
    from urllib2 import urlopen

import sys


def get_category_links(url):
    response = urlopen(url)
    # do smth with response
    print(response)


get_category_links(sys.argv)

I'm trying to think whether this kind of an error can be caught statically with either smart IDEs like PyCharm, static code analysis tools like flake8 or pylint, or with language features like type annotations.

But, I'm failing to detect the problem:

• it is probably too specific for flake8 and pylint to catch - they don't warn about the problem

• PyCharm does not warn about sys.argv being passed into urlopen, even though, if you "jump to source" of sys.argv it is defined as:

   argv = [] # real value of type <class 'list'> skipped
  

• if I annotate the function parameter as a string and pass sys.argv, no warnings as well:

   def get_category_links(url: str) -> None:
       response = urlopen(url)
       # do smth with response
  
  
   get_category_links(sys.argv)
  

Question

Is it possible to catch this problem statically (without actually executing the code)?

Answer 1

Instead of keeping it editor specific, you can use mypy to analyze your code. This way it will run on all dev environments instead of just for those who use PyCharm.

from urllib.request import urlopen
import sys


def get_category_links(url: str) -> None:
    response = urlopen(url)
    # do smth with response


get_category_links(sys.argv)
response = urlopen(sys.argv)

The issues pointed out by mypy for the above code:

error: Argument 1 to "get_category_links" has incompatible type List[str]; expected "str"
error: Argument 1 to "urlopen" has incompatible type List[str]; expected "Union[str, Request]"

Mypy here can guess the type of sys.argv because of its definition in its stub file. Right now some standard library modules are still missing from typeshed though, so you will have to either contribute them or ignore the errors related till they get added :-).

---

When to run mypy?

1. To catch such errors you can run mypy on the files with annotations with your tests in your CI tool. Running it on all files in project may take some time, for a small project it is your choice.

2. Add a pre-commit hook that runs mypy on staged files and points out issues right away(could be a little annoying to the dev if it takes a while).