Checking for Type `NoneType` fails in randomly

Question

I'm currently working on function that gets some data from an API and outputs it to csv file. In this function there is a part where I filter the retrieved data before passing it on. Since there is a possibility that the request returns None I decided to explicitly check for that and implement a special behavior in this case. Please take a look at the following code snippet.

r = requests.post(self.settings['apiurl'],
                  headers=self.settings['header'],
                  json={'query': query_string, 'variables': vars})

jsd = json.loads(r.text)
if jsd is not None:
    lists = jsd["data"]["User"]["stats"]["favouredGenres"]
    newlist = [entry for entry in lists if entry["meanScore"] is not None]
    if not newlist:
        return None
    else:
        jsd["data"]["User"]["stats"]["favouredGenres"] = newlist
        try:
            jsd = jsd  # json.loads(jsd)
        except ValueError:
            return None
        else:
            return jsd
else:
    return None

The if jsd is not None part is the before mentioned check. If jsd is not None I filter again some parts out that I don't need and return a modified version of jsd.

The problem now is that I sporadically get the error message:

    lists = jsd["data"]["User"]["stats"]["favouredGenres"]
TypeError: 'NoneType' object is not subscriptable

The first thing that really confuses me is that this error appears completely randomly. In one run it doesn't work on user_id=7 in the next one it doesn't work on user_id=8475 but works fine for user_id=7, etc. The second thing that confuses me is that it is even possible for the error to pop up since I explicitly check before accessing the variable if it is of type NoneType. Beside from these isolated cases where an error occurs the code produces exactly the results I expect....

I hope I provided you with everything necessary, if not, please let me know. Any kind of suggestions on how to approach this kind of problem are welcome.

Answer 1

Blindly sub-scripting a dictionary is usually fraught with this kind of problem. You should switch to using the get method on dictionaries. It can handle this case pretty nicely:

lists = (jsd.get("data", {}).get("User", {})
         .get("stats", {}).get("favouredGenres", []))

Here we exploit the fact that get() can take a default argument, which it will use if the given key isn't found. In the first 3 cases, we provide an empty dictionary as the default, and we provide an empty list for the last case (since that's what I'm guessing you're expecting).

If you get into the habit of using this accessing method, you'll avoid these kinds of problems.

The Python Anti-Patterns book lists this as one to be aware of. The corollary is the setdefault() call when setting values in a dictionary.