How to reduce request time in a JSON or replace a dictionary key with a default one?

Question

I have a list of dictionaries and I'm filling it out as I search a JSON url. The problem is that JSON (provided by the Google Books API) is not always complete. This is a search for books and from what I saw, all of them have id, title and authors, but not all of them have imageLinks. Here's a JSON link as an example: Search for Harry Potter.

Note that it always returns 10 results, in this example there are 10 IDs, 10 titles, 10 authors, but only 4 imageLinks.

@app.route('/search', methods=["GET", "POST"])
@login_required
def search():
    if request.method == "POST":
        while True:
            try:
                seek = request.form.get("seek")
                url = f'https://www.googleapis.com/books/v1/volumes?q={seek}'
                response = requests.get(url)
                response.raise_for_status()
                search = response.json()
                seek = search['items']
                infobooks = []
                for i in range(len(seek)):
                    infobooks.append({
                        "book_id": seek[i]['id'],
                        "thumbnail": seek[i]['volumeInfo']['imageLinks']['thumbnail'],
                        "title": seek[i]['volumeInfo']['title'],
                        "authors": seek[i]['volumeInfo']['authors']
                    })
                return render_template("index.html", infobooks=infobooks)
            except (requests.RequestException, KeyError, TypeError, ValueError):
                continue
    else:
        return render_template("index.html")

The method I used and that I'm demonstrating above, I can find 10 imageLinks (thumbnails) but it takes a long time! Anyone have any suggestions for this request not take so long? Or some way I can insert a "Book Without Cover" image when I can't find an imageLink? (not what I would like, but it's better than having to wait for the results)

Answer 1

Firstly your function will never result in 10 imageLinks as the api will always return the same results. So if you retrieved 4 imageLinks the first time it will the same the second time. Unless google updates the dataset, but that is out of your control.

The Google Books Api allows max to 40 results and has default of max 10 results. To increase that you can add the query parameter maxResults=40 where 40 can be any desired number equal or lower than 40. Here you can then decide to programmatically filter out all results without imageLinks, or to leave them and add a no results image url to them. Also not every result returns a list of authors, that has also been fixed in this example. Take no risks with third party api's always check on empty/null results because it can break your code. I have used .get to avoid any exceptions from occurring when processing json.

Although I have not added it in this example you can also use pagination which google books provides to paginate for even more results.

Example:

@app.route('/search', methods=["GET", "POST"])
@login_required
def search():
    if request.method == "POST":
        seek = request.form.get("seek")
        url = f'https://www.googleapis.com/books/v1/volumes?q={seek}&maxResults=40'
        response = requests.get(url)
        response.raise_for_status()
        results = response.json().get('items', [])
        infobooks = []
        no_image = {'smallThumbnail': 'http://no-image-link/image-small.jpeg', 'thumbnail': 'http://no-image-link/image.jpeg'}
        for result in results:
            info = result.get('volumeInfo', {})
            imageLinks = info.get("imageLinks")
            infobooks.append({
                "book_id": result.get('id'),
                "thumbnail": imageLinks if imageLinks else no_image,
                "title": info.get('title'),
                "authors": info.get('authors')
            })
        return render_template("index.html", infobooks=infobooks)
    else:
        return render_template("index.html")

Google Books Api Docs: https://developers.google.com/books/docs/v1/using