Reading cherrypy's response as bytes/string after migrating from py2 to py3

Question

I just migrated a cherrypy app from 2.7 to python (running on 3.6). I had a bunch of tests setup before based on this recipe. The point of the recipe is to emulate the network and perform tests units on individual endpoints.

Now my server on its own seems to run fine. However if I run the test units, they mostly return errors in py3 (all pass in py2), which seems to have to do about the response being in bytes (in py3) as opposed to string (in py2).

A test response
    ======================================================================
    FAIL: test_index (__main__.EndpointTests)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/anonymous/PycharmProjects/server-py3/tests/test_DServer.py", line 67, in test_index

        self.assertEqual(response.body, ['Hello World!'])
    AssertionError: Lists differ: [b'Hello World!'] != ['Hello World!']

    First differing element 0:
    b'Hello World!'
    'Hello World!'

    - [b'Hello World!']
    ?  -

    + ['Hello World!']

Similarly:

======================================================================
FAIL: test_valid_login (__main__.EndpointTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/anonymous/PycharmProjects/server-py3/tests/test_DServer.py", line 73, in test_valid_login
    self.assertEqual(response.output_status, '200 OK')
AssertionError: b'200 OK' != '200 OK'

I know that the behavior is somewhat different with regards to bytes (such as explained here).

Actually 2 questions:

What's the best way to deal with this in my test? Do I need to prepend b in front of every string that is asserted in the server response?

Quick test on the server seems to indicate it works. However am I likely be bitten by this issue in other fashion? Any words of wisdom as to other gotchas with cherrypy & migration to py3?

I am not supporting py2 after that, I can do a clean migration.

Answer 1

Found it.

The edits need to be made in cptestcase. The core of the issue is that this recipe is somewhat dependent on the internal workings of Cherrypy, an 2to3 (the tool that I used to do the grunt work of the migration) couldn't manage the details well enough to cp's liking.

The summary is that instead of io.StringIO (which is what 2to3 provided by default) you need to switch to io.BytesIO. Therefore previous calls to StringIO(data) should be BytesIO(data). The point being that cp internally now expects those strings/bytes (since py2 didn't really make any difference between the 2) to be actual bytes (because py3 actually DOES differentiate). And yes, in the actually test when making asserts you have to either convert response.output_status & response.body from bytes to strings or compare them to bytes, like so:

self.assertEqual(response.output_status, b'200 OK')

However the query_string (for GET) must still remain, well, a string.

Here's the full edit of the code that worked for me:

from io import BytesIO
import unittest
import urllib.request, urllib.parse, urllib.error

import cherrypy

cherrypy.config.update({'environment': "test_suite"})
cherrypy.server.unsubscribe()

local = cherrypy.lib.httputil.Host('127.0.0.1', 50000, "")
remote = cherrypy.lib.httputil.Host('127.0.0.1', 50001, "")

__all__ = ['BaseCherryPyTestCase']

class BaseCherryPyTestCase(unittest.TestCase):
    def request(self, path='/', method='GET', app_path='', scheme='http',
                proto='HTTP/1.1', data=None, headers=None, **kwargs):

        h = {'Host': '127.0.0.1'}

        if headers is not None:
            h.update(headers)

        if method in ('POST', 'PUT') and not data:
            data = urllib.parse.urlencode(kwargs).encode('utf-8')
            kwargs = None
            h['content-type'] = 'application/x-www-form-urlencoded'

        qs = None
        if kwargs:
            qs = urllib.parse.urlencode(kwargs)

        fd = None
        if data is not None:
            h['content-length'] = '%d' % len(data.decode('utf-8'))
            fd = BytesIO(data)

        app = cherrypy.tree.apps.get(app_path)
        if not app:
            raise AssertionError("No application mounted at '%s'" % app_path)

        app.release_serving()

        request, response = app.get_serving(local, remote, scheme, proto)
        try:
            h = [(k, v) for k, v in h.items()]
            response = request.run(method, path, qs, proto, h, fd)
        finally:
            if fd:
                fd.close()
                fd = None

        if response.output_status.startswith(b'500'):
            print(response.body)
            raise AssertionError("Unexpected error")

        response.collapse_body()
        return response