Getting Typeerror: can't concat str to bytes, when writing a test case to enter request files in django rest framework?

Question

I was working on a project where I can add a csv to upload shirts details to the database.

I wrote a viewset that takes shirt csv files input from request.data in django rest framework, and am writing a test case for the same. Though, while writing the test case, rather than creating a seperate csv file, I created a csv from pandas dataframe inside the test case, and converting it into TextIOWrapper to pass as request data. Doing this throws the following error:

  File "/Users/prasoon/projects/shirt_details/post_shirt_csv/tests/test_post_shirt_dataset.py", line 175, in test_shirt_dataset_post
    response = self.api_client.post(reverse('post-shirt-csv-dataset', kwargs={'session_id': self.session.id}), {'csv_files': [fp1, fp2]})
  File "/Users/prasoon/projects/venv/lib/python3.6/site-packages/rest_framework/test.py", line 300, in post
    path, data=data, format=format, content_type=content_type, **extra)
  File "/Users/prasoon/projects/venv/lib/python3.6/site-packages/rest_framework/test.py", line 212, in post
    data, content_type = self._encode_data(data, format, content_type)
  File "/Users/prasoon/projects/venv/lib/python3.6/site-packages/rest_framework/test.py", line 184, in _encode_data
    ret = renderer.render(data)
  File "/Users/prasoon/projects/venv/lib/python3.6/site-packages/rest_framework/renderers.py", line 920, in render
    return encode_multipart(self.BOUNDARY, data)
  File "/Users/prasoon/projects/venv/lib/python3.6/site-packages/django/test/client.py", line 198, in encode_multipart
    lines.extend(encode_file(boundary, key, item))
  File "/Users/prasoon/projects/venv/lib/python3.6/site-packages/django/test/client.py", line 247, in encode_file
    to_bytes(file.read())
  File "/Users/prasoon/projects/venv/bin/../lib/python3.6/codecs.py", line 320, in decode
    data = self.buffer + input
TypeError: can't concat str to bytes

The thing works when I am opening a file and passing it instead of creating the string. It will be more clear below:

The tests file is:

class TestPostNewDataset(TestCase):
    @classmethod
    def setUpTestData(cls):
        cls.user = CurrentUserFactory()

        cls.test_data1 = {"no":[1,2,3],"colour":["Red","Blue","Gray"],"Size":["XL","L","M"],"Brand":["Nike","Polo","Adidas"],"Price":[800,600,1000]}
        cls.test_data2 = {"no":[1,2,3],"colour":["Black","Blue","Red"],"Size":["S","M","L"],"Brand":["Nike","Levis","Nike"],"Price":[2000,900,1000]}

        cls.test_df1 = pd.DataFrame(cls.test_data1)
        cls.test_df2 = pd.DataFrame(cls.test_data2)
        cls.session = CurrentSessionFactory(user=cls.user)

    def test_shirt_dataset_post(self):
        "Test request to post shirt dataset"

        fp1 = TextIOWrapper(StringIO(self.test_df1))
        fp2 = TextIOWrapper(StringIO(self.test_df2))

        response = self.api_client.post(reverse("post-shirt-csv-dataset", kwargs={'session_id':self.session.id}),{'csv_files':[fp1,fp2]})

        self.assertEqual(response.status_code,status.HTTP_200_OK)

The corresponding views file is:

class CsvReadViewset(viewsets.ModelViewSet):
    authentication_classes = (BasicAuthentication,)
    permission_classes = (IsAuthenticated,)
    serializer_class = ShirtSerializer
    queryset = Shirt.objects.all()

    def addShirt(self,request,*args,**kwargs):
        csvs = request.data["csv_files"]
        current_session = CurrentSession.objects.get(session_id=kwargs["session_id"])
        for shirt_csv in csvs:
            shirt_df = pd.read_csv(shirt_csv)
            for index,row in short_df.iterrows():
                s = Shirt(colour=row["colour"],
                          brand=row["Brand"],
                          size=row["Size"],
                          price=row["Price"],
                          created_in=current_session)
                s.save()
        return Response(status=status.HTTP_200_OK)

The interesting thing is that when the test case takes a seperate file as input, it works fine. That is if I replace test_shirt_dataset_post with following:

    def test_shirt_dataset_post(self):
        with open(os.path.join(os.path.abspath(__file__ + "/../"), 'shirtcsv1.csv')) as fp1, open(os.path.join(os.path.abspath(__file__ + "/../"), 'shirtcsv2.csv')) as fp2:
             response = self.api_client.post(reverse("post-shirt-csv-dataset",kwargs={'session_id':self.session.id}),{'csv_files':[fp1,fp2]})

        self.assertEqual(response.status_code,status.HTTP_200_OK)

The expected result is the test case to work. But, here it throws an error as shown. The reason for adding TextIOWrapper is because the type of fp in with open(....) as fp is '_io.TextIOWrapper'. Though, then it throws this error. Any help will be appreciated.

Answer 1

You can use io.BytesIO to send an encoded byte stream:

def test_shirt_dataset_post(self):
    "Test request to post shirt dataset"

    fp1 = BytesIO(self.test_df1.to_csv().encode('utf-8'))
    fp2 = BytesIO(self.test_df2.to_csv().encode('utf-8'))

    response = self.api_client.post(reverse("post-shirt-csv-dataset", kwargs={'session_id':self.session.id}),{'csv_files':[fp1,fp2]})

    self.assertEqual(response.status_code,status.HTTP_200_OK)