Invalid unicode while sending a POST request to django with libcurl and jsoncpp

Question

I've created a REST endpoint in Django using the rest-framework module; the simple code goes as below:

models.py

class Data(models.Model):
    name = models.CharField(max_length=256)
    description = models.TextField()

    def __str__(self):
        return self.name

serializers.py

class DataSerializer(serializers.ModelSerializer):
    class Meta:
        model = Data
        fields = ('name', 'description')

views.py

def data_list(request):
    """
    List all data.
    """
    if request.method == 'GET':
        categories = Data.objects.all()
        serializer = DataSerializer(categories, many=True)
        return JSONResponse(serializer.data)

    elif request.method == 'POST':
        data = JSONParser().parse(request)
        serializer = DataSerializer(data=data)
        if serializer.is_valid():
            serializer.save()
            return JSONResponse(serializer.data, status=201)
        return JSONResponse(serializer.errors, status=400)

I've tried sending POST requests using the RESTClient plugin for Firefox, and can validate that it works as-is.

However, my use-case is that I'd like to write to the database using libcurl in a C++ application.

If I use jsoncpp to create a JSON object, and then use libcurl to make a POST request, as below:

void main() {
    Json::Value submitted_data;
    submitted_data["name"] = "data id";
    submitted_data["description"] = "data description";
    Json::StyledWriter writer;

    CURL *curl;
    CURLcode res;

    curl_global_init(CURL_GLOBAL_ALL);

    curl = curl_easy_init();
    if (curl) {
        curl_easy_setopt(curl, CURLOPT_URL, "http://127.0.0.1:8000/data/");
        struct curl_slist *headers = NULL;
        headers = curl_slist_append(headers, "Content-Type: application/json; charset=UTF-8");
        curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
        curl_easy_setopt(curl, CURLOPT_POST, 1);
        curl_easy_setopt(curl, CURLOPT_POSTFIELDS, writer.write(submitted_data).c_str());

        res = curl_easy_perform(curl);
        if (res != CURLE_OK)
            fprintf(stderr, "curl_easy_perform() failed: %s\n",
            curl_easy_strerror(res));

        curl_easy_cleanup(curl);
    }
}

I get an error from the django server:

  File "C:\Python27\lib\site-packages\rest_framework\parsers.py", line 67, in parse
    raise ParseError('JSON parse error - %s' % six.text_type(exc))
ParseError: JSON parse error - 'utf8' codec can't decode byte 0xdd in position 0: invalid continuation byte

And the post request isn't successful. My understanding is that:

• django-rest-framework expects a utf-8 encoded json string,
• jsoncpp encodes strings in utf-8, and
• libcurl is agnostic about encoding and deals with data at the byte level.

so I'm a bit surprised by this and not sure how to begin troubleshooting. Can someone help me figure out how to have my C++ application and django application work together?

thanks!

Answer 1

Per the CURLOPT_POSTFIELDS documentation:

The data pointed to is NOT copied by the library: as a consequence, it must be preserved by the calling application until the associated transfer finishes. This behaviour can be changed (so libcurl does copy the data) by setting the CURLOPT_COPYPOSTFIELDS option.

You are passing a temporary char* pointer to CURLOPT_POSTFIELDS. This is because Json::StyledWriter::write() returns a temporary std::string that you are then calling c_str() on. When the call to curl_easy_setopt() is complete, that std::string gets destroyed, and thus the char* pointer that curl is holding on to is no longer valid. Curl ends up transmitting garbage data from freed memory. This is undefined behavior, you are lucky your code did not just crash altogether.

So, you need to either:

1. preserve the std::string in a local variable until curl_easy_perform() is finished:

   std::string json = writer.write(submitted_data);
   curl_easy_setopt(curl, CURLOPT_POSTFIELDS, json.c_str());
   

2. use CURLOPT_COPYPOSTFIELDS instead of CURLOPT_POSTFIELDS:

   curl_easy_setopt(curl, CURLOPT_COPYPOSTFIELDS, writer.write(submitted_data).c_str());