Search code examples
curlcontent-length

why is cURL not reading full @file and setting Content-Length: 8186?

I am using cURL on Windows 7 to test a RESTful API.

My data is in a JSON file that is 16,987 characters long.

When I do not set the content length cURL assumes Content-Length: 8186 and does not send the entire JSON payload. I get Unexpected end-of-input from the server.

When I do set -H "Content-Length: 17000" cURL seems to send off the first 8186 bytes without properly structuring the request body (or something)... the server takes some time thinking about this and finally gives 400 Bad Request.

with no explicit content type header:

C:\...>curl -v -X POST -H "Content-Type: application/json"  -H "Expect:" -H "Authorization=Bearer ...etc..." --data @quotedGood.json https://web.domain.com/NewApp/rest/invoice/
Note: Unnecessary use of -X or --request, POST is already inferred.
* STATE: INIT => CONNECT handle 0x60006a600; line 1103 (connection #-5000)
* Added connection 0. The cache now contains 1 members
*   Trying 192.168.0.0...
* STATE: CONNECT => WAITCONNECT handle 0x60006a600; line 1156 (connection #0)
* Connected to web.domain.com (192.168.0.0) port 443 (#0)
* STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x60006a600; line 1253 (connection #0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/ssl/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x60006a600; line 1267 (connection #0)
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*        subject: C=CA; ST=prov; L=town; O=Company; CN=*.domain.com
*        start date: Mar  5 17:57:58 2015 GMT
*        expire date: Nov 10 23:57:20 2018 GMT
*        subjectAltName: web.domain.com matched
*        issuer: C=US; O=Entrust, Inc.; OU=...etc...
*        SSL certificate verify ok.
* STATE: PROTOCONNECT => DO handle 0x60006a600; line 1288 (connection #0)
> POST /NewApp/rest/invoice/ HTTP/1.1
> Host: web.domain.com
> User-Agent: curl/7.47.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 8186
>
* upload completely sent off: 8186 out of 8186 bytes
* STATE: DO => DO_DONE handle 0x60006a600; line 1350 (connection #0)
* STATE: DO_DONE => WAITPERFORM handle 0x60006a600; line 1477 (connection #0)
* STATE: WAITPERFORM => PERFORM handle 0x60006a600; line 1487 (connection #0)
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 400 Bad Request
< Date: Mon, 05 Mar 2018 21:37:26 GMT
< Content-Type: application/json
< Set-Cookie: SERVERID=web8; path=/; HttpOnly; Secure
< Connection: close
< Transfer-Encoding: chunked
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=31536000; includeSubDomains
<
* STATE: PERFORM => DONE handle 0x60006a600; line 1645 (connection #0)
* Curl_done
* Closing connection 0
* The cache now contains 0 members
* TLSv1.2 (OUT), TLS alert, Client hello (1):
* Expire cleared
{"summary":"There was a problem with your invoice request. If you require technical assistance, please note the following identifier [NEWAPP-EX-646744087] to help trace the issue.","timestamp":"2018-03-05 16:37:26.105","errorMsgs":["Unexpected end-of-input in field name\n at [Source: org.jboss.resteasy.core.interception.MessageBodyReaderContextImpl$InputStreamWrapper@74d44e80; line: 245, column: 205]"]}

With an explicit content length. Note the same log as above * upload completely sent off: 8186 out of 8186 bytes, even though the Content-Length is 17000... make me think cURL is not reading the whole file:

C:\...>curl -v -X POST -H "Content-Type: application/json" -H "Content-Length: 17000" -H "Expect:" -H "Authorization=Bearer ...etc..." --data @quotedGood.json https://web.domain.com/NewApp/rest/invoice/
Note: Unnecessary use of -X or --request, POST is already inferred.
* STATE: INIT => CONNECT handle 0x60006a690; line 1103 (connection #-5000)
* Added connection 0. The cache now contains 1 members
*   Trying 192.168.0.0...
* STATE: CONNECT => WAITCONNECT handle 0x60006a690; line 1156 (connection #0)
* Connected to web.domain.com (192.168.0.0) port 443 (#0)
* STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x60006a690; line 1253 (connection #0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/ssl/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x60006a690; line 1267 (connection #0)
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*        subject: C=CA; ST=prov; L=town; O=Company; CN=*.domain.com
*        start date: Mar  5 17:57:58 2015 GMT
*        expire date: Nov 10 23:57:20 2018 GMT
*        subjectAltName: web.domain.com matched
*        issuer: C=US; O=Entrust, Inc.; OU=...etc...
*        SSL certificate verify ok.
* STATE: PROTOCONNECT => DO handle 0x60006a690; line 1288 (connection #0)
> POST /NewApp/rest/invoice/ HTTP/1.1
> Host: web.domain.com
> User-Agent: curl/7.47.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 17000
>
* upload completely sent off: 8186 out of 8186 bytes
* STATE: DO => DO_DONE handle 0x60006a690; line 1350 (connection #0)
* STATE: DO_DONE => WAITPERFORM handle 0x60006a690; line 1477 (connection #0)
* STATE: WAITPERFORM => PERFORM handle 0x60006a690; line 1487 (connection #0)
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 400 Bad Request
< Date: Mon, 05 Mar 2018 21:48:11 GMT
< Content-Length: 226
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=31536000; includeSubDomains
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
</body></html>
* STATE: PERFORM => DONE handle 0x60006a690; line 1645 (connection #0)
* Curl_done
* Closing connection 0
* The cache now contains 0 members
* TLSv1.2 (OUT), TLS alert, Client hello (1):
Solution

  • here's what I had in the end when I finally got it working, in a cmd prompt on Windows 7

    curl --request POST ^ --url http://127.0.0.1:8080/MyApp/rest/invoice ^ --header "Content-Type:application/json" ^ --header "Authorization: some-proprietary-auth-code" ^ --header "Content-Length:9905" --header "Expect:" ^ --data-binary "@.\compressedGood.json"

    compressedGood.json is a file with un-escaped, double-quoted fields and all spaces and eols removed. Like:

    {"externalId":null,"companyName":ACME,"companyProvince":ON,"salesFirstName":null,"salesLastName":null,"products":[ {"name":"Name of Product", ... etc  ... } ] }