lwIP call remote server API

This question will be rather general, as I haven't even gotten to coding yet and am just wondering if it is possible to achieve what I want with lwIP.

What I would like, is for my embedded STM32F769I-Disco board to call a website URL API, e.g. http://test.com/items/1, which returns JSON, which I would then like to parse.

Since lwIP implements a TCP/IP stack, this should theoretically be possible with it, or am I mistaken?

I haven't really found any examples that would do this, or maybe I don't even know how to search for it. Any helpful pointers?

The question is also, should I perhaps use a socket connection rather than trying to call an API url? If I decided to use something like nanopb, would I still need lwIP, or could I do without it?

Solution

Whatever involves retrieving data from a server involves either using either TCP or UDP, so it's not only "theoretically possible", it's the actual way your browser receives data from a web server: by opening a TCP/IP connection to the server's port 80.

UDP is not practical for receiving data from the server, because it's connectionless, and doesn't guarantee the packets will be received in order, or received at all. With TCP, it's "all or nothing" - either you will receive all the packets in order, or the socket will be closed at some point as the sender's buffer gets filled with unacknowledged packets.

So HTTP will use TCP/IP, that's for sure, and there is several steps you will need to do:

Use lwip to create a DNS lookup for your www.test.com website.
Use lwip to open a TCP/IP connection to the IP address, probably port 80.
Attach a recv callback function (among others) to lwip, which will be called by Lwip whenever a packet is received. You will also want to know if the socket gets closed during the conversation, so make sure you register all the callbacks.
Send a HTTP request string to the port 80. You are basically sending a string of characters to the opened TCP socket. This might be something as simple as:
```
GET /items/1 HTTP/1.1
Host: www.test.com
<crlf>
```
The <crlf> is an empty line at the end, it marks the end of the HTTP request.
Your recv function will accept incoming packets from the server, and should write them to a FIFO buffer for further processing. There is no guarantee you will get the entire HTTP response in a single packet, so don't expect to handle them in one pass, unless the payload is really short, there are no proxies in between, and you feel lucky in general.
So, you will then need to parse the incoming data as it arrives. One of the HTTP headers will give you the length of the entire response, or you will need to handle chunked encoding. This is an example of a response you might get from the server (in one or more TCP packets):
```
HTTP/1.1 200 OK
Date: Mon, 23 Aug 2017 22:38:34 GMT
Content-Type: text/html; charset=UTF-8
Content-Encoding: UTF-8
Content-Length: 18

{ 
    "value":5, 
    "something_else": [0, 1, 2] 
}
```
Note the empty new-line between the headers and the start of the payload.
Once you parse all the headers and get to the JSON, then you will probably use an existing C library for parsing JSON, which will also likely require some work to get an understanding of. Replacing this final step with Protocol Buffers might be a sensible idea from the point of development (likely less work on the C side), but the protocol will be less portable than JSON (although certainly smaller).