I have a go server that is reading & returning large datafiles bundled into a tar. I have tested that this works & writes the tar in pieces and when all the data loads, it's all good.
The problem is that there can be unexpected errors that break the download. I currently write an HTTP error code & an error message, but the error message just gets put at the end of the stream/file. Is there a good way to communicate that the export failed partway through? Is it possible to do this with HTTP status codes & also providing error messages?
I am using the following curl command:
curl --insecure https://127.0.0.1/api/export/030e28f3-4ab6-446a-852e-fda0a497ffe2 -o "test.tar"
Do I have to change the curl command to detect errors (too)?
If the download started already, then all the HTTP headers are already sent to the HTTP client. You cannot rewrite the status code anymore (it was on the first line).
The only thing you could do is cut the TCP/IP connection. If you used a Content-Length header the client will see that the transfer was incomplete. If you used Transfer-Encoding: chunked the client will see that the end-of-chunks marker was not received. In all cases this will simply invalidate the whole transfer.
You could try to play with Range requests and Partial Content responses, and send the content in several HTTP request-response dialogs. But if you manage the big file transmission over a single HTTP dialog the only thing you can do is breaking the transmission, and starting it back completely.
Usually big file transmissions in chunks are managed on the application layer, on top of HTTP. Like in case of uploads a JavaScript thing will explode the file in chunks, and send it back to a dedicated application server, rebuilding the file, with a dedicated protocol on top of HTTP to ask for missing pieces. This is because in real-life wild HTTP environments, long transmissions are hard, and range/partial content transmissions not very well managed by all potential proxies. On the other hand, using simple medium size HTTP requests works almost everywhere. So if you control both the client and server side you can build your own dialog on top of HTTP and make your own chunked transmission protocol, with good error management.