I have a microservice, hosted in Service Fabric, that handles uploading files to blob storage. The microservice is implemented with Nancy and OWIN. When the request is over a certain size, something like a couple hundred KB maybe, the request gets written to disk in a temp directory. Occasionally these .tmp files fail to get cleaned up, and eat up the limited disk space on the SF Cluster VM.
I have not been able to find anything about requests automatically getting written to disk. And nothing in the code creates .tmp files. What could be generating these files: Service Fabric, Nancy, OWIN?
Nancy is doing this, it has something called "request stream switching" which, as you say, switches from a memory stream to a file based stream over a certain size to avoid being able to fill all the memory up by uploading a large (or neverending file).
They should get cleaned up after every request, I haven't see any reports of them not being for a long time (we've fixed bugs around this in the past), but if you want to disable it completely (and accept the potential issue above) you can use "StaticConfiguration.DisableRequestStreamSwitching" in your bootstrapper app startup to turn it off.