Search code examples
c#jsonasp.net-corecorrupt

ASP.NET core REST service returns invalid json

We have an ASP.NET core REST service (.NET5). It works fine. But if we call a GET method from two browsers simultaneously, the returned JSON becomes invalid.

The JSON looks scrambled:

{
   "assemblyId":null,                             ┌─ something invalid was inserted here
   "stepScrapQuantity":0,                         ▼
   "webViewer3dFileUrl":"http://172.20.33.188:8000me":"00:00:00",
   "currentProcessTime":"00:00:00"                
}                   ▲                             
                    └─ it looks like this string was inserted above
  • The returned DTO is automatically serialized by the built-in JSON-serializer (System.Text.Json) from ASP.NET.
  • On each GET, the JSON is messed up differently (random). Sometimes keys are messed up, sometimes values are messed up, sometimes it's all fine.
  • It looks like there's a multithreading issue in System.Text.Json
  • Our backend runs in a docker container.
  • It's not reproducible in debug-mode (localhost).
  • It's only reproducible with large JSON-Data (in our case 1.5 MB)
  • It only fails on a real notwork interface (on localhost it's all fine). So maybe it's a network issue.

Did anyone notice a similar issue?

EDIT (2022/10/14) The longer we investigate the problem, the stranger it becomes. The logging of the JSON message in our backend-middleware shows a valid JSON. The received JSON on frontend side is invalid. That means the bug is not in our backend. The payload is broken somewhere between backend and frontend. That's only reproducible when the backend runs as docker container and the service is called remotely (not from localhost).

Looking at this layer model, I wonder if it's possible that docker engine can break the JSON:

enter image description here

source: https://docs.mirantis.com/containers/v3.0/dockeree-ref-arch/networking/scalable-container-networks.html

Solution

  • This error seems to be beyond our control.

    We recorded the traffic with Wireshark. That sowed us that all malformed strings in the JSON are starting points of a TCP package. So it seems the TCP packages are reassembled incorrectly.

    I don't know which component is responsible for splitting and assembling the TCP packages, but it is certainly outside our software.

    enter image description here

    What supports this theory are lots of packages that are "out-of-order": enter image description here

    Solution: Disabling TCP Segmentation Offload (TSO) finally solved the problem. https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-D80AEC2F-E0DA-4172-BFFD-B721BF36C2E8.html