I am using .net core worker app (3.1) with MySQL & Dapper.
This is not a new issue and I was experiencing it for > 1 year, never was causing a big issues as always the task would be retried.
However now I am developing new feature that has to process > 300-600k rows within few seconds connecting hundreds of times to database.
The issue is that randomly in logs I can see MySQL.Data.MySqlClient.MySqlExceptions (get records via Dapper or execute SQL command).
They all are generic without anything beyond stream has failed and operation was cancelled.
The weirdest issue is that it is happening ONLY in Linux environment (client is then local to the MySQL instance). When I run it on Windows (via VS or Docker image) it runs fine for > 30x bigger test loads.
When I deploy to linux (same docker image using 3.1 aspnet image from microsoft) it fails.
I've tested it a bit directly publish to ubuntu & dotnet run dll and worked fine, but I've tried that only ~ 10 times.
So not a big test.
Any idea where I should look ? I've tried to make sure that Dapper query exec will be using opened connection but it is not it. There is a retry too but sometimes it works fine at first try, sometimes it will retry 2 times and sometimes it will fail 20 times five times in a row.
Can't pinpoint this issue, not sure if this is a Dapper or maybe Microsoft image is not rock solid to be run on Linux....
Ok guys, I've found the solution...
First, thank you all for helping me out.
Reason ?
Bizarre at least...
Turned out that > 15 of different services/apps were using same library that was created some time ago. This was a helper library build from a source, but for some reason was created using dependencies .net framework / .net core. In addition another library was using right .net core library.
Those libraries were hidden basically from all apps as would be pulled along as dependency.
Sometimes app would use corrupted library that had dependency in .net framework, and in that case MySQL would be crashing with all those different errors (that is why I wasn't able to pin point the cause of it). In dev it worked, but for dev we're using Windows, that is why we couldn't catch it in the act.
That is why some apps would be freezing in a weird way and causing lot bunch of issues when under the load.
Cleared corrupted library, fixed dependencies...
Works flawlessly for > 14 hours now (it is under constant load test scenario. No single error.