Inserting large amount of data in Apache Ignite

I am using Apache Ignite .NET thin client 2.8.1 to insert a large amount of data in the Apache Ignite node. Ignite is hosted on Amazon Linux AMI. I am trying to insert over 500000 records using PutAllAsync method:

await cacheClient.PutAllAsync(entities); // ICacheClient<int, T>

After that I see the following exception in client logs:

2021-02-03 07:23:37.9917 - NO_TRACE - ****************** - Error: Could not get data from cache
Exception has been thrown by the target of an invocation. System.Reflection.TargetInvocationException System.Object InvokeMethod(System.Object, System.Object[], System.Signature, Boolean) System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at Apache.Ignite.Core.Impl.Client.ClientSocket.SocketRead(Byte[] buf, Int32 pos, Int32 len)
   at Apache.Ignite.Core.Impl.Client.ClientSocket.ReceiveBytes(Int32 size)
   at Apache.Ignite.Core.Impl.Client.ClientSocket.ReceiveMessage()
   at Apache.Ignite.Core.Impl.Client.ClientSocket.SendRequest(RequestMessage& reqMsg)
   at Apache.Ignite.Core.Impl.Client.ClientSocket.DoOutInOp[T](ClientOp opId, Action`1 writeAction, Func`2 readFunc, Func`3 errorFunc)
   at Apache.Ignite.Core.Impl.Client.Cache.CacheClient`2.DoOutInOp[T](ClientOp opId, Action`1 writeAction, Func`2 readFunc)
   at Apache.Ignite.Core.Impl.Client.Cache.CacheClient`2.DoOutOp(ClientOp opId, Action`1 writeAction)

And the following error in ignite logs:

[07:44:53,529][WARNING][grid-timeout-worker-#22][ClientListenerNioListener] Unable to perform handshake within timeout [timeout=10000, remoteAddr=/172.31.56.14:52631]

Are there any best practices of how to insert over 5000 records in ignite cache? Insert batches within transaction?

Solution

Ignite is hosted on Amazon Linux AMI

Where do you run the thin client? I suspect that the issue can be simply the low connection speed between the server and the client.

Can you test the connection speed?
What is the size of the data, in megabytes?
How much time does PutAll take for 10, 100, 1000 entries?

There is no DataStreamer in the thin client (yet), so my suggestions are:

Split one big PutAll into multiple smaller ones (e.g. 100 entries at a time)
Increase timeouts on server and client

UPDATE: DataStreamer is now available in Ignite.NET thin client: