I am using HA proxy as the load balancer. I have specified two nodes for load balancing in round robin fashion. The things are working fine if i post request manually i can see that the req is getting routed to each node in a round robin fashion. When i use curl to fire some 5000 req then i see the that around 100 req are going to one node and then 100 to another another node. so it is not alternating based on req. I dont think this is the correct beahviour. Is there some config which i need to change ? or is it curl behaviour?
curl -k <url>?name=[1-5000]
You're directing curl to make sequential requests with an incrementing counter, with those numbers in brackets in the URL, and curl is going to reuse the connection to HAProxy, via http keep-alive.
HAProxy is also going to reuse the connection to the backend by default, because that's the most efficient behavior, avoiding the overhead of estsblishing a new TCP connection. There's no reason to artificially force a round robin distribution when all the requests are coming from a single client on a reused connection, with a concurrency of 1.
You can force a change to this behavior with one of several options, but it should be unnecessary since this is primarily an artifact of your test methodology.
By default HAProxy operates in keep-alive mode with regards to persistent connections: for each connection it processes each request and response, and leaves the connection idle on both sides between the end of a response and the start of a new request. This mode may be changed by several options such as "option http-server-close", "option forceclose", "option httpclose" or "option http-tunnel". Setting "option http-server-close" enables HTTP connection-close mode on the server side while keeping the ability to support HTTP keep-alive and pipelining on the client side. This provides the lowest latency on the client side (slow network) and the fastest session reuse on the server side to save server resources, similarly to "option forceclose". It also permits non-keepalive capable servers to be served in keep-alive mode to the clients if they conform to the requirements of RFC7230. Please note that some servers do not always conform to those requirements when they see "Connection: close" in the request. The effect will be that keep-alive will never be used. A workaround consists in enabling "option http-pretend-keepalive".
http://cbonte.github.io/haproxy-dconv/1.6/configuration.html#4-option%20http-server-close
So why does it alternate every ~100? Keep-alive connections can't usually be used indefinitely, so either the client or the server will eventially close the connection and a new one will be established.