Is there an application server where I can set a defined maximum of concurrent client connections? Also no connections should be queued.
Nearly every application server has a feature which is similar to what you want.
In Glassfish you have the possibility to use the so called Connection Pooling
which nearly does what you want. It limits the number of concurrent client connections to the size of the connection pool. Normally it will queue the requests which are above the limitation, but you can also just disable the queueing (or if not possible set it to 1).
Here is a link with detailed information: GlassFish Tuning - HTTP Connection Queue & Keep Alive
In Tomcat you have an option called maxConnections
which does exactly what you want, but be aware that Tomcat is not a Java EE application server (you may check TomEE).
Here is a link with some Tomcat information: Tomcat HTTP Connector Configuration Attributes
In JBoss there is something similar, you have the option maxThreads
which is effectivly the same as maximum connections.
Here is some information about JBoss: JBoss HTTP Connector Configuration Parameters