Let's say Tomcat supports max 5 threads, and 5 threads are in progress (assume these request will take a lot of time). Now 1 of the request creates 2 more threads which do something,
Here's the documentation for maxThreads:
The maximum number of connections that the server will accept and process at any given time. When this number has been reached, the server will accept, but not process, one further connection. This additional connection be blocked until the number of connections being processed falls below maxConnections at which point the server will start accepting and processing new connections again. Note that once the limit has been reached, the operating system may still accept connections based on the acceptCount setting. The default value varies by connector type. For BIO the default is the value of maxThreads unless an Executor is used in which case the default will be the value of maxThreads from the executor. For NIO the default is 10000. For APR/native, the default is 8192.
The limit on threads is the maximum number configured for an internal pool that Tomcat uses to assign to incoming requests. That's not a hard limit on the number of threads used by the JVM.
If your request creates more threads (calling new Thread()) in the course of its work, those threads don't come from the pool, they're requested from the OS.
But the best course is probably to use a separate, dedicated threadpool. You can always configure your own threadpool and have your request take threads from that pool. That way you don't run the system out of threads (which you would risk doing with creating new threads, if something gets out of hand and threads start hanging) and you don't draw down Tomcat's capacity to serve requests (which you would do if you used Tomcat's pool).