Usually, in a connection pool for a DBMS, it is acceptable and sometime preferable to have even several hundreds connections.
Looking examples online, I always see connection pools with just a few connections and a large number for the sessions associated to a connection.
Is this because the connections to ActiveMQ are somehow expensive?
Both database and JMS connections are generally considered "expensive" and should usually be pooled.
In a pooled use-case the number of connections in the pool is really defined by your performance requirements, hardware capacity, etc. It's usually refined via benchmarking and tuning. There are almost always trade-offs between performance and resource utilization so it's your job to find the right balance. Even general recommendations are hard to make for pool size because the use-cases can vary so widely.
In a non-pooled use-case it's worth noting that a single JMS connection can service multiple sessions concurrently (although the sessions themselves can't be used concurrently). JMS sessions are much less expensive to create than JMS connections. Therefore, in a non-pooled use-case it generally makes sense to reduce the number of JMS connections to be as low as reasonably possible and make use of multiple sessions.
Aside from a few bits of setup for the connection all substantive JMS client work is done through the session. JDBC connections have no such "session" idea. Most of the work is done directly on the connection object. Therefore, I'm not sure the comparison here is really valid.