I need to design a web service layer on top of an existing application. Really, I'm most interested in WS-Security and JMS support, possibly JTA.
What is the best server to use for this? Would it be a wise decision to go with lightweight Tomcat (which I'm experienced with), or would I be better off with a server that implements the full Java EE stack, such as JBoss AS, WebLogic, or even GlassFish?
Generally, unless you know that you're trying to use Java EE services particularly, you're better off with just Tomcat. The Java EE environment is very heavyweight; J2EE was designed with really big environments like eBay in mind. (I wrote Sun's first J2EE architecture course; I've been following these arguments for a long time.)
But then, you say "JMS" which is strictly a Java EE service.