Can you please let me know what are difference between websphere MQ class for java vs Websphere MQ classes for JMS?
Which of the above should I use to connect to an IBM MQ client for performing distributed transactions (XA)?
The JMS API is the JavaEE open standard for a messaging API. IBM MQ v8 now implements the latest version 2.0 JMS standard (MQ 8.0.0.2 has the JMS2.0 errata as well).
WMQ Java classes is a proprietary API to provide messaging; it's OO based around classic MQ objects, such as QueueManagers and Queues etc. There are some features such as support for MQ Multi-instance QMs and async call backs that it doesn't have.
For any new application I would recommended that you use JMS; if you encounter function that you really need that isn't in JMS please get in contact (RFE).
As to XA; what is the transaction co-coordinator that you wish to use? Via JMS you can use a JavaEE server to do the XA co-ordination. Non JavaEE are not supported (though there is an RFE for thinks like Atomikos to be supported).
What is the scenario here? Is it full XA that is required or would a single phase transacted JMS Session be sufficient.
Via the Java Classes you can only co-ordinate messaging work via the MQ QM. DB2/Oracle can also be co-ordinated.