Why aren't nested transactions supported by JTA? Is it because of the complexity of implementing them (which I doubt) or some design principle?
(As @Piotr Nowicki points out, JTA does allow nested transactions, but this is optional not mandatory.)
Why? This is one of those questions that is impossible to answer with any certainty, unless you were one of the people "in the room" when the decisions were made.
It could be the inherent complexity of including nested transactions as part of the spec. Or apparent complexity at the time; i.e. they weren't sure they knew how to do a good job of specifying them.
It could be that they thought there wasn't enough demand.
It could be time pressure ... or just plain exhaustion.
It could be "commercial reasons"; e.g. certain vendors not wanting to interfere with product launch schedules by expanding the scope of the spec.
But, the bottom line is that if you want the real answer, you would need to ask the people in the working group that wrote JTA specifications. (And I doubt they would tell you ... on the record.)