Is it possible to have any kind of payload when using JOSE?
I was thinking of doing something like: {"alg":"ES512", "cty":"XML" }
and just make a string out of the XML file, on the server side I check cty and create the XML.
I'm pretty sure its possible, but I didn't see any example so I started thinking that maybe that's not the idea behind jose and that cty is only meant to indicate payload is a JWT or something related to jose.
As you can see from http://www.rfc-editor.org/rfc/rfc7515.txt:
JWS Payload
The sequence of octets to be secured -- a.k.a. the message. The payload can contain an arbitrary sequence of octets.
So your hunch is right. And as indicated in https://www.rfc-editor.org/rfc/rfc7515#section-4.1.10, the cty element can be used to indicate XML content following http://www.iana.org/assignments/media-types/media-types.xhtml with the ommission of the application/ prefix. So xml would be appropriate in your case but in (the regular) case that both the sender and the receiver know that they're going to exchange XML, the cty element can be omitted.
For JWE's it is the same. There's an example with non-JSON plaintext here: https://www.rfc-editor.org/rfc/rfc7516#appendix-A.3