I'm trying to decode a STUN success response basing on RFC 5389:
If the IP address family is IPv6, X-Address is computed by taking the mapped IP address in host byte order, XOR'ing it with the concatenation of the magic cookie and the 96-bit transaction ID, and converting the result to network byte order.
Magic cookie is a constant and it is 0x2112A442.
Transaction ID in my case is: 0x6FA22B0D9C5F5AD75B6A4E43.
My X-Address (IPv6) in Host Byte Order is:
0x034A67D82F4B3657B193039A8BA8FDA1
Do I have to xor Host Byte Order X-Address with the concatenation of Magic Cookie and Transaction ID in Network or Host Byte Order?
In the first case, Network Byte Order concatenation is equal to:
0x2112A442 6FA22B0D9C5F5AD75B6A4E43
The first byte 0x03 is xored with 0x21, the last byte 0xA1 is xored with 0x43
But in the second case, the Host Byte Order concatenation is:
0x434E6A5BD75A5F9C0D2BA26F 42A41221
The first byte 0x03 is xored with 0x43, the last byte 0xA1 is xored with 0x21.
Another possible behavior is if it takes and converts separately Magic cookie and Transaction ID to Host Byte Order but it concatenate them preserving header order:
0x42A41221 434E6A5BD75A5F9C0D2BA26F
The first byte 0x03 is xored with 0x42, the last byte 0xA1 is xored with 0x6F.
Everything is done in Network Byte Order.
But here's the thing, for IPv6 addresses, there is no difference between "host byte order" and "network byte order". IPv6 addresses are always understood to be an array of 16 bytes. And individual bytes don't have a "byte order". In "C" code we'd just express that IPv6 address as:
unsigned char ipv6address[16];
Or in terms of the sockaddr_in6 struct;
struct sockaddr_in6 addr;
unsigned char* ipv6addresss = addr.sin6_addr.s6_addr; // points to a sequence of 16 bytes
Contrast that with IPv4, which are often passed around in code as 32-bit integers. In the IPv4 case, you often wind up having to invoke the htonl and ntohl functions.
Unless you are doing something like maintaining the the IPv6 address as an array of 8 16-bit integers instead of an array bytes, then you shouldn't have to think about endianness and byte order too much. (As a matter of fact, I'd encourage you to not think about byte order with regards to 16-byte ip addresses).
Example:
My IPv6 address is this:
2001:0000:9d38:6abd:347d:0d08:3f57:fefd
As an array of hex bytes that's logically written out as:
200100009d386abd347d0d083f57fefd
When my STUN server receives a binding request from this IPv6 address, it applies the following XOR operation to send back the XOR-MAPPED-ADDRESS. Let's assume it's the same transaction id as yours and it includes the magic cookie to indicate RFC 5389 support (2112A442 6FA22B0D9C5F5AD75B6A4E43)
XOR:
200100009D386ABD347D0D083F57FEFD
2112A4426FA22B0D9C5F5AD75B6A4E43
The result:
0113A442F29A41B0A82257DF643DB0BE
Similarly, the client receiving the STUN binding response applies the inverse XOR operation on this byte array with the same transaction id.
XOR:
0113A442F29A41B0A82257DF643DB0BE
2112A4426FA22B0D9C5F5AD75B6A4E43
The result:
200100009D386ABD347D0D083F57FEFD
You can reference the source code to Stuntman if that helps for an example of how to apply the xor mapping operation. The XOR operation is here on GitHub. The code here doesn't distinguish between transaction ids with a magic cookie and those that don't. It just treats the transaction id as a logical sequence of 16 bytes.
The above takes care of the IPv6 address. But the 16-bit port value does have to get byte flipped if it is getting treated as a short or 16-bit integer. In C code, that's typically handled with a call to ntohs.