How does the endian_tag in dexes file actually work? Documentation states that
The constant ENDIAN_CONSTANT is used to indicate the endianness of the file in which it is found. Although the standard .dex format is little-endian, implementations may choose to perform byte-swapping. Should an implementation come across a header whose endian_tag is REVERSE_ENDIAN_CONSTANT instead of ENDIAN_CONSTANT, it would know that the file has been byte-swapped from the expected form.
suggesting that if there is REVERSE_ENDIAN_CONSTANT the file is supposed to use big-endian. It this correct? Moreover -- what part of the file is supposed to use this big-endian -- whole file including header_items?
The big-endian flag would apply to any multi-byte value anywhere in the file. You can see the exact set of things it applies to by looking at the DexSwapVerify source, which is responsible for converting all multi-byte values from the file's byte order to the host's byte order. This is done so that other parts of the VM or tools don't have to swap every time they access a value.
Most Android devices are little-endian ARM or x86, so the swap operations are no-ops. Some Android OS developers used PPC Macs in the early days, and there are some big-endian devices (e.g. MIPS), so the swap code does matter.
One line of particular interest is this:
if (pHeader->endianTag != kDexEndianConstant) {
ALOGE("Unexpected endian_tag: %#x", pHeader->endianTag);
return false;
}
Note it's not checking for the reverse endian constant -- the code does not expect to find a big-endian DEX file. "Raw" DEX files should be little-endian -- the format provides a way for them to be big-endian, but I would expect most tools to choke on them.
An optimized DEX (.odex) file is stored in host byte order, so the byte-swapping step is skipped when reading from .odex. Optimized DEX files are only expected to be read on the system that generated them.