Do I need to consider endianness when serializing/deserializing data to write/read binary to/from the same machine? There's no network communication involved.
In what cases do I need to consider byte order?
Always when de-/serialising. The reader has to interpret the bytes in the same order as the writer.
When the processes are on the same system, it is generally safe to use the native byte order. That is not safe when multiple systems may be involved because the native byte order may differ.