As far as a program is concerned, a stack is just a bunch of memory. How does the backtrace function determine which stack bytes are instruction pointer return addresses as opposed to which are just function arguments, etc?
As far as a program is concerned, a stack is just a bunch of memory.
That is correct. If you knew nothing about the structure of the program, you wouldn't be able to unwind its stack.
The easiest structure to use for unwinding is one where a single "frame pointer" register is reserved to always point to the current frame, and the frame contains previous frame pointer at a known offset.
That is the mechanism that was used for a long time on i*86, and is still used on many RISC machines. It makes stack unwinding easy and very fast, but it is not efficient on register-starved machines, such as i*x86 and to lower extent x86_64, because that frame register can be put to better use elsewhere.
The solution is to create augmentation data, which describes how to find the current frame given current set of registers and memory contents (e.g. "current frame is at offset NN from $rsp", or "at offset MM from $rbp", "previous frame is at offset J from current frame", etc.)
That is effectively what .eh_frame is. Remove it from your binary, and no stack unwinding will be possible.
This mechanism is also significantly slower than a simple frame pointer walk, requires complicated code, and is error-prone (compilers are not guaranteed to emit correct unwind descriptors, and hand-coded assembly may not have them at all).
There are other solutions which make unwinding fast and easy, but they require a different calling convention.