Search code examples
x86x86-64obfuscation

Is this fundamental premise for x86 binary obfuscation accurate? (That only system calls and their arguments matter to the outcome of the program)

This pertains to code running in user-mode. For simplicity of the question, let's say we've taken any shared libraries / OS APIs called on by a program and statically linked them in memory, so we're not calling out to any abstraction layer, just using system calls directly.

I'm doing an experiment to achieve binary obfuscation by identifying certain non-mutable states (I think system calls and jumps) without having to build a relatively advanced obfuscation approach. Basically I emulate the program storing state changes for every instruction executed. When I reach a system call or jump, I mark that as a boundary and every instruction executed in-between two boundaries as a "function". My theory is that in user-mode programs, system calls are the only states (by states I mean instructions being executed along with the register and stack state at that instant) that have an "effect" outside of the program. In other words, whatever changes you want your program to make to the system in user-mode, system calls are the only means with which that happens.

If I'm wrong about that stop me here.

So based on that understanding, I hypothesize that I can mutate each of those functions in an almost infinite number of ways so long as the resulting instructions lead to the same state at the end of the function so that the system call arguments remain identical. Of course control flow also needs to be preserved so I treat jumps as preserved states too. I'm achieving this by using a Monte Carlo Tree Search to solve for the desired state from the mutated state. In order words, if I follow those rules and patch that mutated program back together and update all the jumps to reach the same functions they were previously pointing to, my program should externally perform the same goal it originally did, but through different instructions.

Here's a visual diagram. If zoomed in on, it's legible but SO's compression makes it a bit fuzzy.

enter image description here

This concept only intends to obfuscate memory, register, and instruction-sequence analysis channels (and without further specifications, would not obfuscate them 'entirely')

Is my premise flawed?

Solution

  • After mmap(MAP_SHARED, PROT_READ|PROT_WRITE), writing to memory will eventually affect contents of files on disk, and/or be visible to other processes reading the shared memory region. (Most usage of shared-memory for inter-process communication is not via disk files, but rather anonymous SHM like shmat(2) or shm_open(3))

    Many programs don't do that for any files, but it's somewhat common to use shared memory for IPC (especially to an X server).

    Still, it probably doesn't fully invalidate your argument about reg/mem state before/after a syscall being the only relevant thing most of the time.

    Multi-threaded programs communicate with themselves via memory; threads share their entire address-space, and order of load/store instructions can be significant for memory ordering. And for atomicity, whether a 4-byte load is done with one dword mov or four byte loads that you later merge. Atomic pure-load and pure-store are just plain instructions, only atomic RMW needs a lock prefix (assuming you're compiling to run on a multi-core machine, otherwise the lock prefix can be omitted.)