I am interested in precautionary measures you could take to make reverse engineering a C shared library more difficult
It seems that it is impossible to prevent entirely but you can take steps that might make it take longer and therefore be less attractive/more costly to do
Things like code obfuscation, compiler optimisation options compiler debug flags etc. (I would be specifically interested in the gcc compiler)
Also interested in any thoughts on the relative difficulty of reverse engineering a ~5KLOC C shared library (.so size of ~200kb) in terms of $ or man hours etc.
Those are some tricks that I know of:
Use strip with the --strip-unneeded option to remove all symbols except the ones that are needed for relocation (since we're talking about a shared library):
strip --strip-unneeded libmylib.so
Link with the standard libc statically, this results in a bigger library/binary which means more code to go through and also obfuscation since it will be more difficult to separate your functions from the library functions:
gcc -static ...
Force the compiler to inline small functions, this embeds small functions in your code instead of calling them, so it makes it almost impossible to distinguish between your code and the standard library code.
Having said that, I must add that I've had a binary that used the above measures and I was able to reverse engineer it in a few hours, I started by rebuilding the symtab if you're curious, and I'm no reverse engineering guru.