I'm trying to get the dropbear ssh C code to build a library instead of a binary. Dropbear is built from generating a configure from configure.ac using autoconf. The configure is then used to generate a Makefile that builds executables. I would like to create a target for the generated Makefile that creates a statically linked library so that my application can link it in and have access to dropbear functionality.
Is there a straight forward way to modify the configure.ac to achieve this goal?
No. There is, however, a straightforward way to modify the Makefile.in for this. Makefile.in is the template for the Makefile; the configure script does little more than replace expressions encased in @ in the Makefile.in with values it deduced during its run, so you can add rules to Makefile.in very nearly the same way you would to a plain Makefile.
In this particular case, you could add the following:
dropbearlibobjs = $(filter-out svr-main.o,$(dropbearobjs))
libdropbear.a: $(dropbearlibobjs) $(HEADERS) $(LIBTOM_DEPS) Makefile
ar rcs $@ $(dropbearlibobjs)
Most of this is taken from the dropbear: target, except the recipe builds a static library and the translation unit that contains the main function is filtered out. You can then use
./configure
make libdropbear.a
to build the library.
I am not certain how easily this library can be integrated with a new program, though. The authors of dropbear almost certainly did not consider this use case when they wrote it, so portions of the code may depend on global state (such as signal handlers) that your application cannot provide, and generally there's no documented API on which you can depend. I don't know what exactly you plan to do, but this path is unlikely to be easy and straightforward.
Perhaps you would rather take a look at libssh or libssh2? (Interestingly, the former seems to be more recent than the latter)