I have an executable project prog on autotools build-system. This program links against libfoo which is also an autotools project.
I want to give my users (who wish to compile prog) the freedom to choose between linking against a previously installed libfoo, or to configure, compile (static linkage) and link to the libfoo source tree which is shipped within package (in third-party directory)
libfoo uses libtool and pkg-config if it matters...
Ideally this would be done with some autoconf arguments:
./configure --with-local-libfoo
vs
./configure --with-system-libfoo
The question is how to tell the buildsystem to compile libfoo before prog (in second case) and how to provide correct link flags?
Since libfoo is also an Autotools project, a copy of which is distributed inside the main project, it would be natural to set it up as a subproject. The key Autoconf macro here is AC_CONFIG_SUBDIRS. Thus, if the libfoo package were in a subdirectory libfoo/ of the directory containing the top-level configure script, the top-level configure.ac would use
AC_CONFIG_SUBDIRS([libfoo])
In that case, the top-level configure script will run libfoo's configure script automatically, passing it all the same options it received (including any that are meaningful only to libfoo). Moreover, the top-level script's --help option will print both its own and libfoo's options. You can make that conditional on a --with-* argument (but just one, don't use two different ones).
On the Automake side, look into Automake conditionals and conditional subdirectories. You can use conditionals, with the condition controlled by your --with-system-libfoo or whatever, to select the right set of link options and to determine whether make will recurse into the libfoo package. When you're building your own copy of libfoo, it might be helpful to include the -static-libtool-libs option among the main program's link flags.
The last bits revolve around preventing the local libfoo, when you build it, from being installed on the system along side the main program. I haven't come up with a good solution that doesn't involve tweaking libfoo's build system. Do switch libfoo.la to be a convenience library (named in noinst_LTLIBRARIES instead of in lib_LTLIBRARIES or such) and in fact generally switch build targets in the libfoo subpackage from installed ones to noinst_ ones.