Search code examples
matlablinkerg++mex

Control mex link options with g++ from command line

I'm trying to link a library using mex from command line, or more exactly, from a makefile. I do this from a Makefile which I post here:

BDDM_MATLAB = @matlabhome@

MEXCC = $(BDDM_MATLAB)/bin/mex
MEXFLAGS = -v -largeArrayDims -O
MEXEXT = mexa64

TDIR = $(abs_top_srcdir)/test
IDIR = $(abs_top_srcdir)/src
LDIR = $(abs_top_srcdir)/lib

LOP1 = $(CUDA_LDFLAGS) $(LIBS)

SOURCES := $(wildcard *.cpp)
OBJS = $(SOURCES:.cpp=.o)
mTESTS = $(addprefix $(TDIR)/, $(SOURCES:.cpp=.$(MEXEXT)))

all: $(TDIR) $(mTESTS)

$(OBJS) : %.o : %.cpp
    $(MEXCC) $(MEXFLAGS) -c -outdir ./ -output $@ $(CUDA_CFLAGS) -I$(IDIR) CFLAGS="\$$CFLAGS -std=c99" $^

$(mTESTS) : $(TDIR)/%.$(MEXEXT) : %.o
    $(MEXCC) $(MEXFLAGS) -L$(LDIR) -outdir $(TDIR) $^ $(LOP1) -lmpdcm LDFLAGS="-lcudart -lcuda"

.PHONY = $(TDIR)

$(TDIR):
    $(MKDIR_P) $@

clean:
    $(RM) *.o

libmpdcm is a static library that includes calls to two shared libraries libcuda and libcudart. My environment has

export LD_LIBRARY_PATH=/usr/local/cuda-7.0/lib64:$LD_LIBRARY_PATH:

My make rule produces

/usr/local/MATLAB/R2014a/bin/mex -v -largeArrayDims -O -L/home/eaponte/projects/test_cpp/lib -outdir /home/eaponte/projects/test_cpp/test test_LayeredEEG.o -L/usr/local/cuda/lib64 -lcudart -lcuda  -lmpdcm LDFLAGS="-lcudart -lcuda"

This produces the following g++ command:

/usr/bin/gcc -lcudart -lcuda -shared  -O -Wl,--version-script,"/usr/local/MATLAB/R2014a/extern/lib/glnxa64/mexFunction.map" test_LayeredEEG.o  -lcudart  -lcuda  -lmpdcm   -L/home/eaponte/projects/test_cpp/lib  -L/usr/local/cuda/lib64   -L"/usr/local/MATLAB/R2014a/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -o /home/eaponte/projects/test_cpp/test/test_LayeredEEG.mexa64

The problem is that afterwards I get a linking error in Matlab:

Invalid MEX-file '/home/eaponte/projects/test_cpp/test/test_Fmri.mexa64': /home/eaponte/projects/test_cpp/test/test_Fmri.mexa64: undefined symbol: cudaFree

I know that the solution is simply to put the cuda libraries at the end of the g++ command

/usr/bin/gcc -lcudart -lcuda -shared  -O -Wl,--version-script,"/usr/local/MATLAB/R2014a/extern/lib/glnxa64/mexFunction.map" test_LayeredEEG.o  -lmpdcm   -L/home/eaponte/projects/test_cpp/lib  -L/usr/local/cuda/lib64   -L"/usr/local/MATLAB/R2014a/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -lcudart  -lcuda -o /home/eaponte/projects/test_cpp/test/test_LayeredEEG.mexa64

How can achieve that running mex from command line (or from a Makefile)?

Solution

  • Just to illuminate the problem and solution and offer some help in avoiding the like:

    The fundamental rule of linkage with the GNU linker that your problem makefile transgressed is: In the commandline sequence of entities to be linked, ones that need symbol definitions must appear before the ones that provide the definitions.

    An object file (.o) in the linkage sequence will be incorporated entire in the output executable, regardless of whether or not it defines any symbols that the executable uses. A library on the other hand, is merely examined to see if it provides any definitions for symbols that are thus-far undefined, and only such definitions as it provides are linked into in the executable (I am simplifying somewhat). Thus, linkage doesn't get started until some object file is seen, and any library must appear after everything that needs definitions from it.

    Breaches of this principle usually arise from inept bundling of some linker flag-options and some library-options together into a make-variable and its placement in the linkage recipe, with the result that the bundled options are interpolated at a position that is valid for the flags but not valid for libraries. This was so in your problem makefile, with LOP1 the bad bundle.

    In the typical case, the bundling causes all of the libraries to be placed before all the object files, and never mentioned again. So the object files yield undefined symbol errors, because the libraries they require were seen by the linker before it had discovered any undefined symbols, and were ignored. In your untypical case, it resulted in libcudart and libcuda being seen later than your only object file test_LayeredEEG.o - which however required no symbols from them - but earlier than the only thing that did require symbols from them, the library libmpdcm. So they were ignored, and you built a .mex64 shared library that had not been linked with them.

    Long ago - pre-GCC 4.5 - shared libraries (like libcudart and libcuda) were exempt from the requirement that they should be needed, at the point when the linker sees them, in order to be linked. They were linked regardless, like object files, and the belief that this is so has not entirely died out. It is not so. By default, shared libraries and static libraries alike are linked if and only if needed-when-seen.

    To avoid such traps it is vastly helpful to understand the canonical nomenclature of the make variables involved in compilation and linkage and their semantics, and their canonical use in compilation and linkage recipes for make. Mex is a manipulator of C/C++/Fortran compilers that adds some commandline options of its own: for make purposes, it is another compiler. For the options that it inherits from and passes to the underlying compiler, you want to adhere to the usage for that compiler in make recipes.

    These are the make variables most likely to matter to you and their meanings:

    • CC = Your C compiler, e.g. gcc
    • FC = Your Fortran compiler, e.g. gfortran
    • CXX = Your C++ compiler, e.g. g++.
    • LD = Your linker, e.g. ld. But you should know that only for specialized uses should the linker be directly invoked. Normally, the real linker is invoked on your behalf by the compiler. It can deduce from the options that you pass it whether you want compiling done or linking done, and will invoke the appropriate tool. When you want linking done, it will quietly augment the linker options that you pass with additional ones that it would be very tiresome to specify, but which ensure that the linkage acquires all the the correct flags and libraries for the language of the program you are linking. Consequently almost always, specify your compiler as your linker.
    • AR = Your archiving tool (static library builder)
    • CFLAGS = Options for C compilation
    • FFLAGS = Options for Fortran compilation
    • CXXFLAGS = Options for C++ compilation
    • CPPFLAGS = Options for the C preprocessor, for any compiler that uses it. Avoid the common mistake of writing CPPFLAGS when you mean CXXFLAGS
    • LDFLAGS = Options for linkage, N.B. excluding library options, -l<name>
    • LDLIBS = Library options for linkage, -l<name>

    And the canonical make rules for compiling and linking:

    C source file $< to object file $@:

    $(CC) $(CPPFLAGS) $(CFLAGS) -c $@ $<

    Free-from Fortran file $< to object file $@, with preprocessing:

    $(FC) $(CPPFLAGS) $(FFLAGS) -c $@ $<

    (without preprocessing, remove $(CPPFLAGS))

    C++ source file $< to object file $@:

    $(CXX) $(CPPFLAGS) $(CXXFLAGS) -c $@ $<

    Linking object files $^ into an executable $@:

    $(<compiler>) $(LDFLAGS) -o $@ $^ $(LDLIBS)

    If you can as much as possible write makefiles so that a) you have assigned the right options to the right variables from this glossary, and b) used the canonical make recipes, then your path will be much smoother.

    And BTW...

    Your makefile has the following bug:

    .PHONY = $(TDIR)

    This is apparently an attempt to make $(TDIR) a phony target, but the syntax is wrong. It should be:

    .PHONY: $(TDIR)

    what the assignment does is simply create a make variable called, .PHONY with the value of $(TDIR), and does not make $(TDIR) a phony target.

    Which is fortunate, because $(TDIR) is your output directory and not a phony target.

    You wish to ensure that make creates $(TDIR) before you need to output anything into it, but you do not want it to a normal prequisite of those artefacts, which would oblige make to rebuild them whenever the timestamp of $(TDIR) was touched. That is presumably why you thought to make it a phony target.

    What you actually want $(TDIR) to be is an order-only prerequsite of the $(mTESTS) that will be output there. The way to do that is to amend the $(mTESTS) rule to be:

    $(mTESTS) : $(TDIR)/%.$(MEXEXT) : %.o | $(TDIR)

    This will cause $(TDIR) to be made, if needed, before $(mTESTS) is made, but nevertheless $(TDIR) will not be considered in determining whether $(mTESTS) does need to be made.

    On the other hand, the targets all and clean are phony targets: no such artefacts are to be made, so you should tell make so with:

    .PHONY: all clean