Search code examples
luatorch

Require custom C module in Lua by path


This is a problem that plagued me for quite a while.

After I wrote and compiled my custom functions into a shared library, require('mylib') would only work when the shared library was directly in the same directory as the script I called it from.

Any efforts to try require('/path/to/mylib') or similar absolute paths failed. Furthermore, "backtracking" through a relative path (i.e. using .. failed as well.

So how can one specify the bin directory, or wherever the shared library is output to?


Solution

  • Well according to the Lua documentation on the require call (using Lua 5.2 here), there are a few places the loader looks for these loadable modules.

    It seems that require() uses what are called "searchers" (docs linked to in above) to determine where to find these modules. There are four searchers in total. From the docs:

    The first searcher simply looks for a loader in the package.preload table.

    The second searcher looks for a loader as a Lua library, using the path stored at package.path. The search is done as described in function package.searchpath.

    The third searcher looks for a loader as a C library, using the path given by the variable package.cpath. Again, the search is done as described in function package.searchpath. For instance, if the C path is the string "./?.so;./?.dll;/usr/local/?/init.so" the searcher for module foo will try to open the files ./foo.so, ./foo.dll, and /usr/local/foo/init.so, in that order. Once it finds a C library, this searcher first uses a dynamic link facility to link the application with the library. Then it tries to find a C function inside the library to be used as the loader. The name of this C function is the string "luaopen_" concatenated with a copy of the module name where each dot is replaced by an underscore. Moreover, if the module name has a hyphen, its prefix up to (and including) the first hyphen is removed. For instance, if the module name is a.v1-b.c, the function name will be luaopen_b_c.

    The fourth searcher tries an all-in-one loader. It searches the C path for a library for the root name of the given module. For instance, when requiring a.b.c, it will search for a C library for a. If found, it looks into it for an open function for the submodule; in our example, that would be luaopen_a_b_c. With this facility, a package can pack several C submodules into one single library, with each submodule keeping its original open function.

    The searcher of use to us is the third one: it is used for any shared libraries (.dll or .so) which is generally how our custom C modules are built.

    Using the template string (the one with the question marks), the searcher will look in each of the specified paths, substituting the argument of require() in place of the question mark. In order to specify the path for this third searcher, one must set (or append to) package.cpath and then call require().

    So perhaps you have a directory structure as

    - ROOT
        |-lua
        |-bin
    

    where lua contains script.lua and bin contains mylib.so

    To load mylib.so, you just need these two lines of code in script.lua:

    package.cpath = '/ROOT/bin/?.so;' .. package.cpath
    libfuncs = require('mylib')
    

    NOTE: Notice the semicolon. If you append (as opposed to the prepending above), make sure to lead with the semicolon on your added path. It is not there buy default. Otherwise your new path will be merged to the current default cpath, which is just ./?.so.