Ruby extension wrapper around C library fails to load installed library

Question

The Ruby extension davenport-ruby to the C library davenport will not load properly on Ubuntu and Debian. It works alright on a development machine (MacOS), as demonstrated by the README of the smoke test Ruby project, dvt

The RubyGems loader (via bundler) compiles and installs the extension as follows:

~/dvt$ rm -rf ~/.bundle
~/dvt$ bundle install
Fetching gem metadata from https://rubygems.org/.
Using bundler 2.0.2
Fetching davenport 1.0.2.pre
Installing davenport 1.0.2.pre with native extensions
Bundle complete! 1 Gemfile dependency, 2 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
~/dvt$ bundle info davenport
  * davenport (1.0.2.pre)
    Summary: Ruby binding for the Davenport library
    Homepage: https://github.com/wbreeze/davenport-ruby
    Path: /home/deploy/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0\
      /gems/davenport-1.0.2.pre
~/dvt$ ls /home/deploy/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0\
      /gems/davenport-1.0.2.pre/lib/davenport_ruby/
davenport_ruby.so
~/dvt$

However attempting to run the program yields:

deploy@localhost:~/dvt$ ruby test.rb
Traceback (most recent call last):
    6: from test.rb:1:in `<main>'
    5: from /home/deploy/.rbenv/versions/2.6.3/lib/ruby/2.6.0/rubygems\
       /core_ext/kernel_require.rb:34:in `require'
...
    1: from /home/deploy/.rbenv/versions/2.6.3/lib/ruby/2.6.0/rubygems\
       /core_ext/kernel_require.rb:54:in `require':\
  libdavenport.so.0: cannot open shared object file: No such file or\
    directory - /home/deploy/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0\
      /gems/davenport-1.0.2.pre/lib/davenport_ruby/davenport_ruby.so\
(LoadError)

The library file, libdavenport.so.0 exists in /usr/local/lib. Making that part of the load path with, ruby -I /usr/local/lib test.rb yields the same result.

The library file, davenport_ruby.so exists in /home/deploy/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/davenport-1.0.2.pre/lib/davenport_ruby as shown here:

/home/deploy/.rbenv/versions/2.6.3/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require': libdavenport.so.0: cannot open shared\
object file: No such file or directory - /home/deploy/.rbenv/versions/2.6.3\
/lib/ruby/gems/2.6.0/gems/davenport-1.0.2.pre/lib/davenport_ruby\
/davenport_ruby.so (LoadError)
~/dvt$ ls -l ~/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/\
davenport-1.0.2.pre/lib/davenport_ruby/
total 72
-rwxr-xr-x 1 deploy deploy 69640 Jul 25 17:02 davenport_ruby.so
~/dvt$

The published 1.0.2.pre version of the Ruby davenport extension comes from this PR which link shows all of the code bits (and quite a lot of experiments). Is it some detail or piece of the extension gem not quite right?

davenport_ruby.so

The .so file is the shared library file compiled upon installation of the gem. It isn't present in the contents of the gem:

~/davenport-ruby/dltem[i1]$ gem unpack davenport-1.0.2.pre.gem
Unpacked gem: '/Users/dcl/davenport-ruby/dltem/davenport-1.0.2.pre'
~/davenport-ruby/dltem[i1]$ ls -R
davenport-1.0.2.pre davenport-1.0.2.pre.gem

./davenport-1.0.2.pre:
History.txt README.rdoc Rakefile    ext     lib

./davenport-1.0.2.pre/ext:
davenport_ruby

./davenport-1.0.2.pre/ext/davenport_ruby:
davenport_ruby.c    extconf.rb

./davenport-1.0.2.pre/lib:
davenport.rb
~/davenport-ruby/dltem[i1]$

When RubyGems installs the gem, does it detect the s.extensions << 'ext/davenport_ruby/extconf.rb' in the gem spec and execute that file with ruby? Does it run make for the resulting make file?

What will enable ruby test.rb to run without the error?

This gist contains a comparison of the unique global symbols in libdavenport.so.0 and davenport_ruby.so, and output from the gem environment and ruby -e 'puts $:.join("\n")' commands.

ldd

This question about the not found (LoadError) problem suggests using ldd to verify the linking. (There is a related question that is unanswered, but with a similar suggestion in the comment.)

Indeed, the extension library, although built, does not link to the installed libdavenport.so.0:

~/dvt$ bundle install
Fetching gem metadata from https://rubygems.org/.
Using bundler 2.0.2
Fetching davenport 1.0.2.pre
Installing davenport 1.0.2.pre with native extensions
Bundle complete! 1 Gemfile dependency, 2 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
~/dvt$ ldd ~/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/davenport-1.0.2.pre/lib/davenport_ruby/davenport_ruby.so
    linux-vdso.so.1 (0x00007fff2d3e3000)
    libdavenport.so.0 => not found
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f9a5be42000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9a5baa3000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f9a5c349000)
~/dvt$

ldconfig

This very old question over on AskUbuntu suggests the use of the ldconfig command. Since libdavenport.so.0 is installed in /usr/local/lib, the command was ldconfig /usr/local/lib. However to run it required a different login that can get root privileges. Switching back to the deploy account that is doing the installation of the Ruby program:

~/dvt$ ruby test.rb
Hola
[1, 3, 2, 4]
~/dvt$

Voila. It is now working. The question becomes how to get that to work with the deploy/install, and with a user that does not have root privileges. (Installing the library libdavenport required root privileges as well, although it was compiled and installed from source.)

Answer 1

Running ldconfig /usr/local/lib as root turns out to be the answer. It causes the loader to be able to locate the installed libdavenport.so.0 library at runtime.

To the follow-on questions, the answer is that the make install probably won't be able to eliminate the need to run ldconfig on some systems. There is an old issue against another library that made the attempt, still open, at esnet/iperf. They didn't figure it out. If they did, they didn't update the issue. A newer issue against libcheck/check has similar struggles.

The current Gnu documentation for libtool has open in its implementation issues,

The install Makefile target should warn the package installer to set the proper environment variables (LD_LIBRARY_PATH or equivalent), or run ldconfig.

The "solution" in this case, is a documentation solution for the Davenport library, an update to the README made with this PR.

There's a nice short explanation about library linking and loading at Cprogramming.com (which is ad' supported but not painfully so; the article earns the reference).