Why does rl_bind_key with custom function fail in readline on OSX/macOS

Question

I built a shell that tries to make the tab (\t) key do something custom using rl_bind_key(), but it didn't work in macOS Sierra, but it works on Ubuntu, Fedora, and CentOS. Here's the mcve:

#include <stdlib.h>
#include <stdio.h>
#include <readline/readline.h>

static int cmd_complete(int count, int key)
{
        printf("\nCustom tab action goes here...\n");
        rl_forced_update_display();
        return 0;
}

char *interactive_input()
{
        char *buffer = readline(" > ");
        return buffer;
}

int main(int argc, char **argv)
{
        rl_bind_key('\t', cmd_complete); // this doesn't seem to work in macOS
        char *buffer = 0;
        while (!buffer || strncmp(buffer, "exit", 4)) {
                if (buffer) { free(buffer); buffer=0; }
                // get command
                buffer = interactive_input();
                printf("awesome command: %s\n", buffer);
        }
        free(buffer);
        return 0;
}

I compile using Clang like this:

$ cc -lreadline cli.c -o cli

What is the cause of this behavior and how do I fix it?

Answer 1

I was using the flag -lreadline, however unbeknownst to me, Clang appears to secretly use libedit (I've seen it called editline also). In libedit, for some reason (which merits another question), rl_bind_key appears to not work with anything except rl_insert.

So one solution that I found is to use Homebrew to install GNU Readline (brew install readline), and then to ensure I use that version, I compile thusly:

$ cc -lreadline cli.c -o cli -L/usr/local/opt/readline/lib -I/usr/local/opt/readline/include

In fact, when you install readline, it will tell you this at the end of the installation or if you do brew info readline:

gns-mac1:~ gns$ brew info readline
readline: stable 7.0.3 (bottled) [keg-only]
Library for command-line editing
https://tiswww.case.edu/php/chet/readline/rltop.html
/usr/local/Cellar/readline/7.0.3_1 (46 files, 1.5MB)
  Poured from bottle on 2017-10-24 at 12:21:35
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/readline.rb
==> Caveats
This formula is keg-only, which means it was not symlinked into /usr/local,
because macOS provides the BSD libedit library, which shadows libreadline.
In order to prevent conflicts when programs look for libreadline we are
defaulting this GNU Readline installation to keg-only..

For compilers to find this software you may need to set:
    LDFLAGS:  -L/usr/local/opt/readline/lib
    CPPFLAGS: -I/usr/local/opt/readline/include

Source of libedit rl_bind_key

So this is why it doesn't work in libedit. I downloaded the source and this is how the rl_bind_key function is defined:

/*
 * bind key c to readline-type function func
 */
int
rl_bind_key(int c, rl_command_func_t *func)
{
    int retval = -1;

    if (h == NULL || e == NULL)
        rl_initialize();

    if (func == rl_insert) {
        /* XXX notice there is no range checking of ``c'' */
        e->el_map.key[c] = ED_INSERT;
        retval = 0;
    }
    return retval;
}

So it seems designed to not work with anything except rl_insert. That seems like a bug, not a feature. I wish I knew how to become a contributor to libedit.