Mac device access works in C, but equivalent code in Java/JNA not

Question

We work with a serial device that is attached to a Mac on USB and need to configure the DTR/RTS-line settings. Technically this involves usage of open(3), ioctl(3).

We implemented this in C and it worked. Below is a very simplified snippet showing the core part.

Then we migrated the code to Java/JNA and run into the problem that the ported code did not work, though it is basically a line-by-line conversion of the C code.

The question is where do we go wrong?

Symptom of the failure in Java is errno=25 'Inappropriate ioctl for device' returned from the call to ioctl(). Since it works in C it seems that we do somwthing wrong in JNA.

What we did:

• Checked the values from the header constants. Note that the C-code generates Java-compatible constant definitions that are used in the Java code.
• Checked the signature of ioctl(). Seems to be right according to the man-page and include files.
• Guessed that the problem is that the ioctl code for TIOCMSET is not properly passed since it is negative.

We use JNA 5.5.0.

Here comes the C code. The snippet simply reads the settings of the lines and writes them back unmodified for demo purposes. Here's the code (note the hard-coded device name).

int main(int argc, char **argv)
{
    // Print constant values.
    printf( "long TIOCMGET = 0x%x;\n", TIOCMGET );
    printf( "long TIOCMSET = 0x%x;\n", TIOCMSET );
    printf( "int O_RDWR = 0x%x;\n", O_RDWR );
    printf( "int O_NDELAY = 0x%x;\n", O_NDELAY );
    printf( "int O_NOCTTY = 0x%x;\n", O_NOCTTY );

    int value = O_RDWR|O_NDELAY|O_NOCTTY;
    printf( "value=%x\n", value );
    int portfd = open("/dev/tty.usbmodem735ae091", value);
    printf( "portfd=%d\n", portfd );

    int lineStatus;
    printf( "TIOCMGET %x\n", TIOCMGET );
    int rc = ioctl( portfd, TIOCMGET, &lineStatus );
    printf( "rc=%d, linestatus=%x\n", rc, lineStatus );

    rc = ioctl( portfd, TIOCMSET, &lineStatus );
    printf( "rc=%d, linestatus=%x\n", rc, lineStatus );

    if ( rc == -1 )
        printf( "Failure\n" );
    else
        printf( "Success\n" );

    if ( portfd != -1 )
        close( portfd );

    return 0;
}

Output of the above is:

long TIOCMGET = 0x4004746a;
long TIOCMSET = 0x8004746d;
int O_RDWR = 0x2;
int O_NDELAY = 0x4;
int O_NOCTTY = 0x20000;
value=20006
portfd=3
TIOCMGET 4004746a
rc=0, linestatus=6
rc=0, linestatus=6
Success

Here's the Java implementation:

public class Cli
{
    /**
     * Java mapping for lib c
     */
    public interface MacCl extends Library {
        String NAME = "c";
        MacCl INSTANCE = Native.load(NAME, MacCl.class);

        int open(String pathname, int flags);
        int close(int fd);
        int ioctl(int fd, long param, LongByReference request);
        String strerror( int errno );
    }

    private static final MacCl C = MacCl.INSTANCE;

    private static PrintStream out = System.err;

    public static void main( String[] argv )
    {
        long TIOCMGET = 0x4004746a;
        long TIOCMSET = 0x8004746d;
        int O_RDWR = 0x2;
        int O_NDELAY = 0x4;
        int O_NOCTTY = 0x20000;

        int value = O_RDWR|O_NDELAY|O_NOCTTY;
        out.printf( "value=%x\n", value );
        int portfd = C.open(
                "/dev/tty.usbmodem735ae091",
                value );
        out.printf( "portfd=%d\n", portfd );

        LongByReference lineStatus = new LongByReference();

        int rc = C.ioctl( portfd, TIOCMGET, lineStatus );
        out.printf(
                "rc=%d, linestatus=%d\n", rc, lineStatus.getValue() );

        rc = C.ioctl( portfd, TIOCMSET, lineStatus );
        out.printf(
                "rc=%d errno='%s'\n",
                rc,
                C.strerror( Native.getLastError() ) );

        if ( rc == -1 )
            out.print( "Failure." );
        else
            out.print( "Success." );

        if ( portfd != -1 )
            C.close( portfd );
    }
}

Java output is:

value=20006
portfd=23
rc=0, linestatus=6
rc=-1 errno='Inappropriate ioctl for device'
Failure.

Answer 1

We checked many times the third argument. Must a pointer be passed or not? The docs we found -- Man page man -s 4 tty on the Mac -- really documents to pass a pointer. So there seem to be differences between Unix implementations.

Finally we found the solution by printing the passed value using printf( "%xl", ... );. and the resulting value was 0xffffffff8004746d. So we got an unexpected sign extension.

And the problem is the line

        long TIOCMSET = 0x8004746d;

The literal constant is defined as an int literal that gets implicitly converted to long with sign extension. Since 0xffffffff8004746d is not equal to 0x8004746d' this explains the error message inappropriate ioctl for device. When we changed the line above to

        long TIOCMSET = 0x8004746dL; // Note the 'L' at the end.

everything worked perfectly. On other Unixes we did not have the problem because the TIO... constants happened to be positive.