x86 assembly jl doesn't work

I'm coding in x86 assembly (AT&T syntax) on 64-bit Ubuntu (so I'm using as --32 and ld -melf_i386, which has been working fine for other exercises so far).

The jl instruction is working opposite to what I expected. I can actually get the code to work properly with jg, which would basically solve my problem, but I'd like to find out the underlying issue here.

The code snippet is the following:

   # Open file for reading
   movl $SYS_OPEN, %eax         # prepare syscall 5 
   movl $input_file_name, %ebx      # move input file name into %ebx
   movl $0, %ecx                    # open in read-only mode
   movl $0666, %edx     
   int $LINUX_SYSCALL               # execute system call,
                                   # sys_open returns file descriptor in %eax

   movl %eax, ST_INPUT_DESCRIPTOR(%ebp) # store the input file descriptor away

   # This will test and see if %eax is negative. 
   # If it is not negative, it will jump to continue_processing.
   # Otherwise it will handle the error condition that the negative number represents.
   cmpl $0, %eax
   jl continue_processing

   pushl $no_open_file_msg
   pushl $no_open_file_code
   call error_exit

continue_processing:

   # Open file for writing
   movl $SYS_OPEN, %eax         # prepare syscall 5

...and the program continues, though the rest should be irrelevant for this issue.

Debugging with gdbgui, I see the open sys call returns the input file descriptor (eax = 3) without a problem. Then you compare 0 to 3. If I'm not an idiot, 0 < 3 so the jl instruction should take you to continue_processing, but it does not.

However, if I instead use jg, it works. The open sys call returns 3 in eax and jg properly jumps to continue_processing.

I've read that the order of the operands for jumps may depend on the assembler. Could that be the case here? Compiling with gcc -m32 -nostdlib has the same behavior. I also tried swapping the order to cmpl %eax, $0 but I get Error: operand type mismatch for 'cmp'.

Or could it just be a quirk of the fact that this 32-bit code is being run on 64-bit Ubuntu?

I am reading the book Programming From the Ground Up. On page 125 the book example interjects a .section .data right after the jl continue_processing, inserts some labels and .ascii commands, and then resumes the code with a .section .text right before the pushl $no_open_file_msg. To clean up the code, I've consolidated the .section .data up on top and so do not require a second .section .text. It doesn't seem to affect the jl issue but I thought I'd mention it in case the problem does actually lie there.

Solution

In AT&T syntax, the order of operands is exchanged compared to Intel syntax. For this reason,

cmpl $0, %eax

actually computes eax − 0 and sets flags instead of computing 0 − eax as you might expect. Thus, all the flags are set the other way round than you expect initially, causing the issue you observe. There is not really a way to remedy this problem as it isn't really wrong. I recommend you to get used to this quirk.

As a general note, it's usually a good idea to use the testl instruction instead of comparing values with zero. testl %eax,%eax is more compact than cmp $0,%eax in the machine code and faster on some microarchitectures, too. testl sets flags according to the bitwise and of its operands, so if eax is negative, SF is set after testl %eax,%eax which you can check like this:

testl %eax,%eax
jns continue_processing