x64 NASM Assembly 64bit conditional jump doesn't work
Environment: Ubuntu22.04@WSL on Windows 11, NASM v2.15.05, x64, Intel CPU
When I was debugging my assembly program, I found something strange during my debug. The program was executing a loop. The following code compares between _i(0x404054) variable that is stored in BSS section and the variable number(0x40404c). If _i is less or equal than number, jle _loop should be executed.
(gdb-peda)
0x4011d2 <_primeTrue+10> mov rsi, qword ptr [_i] <0x404054>
0x4011da <_primeTrue+18> call printf@plt <printf@plt>
0x4011df <_primeTrue+23> jmp _primeEnd <_primeEnd>
↓
0x4011e2 <_primeEnd> add qword ptr [_i], 1 <0x404054>
0x4011eb <_primeEnd+9> cmp qword ptr [0x404054], 0x40404c
► 0x4011f7 <_primeEnd+21> ✔ jle _loop <_loop>
↓
...(abbreviated)...
At the exact moment when <_primeEnd+21> was running, _i and number's value were the same as following:
gdb-peda$ x/gx 0x404054
0x404054: 0x0000000000000004
gdb-peda$ x/gx 0x40404c
0x40404c: 0x0000000000000003
gdb-peda$
So it means _i values(0x4) is bigger than number value(0x3), so jle _loop shouldn't be executed. However, as you can see on the image below, the program runs jle _loop command even though the condition doesn't seem to be correct.
How is this possible, even though I just obviously confirmed those values Can you let me know what I'm doing wrong? Thanks in advance.
p.s)
Context: The full code I'm writing is below. It's just a program that prints all prime numbers until the user input's value. (If I input 100, the program should print every prime number less or equal to 100.)
; Nasm v 2.15.05 (on Linux x64)
extern printf
extern scanf
true EQU 0x1
false EQU 0x0
section .data
notificationString DB "num: ", 0x0
notificationStringFormat DB "%s", 0x0 ; %s\0
userInputFormat DB "%d", 0x0 ; %d\0
resultPrintFormat DB "%d", 0xA, 0x0 ; %d\n\0
section .bss
number RESB 0x8
_i RESB 0x8
section .text
global main
; validate if the given number(RSI) is a prime number
isPrimeNumber:
mov r10, 0x2 ; looping variable
_inspect:
mov rax, rdi ;rsi
cdq
mov rbx, r10
div rbx
cmp rdx, 0x0
je _itIsNotPrime
jne _notDetermined
; None prime number cases go here
_itIsNotPrime:
mov rax, false
retn
_notDetermined:
inc r10
mov r9, r10
cmp r9, rdi ;rsi
je _itIsPrime ; Fully inspected and confirmed it is a prime number.
jne _inspect ; Not fully inspected. Continue checking.
_itIsPrime:
mov rax, true
retn
main:
; initialize
push rbp
mov rbp, rsp
mov rdi, notificationString
mov rsi, notificationStringFormat
call printf
mov rdi, userInputFormat
mov rsi, number
call scanf
;mov rax, 0x2
;mov [_i], rax
mov QWORD [_i], 0x2
_loop:
mov rdi, [_i]
call isPrimeNumber ; true(0x1) will return if it is a prime number
; false(0x0) will return if it is not a prime number
cmp rax, true
je _primeTrue
jne _primeFalse
_primeTrue:
mov rdi, resultPrintFormat
mov rsi, [_i]
call printf
jmp _primeEnd
_primeFalse:
nop
_primeEnd:
;mov r10, [_i]
;add r10, 0x1
;mov [_i], r10
add QWORD [_i], 0x1
cmp QWORD [_i], number
jle _loop
_loop_end:
; end program
mov rsp, rbp
pop rbp
retn
fix)
With @Jester's help, I fixed number value reserves 0x8 bytes and _i reserves the same amount. So, there aren't any unexpected values in those.
I solved the problem. It looks like just using number in the comparison statement didn't reference the exact value in the memory. So, I changed my code to make it refer to the direct memory value by adding the square bracket to do that and comparing them. (number -> [number])
before:
cmp QWORD [_i], number ; _i referred correctly, but number isn't.
jle _loop
After:
mov r10, [_i]
mov r11, [number]
cmp r10, r11
jl _loop
jnl _loop_end ; Refer to the full code below. It just means the end of _loop.
The full code I finally finished is:
; Nasm v 2.15.05 (on Linux x64)
extern printf
extern scanf
true EQU 0x1
false EQU 0x0
section .data
notificationString DB "num: ", 0x0
notificationStringFormat DB "%s", 0x0 ; %s\0
userInputFormat DB "%d", 0x0 ; %d\0
resultPrintFormat DB "%d", 0xA, 0x0 ; %d\n\0
section .bss
number RESB 0x8
_i RESB 0x8
section .text
global main
; validate if the given number(RSI) is a prime number
isPrimeNumber:
mov r10, 0x2 ; looping variable
_inspect:
mov rax, rdi ;rsi
cdq
mov rbx, r10
div rbx
cmp rdx, 0x0
je _itIsNotPrime
jne _notDetermined
; None prime number cases go here
_itIsNotPrime:
mov rax, false
retn
_notDetermined:
inc r10
mov r9, r10
cmp r9, rdi ;rsi
je _itIsPrime ; Fully inspected and confirmed it is a prime number.
jne _inspect ; Not fully inspected. Continue checking.
_itIsPrime:
mov rax, true
retn
main:
; initialize
push rbp
mov rbp, rsp
mov rdi, notificationString
mov rsi, notificationStringFormat
call printf
mov rdi, userInputFormat
mov rsi, number
call scanf
mov QWORD [_i], 0x2
_loop:
mov rdi, [_i]
call isPrimeNumber ; true(0x1) will return if it is a prime number
; false(0x0) will return if it is not a prime number
cmp rax, true
je _primeTrue
jne _primeFalse
_primeTrue:
mov rdi, resultPrintFormat
mov rsi, [_i]
call printf
jmp _primeEnd
_primeFalse:
nop
_primeEnd:
add QWORD [_i], 0x1
; fixed
mov r10, [_i]
mov r11, [number]
cmp r10, r11
jle _loop
jmp _loop_end
_loop_end:
; end program
mov rsp, rbp
pop rbp
retn
With that code, now you can get the all prime numbers until your input. (You can also try my code.)
knightchaser@3rdfr13nds:~/assembly/loop$ cat Makefile
prime_numbers_linux_x64: prime_numbers_linux_x64.asm
nasm -f elf64 -o prime_numbers_linux_x64.o prime_numbers_linux_x64.asm
gcc -o prime_numbers_linux_x64 prime_numbers_linux_x64.o -no-pie
clean:
rm -f ./prime_numbers_linux_x64
rm -f ./prime_numbers_linux_x64.o
Knightchaser@3rdfr13nds:~/assembly/loop$ make
...
knightchaser@3rdfr13nds:~/assembly/loop$ ./prime_numbers_linux_x64
num: 20
3
5
7
11
13
17
19
- How to list all the removable devices with DBus and UDisks2?
- gobject/gnome/glib bindings for D using GIR?
- How do nested functions get compiled?
- The "this" pointer and message receiving in D
- 64-bit executables with DMD
- GtkD with D lang on Fedora
- Why Android used Java concept instead of D language or C or C++? But Chromium web browser is in C++, its very complicated match
- How to use MongoItemWriter to write a List<T>
- Why a function with protected modifier can be overridden and accessible every where?
- Convert Unicode const(uint)* to a dlang character type
- Compiling D with Code::Blocks
- DMD vs. GDC vs. LDC
- Rendering a font in raylib using freetype
- Digital Mars D compiler; acquiring ASM output
- D Programming: openssl rsa forward reference compiler error
- D compiler DMD doesn't link object files
- OPTLINK: Warning 23: No Stack
- Is there a limit in the amount of temporary generated symbols during a project build using dmd 2.063?
- Is this the right way to combine Garbage collected with none Garbage collected code in D
- D compiler (Digital Mars D Compiler) throwing error
- Which D Compiler to Use?
- Splitting a string treating multiple whitespace as one separator
- Proper way of passing array parameters to D functions
- Detailed Valgrind internals documentation
- Is it possible, in D, to tell the garbage collector to not scan a particular pointer (or anything below it)?
- Iterate over key/value pairs in associative array in D.
- Dlang associative array of an array of strings keyed by a string has unexpected behavior
- How to repeat a statement N times (simple loop)
- Is worth the effort to learn D?
- ld: undefined reference to object I can see in objdump