In my assembly program I am interested to know what's the minimal value %rsp gets (as it grows down) compared to its initial value so I wrote in main:
mov %rsp, %rcx #start
mov %rsp, %rdx #max
and in every part in my code before push (or any other command that may affect %rsp) I wrote:
cmp %rsp, %rdx
jl next3 # current rsp is less that prev_max then skip the update of value for max
mov %rsp, %rdx
next3: # every time this is copied I change the number like next4, next5 etc...
but when I debug my code rdx and rcx share the same value, why is that?
Your stack checking code is in fact correct. Since you provided full code[*] we can see there is a problem starting from line 73:
cmp (%rdi), %esi
cmp %rsp, %rdx
jl next4
mov %rsp, %rdx
next4:
jne continue
Your stack check is in between the cmp (%rdi), %esi and the jne continue so it will use the flags from cmp %rsp, %rdx instead. As it happens those are equal at that point so your function will not go to continue but return. The stack space used will just be 16 bytes. Note this stack check block is useless as the stack pointer can not change from the previous one so you can remove lines 74-80. This will then produce a stack usage of 0x40 bytes as you expect.
As to the question of operand order, at&t uses the reverse to intel. Consider this code:
mov $1, %eax
mov $2, %edx
cmp %eax, %edx
jl next
This will not jump.
[*] https://onlinegdb.com/5CKsK1GNT with ja changed back to jl