question number1:
Having this nasm:
section .data
dat db "write out this:%x", 0xa, 0x0
section .text
global main
extern printf
main:
push rbp
mov rbp, rsp
mov rdi, dat
mov esi, 0xdeedbeef
call printf
leave
ret
gives errno 24 - too many file descriptor opened.
BUT IF CHANGED TO int 80h, instead of
leave
ret
Will terminate without error, how's that?
Also, question number 2: If I do not make calling convention by :
push rbp
mov rbp, rsp
And only mov rbp, rsp , without pushing rbp before, then command terminated, although no function was call before, therefore there is no need to push base pointer. So why is it needed (in eyes of compiler), and will terminate?
You're mistaken about this having anything to do with file descriptors. That isn't what's being reported.
As you explained in comments, 24 is the number shown when you echo $? after running the program. This is the exit code of the program; normally, the value returned from the main function or passed to exit(). It can be whatever you want and normally does not correspond to an errno value.
So why does your program give an exit code of 24? If main returns, then the exit code is its return value. A function's return value is expected to be left in the rax register when it returns (assuming it's of integer or pointer type). But you never touch the rax register, so it still contains the value that was left there when printf returns. Now printf returns the number of characters it successfully printed, which for the string you chose is... 24 (count 'em).
It is just a coincidence that 24 also happens to the errno code for "too many open file descriptors" - that is completely unrelated.
If you want to exit with an exit code of 0 to signal success, then you should xor rax, rax just before your ret.
You didn't show the exact code you used when changing it to int 0x80 to invoke the _exit syscall yourself. But in that case, the exit code would be whatever is in the ebx register when you make the system call. Maybe your code puts zero in ebx, or maybe you are lucky and it happens to already contain zero.
(Side note: int 0x80 is the interface for 32-bit system calls, and is not appropriate in a 64-bit program which is what you seem to be writing, though it may work in a few cases. The 64-bit system call interface uses the syscall instruction and is explained in A.2.1 of the ABI.)
You have to align the stack.
When calling a C function from assembly (as printf in this case), you're required by the x86-64 ABI Section 3.2.2 to align the stack to a 16-byte boundary. The stack is aligned appropriately before the call to main, and the return address pushed by call subtracts 8 bytes.
So if you don't touch the stack at all in your code, it will not be correctly aligned when you call printf, and this can cause it to crash. (The assembler will not help you do this; that's not its job.) But when you push rbp, that subtracts a further 8 bytes and gets the stack aligned properly. So either leave that code in, or align the stack yourself.
And in either case, keep in mind that if you change your code to push more stuff on the stack, you'll have to adjust it accordingly before you make any function calls.