Through some manipulation I have narrowed it down to a problem with the ret op. I know call pushes the return address to the stack; is it illegal to pop it and push it back?
format ELF64 executable 3
entry start
segment readable executable
start:
pop rcx ; argc
mov [argc],cl ; int -> ASCII
add [argc],'0'
push 1 argc 1
call sys_write
mov rdi,0
mov rax,60
syscall
sys_write: ; (fd,*buf,count)
pop r11
pop rdx rsi rdi
mov rax,1
syscall
push r11
ret
segment readable writable
argc rb 1
Output is:
$ ./prog
1Segmentation fault
$ _
The syscall instruction clobbers the contents of R11 with the RFLAGS register, which means that after making a syscall the value you stored in R11 is overwritten.
A solution might be to simply select an appropriate register that goes unmodified by a syscall. According to Intel's instruction reference manual (http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-manual-325462.pdf, page 4-668 Vol. 2B) syscall overwrites RCX, RIP, R11, and RFLAGS as part of its operation, but the OS is of course able to restore RIP and RFLAGS.
This leaves plenty of other options. A good choice would be a register that a function call is allowed to clobber in the standard user-space calling convention, but that Linux system calls will leave unmodified. R8 fits the bill. Since you aren't using the standard x86-64 System V function-calling convention, RBX or RBP would be even better choices (smaller machine-code size since they don't need a REX prefix).