copying a value into the address of a pointer in an assembly function called from c++ (nasm)

Question

I'm trying to learn x86-64 assembly, and I found the book "Modern X86 Assembly Language Programming: Covers x86 64-bit, AVX, AVX2, and AVX-512", but it uses MASM and Visual C++ and I use Linux. So I'm trying to convert some of the programs in it to NASM-syntax, but I encountered a problem with storing the result of a calculation in a pointer passed to the function. The C++ code is

#include <iostream>
#include <iomanip>
#include <bitset>
using namespace std;
extern "C" int IntegerShift_(unsigned int a, unsigned int count, unsigned int* a_shl, unsigned int* a_shr);
static void PrintResult(const char* s, int rc, unsigned int a, unsigned int count, unsigned int a_shl, unsigned int a_shr)
{
        bitset<32> a_bs(a);
        bitset<32> a_shr_bs(a_shl);
        bitset<32> a_shl_bs(a_shr);
        const int w = 10;
        const char nl = '\n';

        cout << s << nl;
        cout << "count = " << setw(w) << count << nl;
        cout << "a = " << setw(w) << a << " (0b" << a_bs << ")" << nl;

        if (rc == 0)
                cout << "Invalid shift count" << nl;
        else
        {
                cout << "shl = " << setw(w) << a_shl << " (0b" << a_shl_bs << ")" << nl;
                cout << "shr = " << setw(w) << a_shr << " (0b" << a_shr_bs << ")" << nl;
        }
        cout << nl;
}

int main()
{
        int rc;
        unsigned int a, count, a_shl, a_shr;
        a = 3119;
        count = 6;
        rc = IntegerShift_(a, count, &a_shl, &a_shr);
        PrintResult("Test 1", rc, a, count, a_shl, a_shr);

    return 0;
}

This code tests the function IntegerShift_, which is written in assembly. (There are a few more tests in the main function that I didn't include here since they are basically the same as the one I did include). The original assembly code in the book is MASM code:

    
.code
IntegerShift_ proc
xor eax,eax 
cmp edx,31            
ja InvalidCount            
xchg ecx,edx    
mov eax,edx  
shl eax,cl    
mov [r8],eax  
shr edx,cl   
mov [r9],edx    
mov eax,1
InvalidCount:    
ret    
IntegerShift_ endp
end

The obvious way to translate this into NASM code (at least from what I know) is the following:

section .text
global IntegerShift_
IntegerShift_:
xor eax,eax
cmp esi,31           
ja InvalidCount            
xchg ecx,esi    
mov eax,esi  
shl eax,cl    
mov [rdx],eax  
shr esi,cl   
mov [rsi],esi    
mov eax,1
InvalidCount:    
ret 

however, assembling, compiling, and running this with:

nasm -f elf64 [asm filename]
g++ -Wall -no-pie [object file filename] [cpp filename] -o prog
./prog

results in a segmentation fault. I tried solving this every way I could think of and spent more than a couple hours on this, but I couldn't get it to work. I'm almost certain the problem is the way I try to store the results in the addresses of the a_shl and a_shr pointers, but I can't understand what I'm doing wrong and I will really appreciate some help. Thanks in advance!

Answer 1

First, the calling conventions are different between Windows and Linux.

https://en.wikipedia.org/wiki/X86_calling_conventions

• It appears you incompletely changed this..

Second, while you can mostly use 32-bit registers, you must treat addresses as their full 64-bit values.

Finally, you are also modifying esi then using rsi - they are overlapping registers - this is what resulted in your segmentation fault.

With those changes:

;extern "C" int IntegerShift_(unsigned int a, unsigned int count, unsigned int* a_shl, unsigned int* a_shr);
; RDI, RSI, RDX, RCX,

section .text
global IntegerShift_
IntegerShift_:
xor eax,eax
cmp esi,31
ja InvalidCount
xchg rcx,rsi    ; Need full 64-bit exchange
mov eax,edi     ; (r)di is the 'a' value
shl eax,cl
mov [rdx],eax
shr edi,cl      
mov [rsi],edi
mov eax,1
InvalidCount:
ret