I have written a function in 6502 assembly that calculates fibonacci number. I am invoking that function in C and then compiling both files into single binary file for 6502. But the output I get is different. The fibonacci of 10 is 55, but it outputs 2649. It calculates fibonacci of the number but not of integer between range of 0 to 10. For example fibonacci of 0 is 257, for 1 it is 258, and for 10 it is 2649 and so on. Why am I getting such output.
My C code: "main.c"
#include<stdio.h>
int fib();
int main() {
unsigned int p = fib();
printf("%u\n",p);
return 0;
}
Where fib() is the external function in 6502 assembly, it is linked together with C code.
My 6502 code: "fib.s"
fib:
//calculates fibonacci
And I compile both the files using cl65 -t sim6502 main.c fib.s -o fib
You left out the code for fib.s.
Most likely the compiler expecting a 16-bit return value, and you're leaving garbage (or the previous Fibonacci number) in the upper 8 bits (another register). Print at %#x to make that more obvious (0xhilo), instead of the 256*hi + lo that you're getting with unsigned decimal.
C requires int to be at least 16 bits, so declaring the return-type as int definitely tells the compiler that two 6502 register have meaningful values.
Declare the return value as uint8_t or unsigned char if your function only returns an 8-bit integer in one 6502 register.
Otherwise fix your asm to return 16-bit int. (Check the compiler-generated asm to learn which registers are used by the calling convention. I don't know 6502 personally, just that it has 8-bit registers.)