I know this is weird, but here goes.
I'm managing a very old pre-ANSI C code base. The following code, believe it or not, actually compiles.
myprog.c:
// Prototype.
int rec_index(); // Zero arguments.
kz = rec_index(1, 19, 2, 0, " ", 0, -1); // Seven arguments.
recindex.c:
int rec_index(flag, type, from, to, c, s, status, t) // Eight arguments.
int flag, type, from, to, s, status, t;
char *c;
{
printf("%d %d %d", flag, type, from);
if (flag < 0 || flag > 2) {
return -1;
}
if (type < 1 || type > 20) { // Line A.
return -1;
}
if ((flag == 1) && (type < 7 || type == 12 || type == 13)) { // Line B.
// Line C.
}
}
In a 32-bit environment, this behaves reasonably well. The missing argument t in the function definition is interpreted to be zero (or maybe more properly, it's undefined), but that's only included in a small subset of calls to this function and the long-gone developers knew to include that when it was needed. So, when running this code, the output would be the first three parameters, as expected:
1 19 2
Great. The method returns with the expected behavior.
However...
We recently began porting our software to a 64-bit environment. In a 64-bit configuration, regardless of what is passed into the function, the second and third parameters of eight are read as zero in the function.
So, pass in 1, 19, and 2 in the function call, and inspect the parameters inside the function, and Visual Studio reports them as 1, 0, and 0.
1 0 0
Oddly, the fourth through eighth parameters pass through correctly. Only the second and third are wrong.
This is like standing next to someone, taking an empty wallet, inserting $3, handing them the wallet, and them opening it to find it empty. Where did the values go?
But it gets stranger. I passed in type as 19, but the function is reporting that its value is zero. (This is backed up by inspecting the memory location &type.) Let's just go with that for a second.
Now we get to the if statement marked as Line A in recindex.c:
if (type < 1 || type > 20) { // Line A.
Since type is being reported to be zero by the debugger, the left side of the || should be true and this will short-circuit the whole boolean to true, jumping into the braces and resulting in return -1;. Oddly, this does NOT happen, and the debugger jumps to the next if at Line B:
if ((flag == 1) && (type < 7 || type == 12 || type == 13)) { // Line B.
Again, type should be zero, and flag is (actually left untouched) as 1. This should make the whole if condition true, and fall into the loop to Line C.
The instruction pointer jumps into the loop as expected, but the value of type changes from 0 to some large value, always different, around 5000000 or so. NOTHING in the if condition has any side effects, so how is this value changing?!
I'd guess that there's some kind of stack corruption going on, probably as a result of undefined behavior due to the very inconsistent function prototyping. Again, this actually works in a 32-bit environment, and has for 20+ years.
The right answer here is to rewrite the function calls so as to all take eight arguments, matching the function declaration, and to use types like int32_t instead of int so that we can be sure of sizing. I'll likely do that.
However, I'd love an explanation of why I'm seeing the behavior I am.
Why are the second and third arguments to the function being re-written as zero once the function is entered?
Why is the value of one of those arguments, although appearing to be zero, isn't behaving like zero when evaluated?
What is causing that value to change when there is nothing to change it?
I figured this out.
I mentioned (with a bit of obfuscation) that I was calling this function:
int rec_index(flag, type, from, to, ck, sec, status, tert)
int flag, type, from, to, sec, status, tert;
char *ck;
{ … }
In the post I'd written that the call to this function was as such:
kz = rec_index(1, 19, 2, 0, " ", 0, -1); // Seven arguments.
But to not avoid sharing too many internal snippets on a public internet forum, I had actually paraphrased this, the real code:
kz = rec_index(1, 19, trp->iztabl, 0, " ", 0, -1);
I was assuming that the inconsistent prototypes were causing the corruption. Of course not – the real problem is with that third parameter, trp->iztabl. The variable trp is of type struct tran *, a pointer to a struct. Upon introspecting the actual value and attempting to dereference this, I found that trp was a bad pointer. Not a NULL pointer, but a BAD pointer – just gibberish, pointing to an unknown region of memory.
When the object code hits a bad pointer, it either doesn't know how to use the arrow operator, or more likely, it goes to the appropriate place in memory specified by trp and jumps forward a number of bytes corresponding to iztabl's location in the struct. Still, one would expect this should just return gibberish.
The problem/compiler bug/general annoyance is that the value returned by the failed application of the arrow operator is a 64-bit pointer, since we're in a 64-bit environment, and not a 32-bit int as trp->iztabl is defined. Since the call to rec_index() consists of 32-bit integers next to each other, the act of inserting a 64-bit pointer is destructive. This is why the second and third arguments disappear. The returned value of trp->iztabl takes out its own memory location and the one BEFORE it (due to endianness).
The other behavior I saw was related to values being overwritten in the debugger. I suspect the debugger's expected behavior is different than what this code provided, and so it got confused as it ran.