Stack around the variable 'ch' was corrupted

Question

I am in the process of writing a decipher algorithm for Vegenere Variant Cipher and ran into some C specific issues(I am not too familiar with C).

I get
"Run-Time Check Failure #2 - Stack around the variable 'ch' was corrupted" error.

If I understand the error right, ch is not available when I try to read/write to it(ch in this case represents a HEX value read from the text file, I have posted the code of the function below).

But, for the life of me, I can't figure out where it happens. I close the file way before the I exit the function(exception is thrown at the time I leave the function).

Can you take a look an let me know where I have it wrong? Thanks in advance.

P.S. I am tagging the question with C++ as well as it should pretty much be the same except, maybe, how we read the file in. Anyways, my code below:

int getKeyLength(char *cipherTxtF){

int potKeyL = 1;
float maxFreq = 0.00;
int winKL = 1;


for (potKeyL = 1; potKeyL <= 13; potKeyL++)// loop that is going through each key size startig at 1 and ending at 13
{
    unsigned char ch;
    FILE *cipherTxtFi;
    cipherTxtFi = fopen(cipherTxtF, "r");
    int fileCharCount = 0;
    int freqCounter[256] = { 0 };
    int nThCharCount = 0;
    while (fscanf(cipherTxtFi, "%02X", &ch) != EOF) {
        if (ch != '\n') {
            if (fileCharCount % potKeyL == 0){
                int asciiInd = (int)ch;
                freqCounter[asciiInd] += 1;
                nThCharCount++;
            }
        }
        fileCharCount++;
    }
    fclose(cipherTxtFi);
    float frequenciesArray[256] = { 0 };
    float sumq_iSq = 0;     
    int k;
    for (k = 0; k < 256; k++){
        frequenciesArray[k] = freqCounter[k] / (float)nThCharCount;
    }

    for (k = 0; k < 256; k++){
        sumq_iSq += frequenciesArray[k] * frequenciesArray[k];
        printf("%f \n", sumq_iSq);
    }

    if (maxFreq < sumq_iSq) {
        maxFreq = sumq_iSq;
        winKL = potKeyL;
    }       
}

return winKL;

}

Answer 1

You are trying to read an hexadecimal integer with fscanf() (format "%02X", where X means "integer in hex format") and store it into a char.

Unfortuantely fscanf() just receives the address of the char and doesn't know that you've not provided the address of an int. As int is larger than a char, the memory gets corrupted.

A solution could be:

int myhex; 
while (fscanf(cipherTxtFi, "%02X", &myhex) != EOF) {
   ch = myhex; 
   ...