Reset handle after ReadFile

Question

I'm trying to open a file on windows and check that the magic bytes match a windows PE32. If I run the code below and return just before the ReadFile call in the function problemFunction the code works fine and it prints 5a4d at the end of the main function. However if I return after the ReadFile call in problemFunction then I exit in the dos->e_magic != PIMAGE_DOS_HEADER check.

#include <Windows.h>
#include <winternl.h>

void problemFunction(HANDLE *fh) {
    DWORD fileSize = GetFileSize(fh, NULL);
    if (!fileSize) { CloseHandle(fh); exit(1); }

    BYTE* pByte = new BYTE[fileSize];
    DWORD dw;
    ReadFile(*fh, pByte, fileSize, &dw, NULL);
    // could be wrong but i think i need to run SetFilePointer here but not sure on what to do.
    return;
}

int main() {
    const char* filepath = "C:\\windows\\file\\path\\to\\exe";
    HANDLE fh = CreateFileA(filepath, GENERIC_READ | GENERIC_WRITE, 0, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
    if(fh == INVALID_HANDLE_VALUE) { CloseHandle(fh); exit(1); }

    problemFunction(&fh);

    DWORD fileSize = GetFileSize(fh, NULL);
    if (!fileSize) { CloseHandle(fh); exit(1); }

    BYTE* pByte = new BYTE[fileSize];
    DWORD dw;
    ReadFile(fh, pByte, fileSize, &dw, NULL);

    PIMAGE_DOS_HEADER dos = (PIMAGE_DOS_HEADER)pByte;

    if (dos->e_magic != IMAGE_DOS_SIGNATURE) { CloseHandle(fh); exit(1); }
    // dos->e_magic should be 5a4d for MZ, windows PE
}

I assume i need to reset the file pointer after the problemFunction read call with something like

LONG reset = -sizeof(DWORD);
SetFilePointer(*fh, reset, NULL, FILE_END);

But i can't get it to work.

Thanks

Answer 1

There are a number of problems with your code.

problemFunction() is taking a HANDLE* pointer as input, but it is not dereferencing that pointer when passing it to GetFileSize() or CloseHandle(). But it is dereferencing the pointer when passing it to ReadFile().

You must be compiling your code with STRICT Type Checking turned off, otherwise your code would fail to compile. You should always compile with STRICT enabled.

HANDLE is a pointer type, so there is no need to pass it around by pointer, unless you are going to modify its value, which this code is not doing. So you should change problemFunction() to take a HANDLE as-is rather than taking a HANDLE* pointer.

Also, GetFileSize() does not return 0 on failure, like your code assumes. It actually returns INVALID_FILE_SIZE which is -1, ie 0XFFFFFFFF as a DWORD. This is clearly stated in the documentation:

If the function fails and lpFileSizeHigh is NULL, the return value is INVALID_FILE_SIZE. To get extended error information, call GetLastError.

But, most importantly, your 2nd call to ReadFile() inside of main() does not read what you are expecting because the 1st call to ReadFile() inside of problemFunction() has already read the data (and leaked it!), but you are not seeking the HANDLE back to the beginning of the file after that read so the 2nd call to ReadFile() can read it again. You are correct that you need to use SetFilePointer() for that.

With that said, try something more like this:

#include <Windows.h>
#include <winternl.h>

bool test(HANDLE fh) {
    DWORD fileSize = GetFileSize(fh, NULL);
    if (fileSize == INVALID_FILE_SIZE) {
        return false;
    }

    BYTE* pByte = new BYTE[fileSize];
    DWORD dw;
    if (!ReadFile(fh, pByte, fileSize, &dw, NULL)) {
        delete[] pByte;
        return false;
    }

    // use pByte as needed...

    delete[] pByte;

    if (SetFilePointer(fh, 0, NULL, FILE_BEGIN) == INVALID_SET_FILE_POINTER) {
        return false;
    }

    return true;
}

int main() {
    const char* filepath = "C:\\windows\\file\\path\\to\\exe";
    HANDLE fh = CreateFileA(filepath, GENERIC_READ, 0, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
    if (fh == INVALID_HANDLE_VALUE) {
        return 1;
    }

    if (!test(fh)) {
        CloseHandle(fh);
        return 1;
    }

    DWORD fileSize = GetFileSize(fh, NULL);
    if (fileSize == INVALID_FILE_SIZE) {
        CloseHandle(fh);
        return 1;
    }

    BYTE* pByte = new BYTE[fileSize];
    DWORD dw;
    if (!ReadFile(fh, pByte, fileSize, &dw, NULL) || dw < sizeof(IMAGE_DOS_HEADER)) {
        CloseHandle(fh);
        return 1;
    }

    PIMAGE_DOS_HEADER dos = (PIMAGE_DOS_HEADER)pByte;

    if (dos->e_magic != IMAGE_DOS_SIGNATURE) {
        delete[] pByte;
        CloseHandle(fh);
        return 1;
    }

    ...

    delete[] pByte;
    CloseHandle(fh);

    return 0;
}