How determine application subsystem from executable file

Question

I'm trying to detect console application from the list of the executables files installed on my computer.

How to implement it?

Every application has a "subsystem" (windows application, console application or library; specified to the linker as option, I think). How to detect it using only the executable file?

Are there alternative methods to detect the application characteristic? Additionally, are there any method for detecting the file is a really executable file?

Any issue for JAR executables?

Answer 1

Without any programming you receive this information from

dumpbin.exe /headers filename

Some information gives you GetBinaryType and SHGetFileInfo functions. All information which you needs you will find in the header of every executable file. See Microsoft Portable Executable and Common Object File Format Specification in http://www.microsoft.com/whdc/system/platform/firmware/PECOFF.mspx.

One can use also Debug Help Library API from DbgHelp.dll (see http://msdn.microsoft.com/en-us/library/ms679309(VS.85).aspx). IMAGE_DOS_HEADER, IMAGE_DOS_SIGNATURE and IMAGE_NT_HEADERS32 structures gives you full information.

UPDATED (add some code): Or you can use only structures defined in WinNT.h. The corresponding code can start like following

// Open source file
hSrcFile = CreateFile (pszSrcFilename, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, 0, NULL);
if (hSrcFile == INVALID_HANDLE_VALUE)
    __leave;

// Map the source file in memory
hMapSrcFile = CreateFileMapping (hSrcFile, NULL, PAGE_READONLY, 0, 0, NULL);    // SEC_IMAGE
if (!hMapSrcFile || hMapSrcFile == INVALID_HANDLE_VALUE)
    __leave;

// Map the entire of the source file is memory
pSrcFile = (PBYTE) MapViewOfFile (hMapSrcFile, FILE_MAP_READ, 0, 0, 0);
if (!pSrcFile)
    __leave;

pDosHeader = (IMAGE_DOS_HEADER *)pSrcFile;

if (pDosHeader->e_magic != IMAGE_DOS_SIGNATURE) {
    printf ("it is not a EXE file.\n");
    return 1;
}
printf ("IMAGE_DOS_HEADER size %d (0x%X) bytes\n",  sizeof(IMAGE_DOS_HEADER), sizeof(IMAGE_DOS_HEADER));

DumpDosHeader (pDosHeader);
pDosExeStart = (PBYTE)pDosHeader + pDosHeader->e_cparhdr*16;

if (g_bDump)
    HexDump (1, pDosExeStart, pDosHeader->e_lfanew - pDosHeader->e_cparhdr*16, (DWORD)pDosExeStart);

if (pDosHeader->e_lfanew) {
    IMAGE_NT_HEADERS32 *pNtHeader = (IMAGE_NT_HEADERS32 *)((PBYTE)pDosHeader + pDosHeader->e_lfanew);
    //IMAGE_NT_HEADERS64 *pNtHeader64 = (IMAGE_NT_HEADERS64 *)((PBYTE)pDosHeader + pDosHeader->e_lfanew);
    IMAGE_SECTION_HEADER *pFirstSectionHeader = (IMAGE_SECTION_HEADER *)((PBYTE)&pNtHeader->OptionalHeader +
                                                                            pNtHeader->FileHeader.SizeOfOptionalHeader);

    if (pNtHeader->Signature == IMAGE_NT_SIGNATURE) {
        int i;

        printf ("\nPE signature\n");
        printf ("\nIMAGE_FILE_HEADER: size %d (0x%X) bytes, offset from the begin of the file: %d (0x%X)\n",
                sizeof(IMAGE_FILE_HEADER), sizeof(IMAGE_FILE_HEADER),
                ((PBYTE)&pNtHeader->FileHeader - (PBYTE)pDosHeader), ((PBYTE)&pNtHeader->FileHeader - (PBYTE)pDosHeader));
        DumpFileHeader (1, &pNtHeader->FileHeader);
        switch (pNtHeader->OptionalHeader.Magic) {
            case IMAGE_NT_OPTIONAL_HDR32_MAGIC:
                printf ("\nIMAGE_OPTIONAL_HEADER32: size %d (0x%X) bytes, offset from the begin of the file: %d (0x%X)\n",
                        sizeof(IMAGE_OPTIONAL_HEADER32), sizeof(IMAGE_OPTIONAL_HEADER32),
                        ((PBYTE)&pNtHeader->OptionalHeader - (PBYTE)pDosHeader), ((PBYTE)&pNtHeader->OptionalHeader - (PBYTE)pDosHeader));
                DumpOptionalHeader32 (1, &pNtHeader->OptionalHeader);
                break;
            case IMAGE_NT_OPTIONAL_HDR64_MAGIC:
                break;
            case IMAGE_ROM_OPTIONAL_HDR_MAGIC:
                break;
        }