Search code examples
c++windowsexecutableportable-executablelow-level

Custom PE executable file - Windows 10 cannot execute run the given file


I'm trying to write a custom PE file exporter for win10 64 bit, and I'm having some trouble after I've added alignments. I've created a basic test case: an .exe file containing the 'ret' instruction:

std::vector<unsigned char> text_bytecode = {
    0xC3 // 'ret'
};

pe_builder builder;
builder.add_section(
    ".text",
    text_bytecode,
    windows::section_header_characteristics::mem_execute | 
    windows::section_header_characteristics::mem_read |
    windows::section_header_characteristics::cnt_code |
    windows::section_header_characteristics::align_16_bytes
);

builder.emit_to_file(path);

This is the backend of the PE builder:

u64 align(u64 value, u64 alignment) {
    return (value + alignment - 1) & ~(alignment - 1);
}

pe_builder::pe_builder() {
    // initialize the dos header
    m_dos_header.magic = 0x5A4D;
    m_dos_header.cparhdr = 4;
    m_dos_header.lfanew = sizeof(windows::dos_header); // place the NT headers directly behind the DOS header

    // NT headers
    // technically, the PE signature goes here

    // initialize the file header
    m_file_header.machine = windows::get_machine_type();
    m_file_header.section_count = 0;
    m_file_header.optional_header_size = sizeof(windows::optional_header_64_bit);
    m_file_header.characteristics = windows::file_header_characteristics::executable;

    // initialize the optional header
    m_optional_header.magic = 0x020B; // PE32+
    m_optional_header.section_alignment = 4096;
    m_optional_header.file_alignment = 512;
    m_optional_header.major_linker_version = 14;
    m_optional_header.minor_linker_version = 0;
    m_optional_header.major_os_version = 6; // win10 compatibility
    m_optional_header.minor_os_version = 0;
    m_optional_header.major_image_version = 1;
    m_optional_header.major_subsystem_version = 6; // win10 compatibility
    m_optional_header.image_base = 0x140000000;
    m_optional_header.win32_version_value = 0;
    m_optional_header.subsystem = 3; // windows console subsystem
    m_optional_header.DLL_characteristics = 0x8160; // NX-Compatible and Terminal Server Aware
    m_optional_header.stack_reserve_size = 0x100000;
    m_optional_header.stack_commit_size = 0x1000;
    m_optional_header.heap_reserve_size = 0x100000;
    m_optional_header.heap_commit_size = 0x1000;
    m_optional_header.loader_flags = 0;
    m_optional_header.RVA_and_size_count = 0;

    u64 header_size =
        sizeof(windows::dos_header) +
        sizeof(g_pe_signature) +
        sizeof(windows::file_header) +
        sizeof(windows::optional_header_64_bit);

    m_optional_header.entry_point_address = align(header_size, m_optional_header.section_alignment);
    m_optional_header.code_base = m_optional_header.entry_point_address;
}

void pe_builder::add_section(
    const std::string& name,
    std::vector<unsigned char> data,
    windows::section_header_characteristics characteristics
) {
    m_file_header.section_count++;

    // If it's a code section, update the SizeOfCode in the optional header
    if ((characteristics & windows::section_header_characteristics::cnt_code) != 0) {
        m_optional_header.code_size += data.size();
    }

    windows::section_header new_section = {};
    memcpy(new_section.name, name.c_str(), std::min<size_t>(name.size(), 8));

    new_section.virtual_size = data.size();
    new_section.raw_data_size = align(data.size(), m_optional_header.file_alignment);

    if (m_section_data.empty()) {
        new_section.raw_data_pointer = align(
            sizeof(windows::dos_header) +
            sizeof(g_pe_signature) +
            sizeof(windows::file_header) +
            sizeof(windows::optional_header_64_bit) +
            sizeof(windows::section_header) * m_file_header.section_count,
            m_optional_header.file_alignment
        );
        new_section.virtual_address = m_optional_header.entry_point_address;
    }
    else {
        const auto& last_section_header = m_sections.back();
        new_section.raw_data_pointer = last_section_header.raw_data_pointer +
            align(m_section_data.back().size(), m_optional_header.file_alignment);
        new_section.virtual_address = last_section_header.virtual_address +
            align(m_section_data.back().size(), m_optional_header.section_alignment);
    }

    new_section.characteristics = characteristics;
    m_sections.push_back(new_section);
    m_section_data.push_back(data);
}

void pe_builder::emit_to_file(
    const filepath& path
) {
    // Open the output file in binary mode
    std::ofstream file(path, std::ios::binary);

    if (!file.is_open()) {
        // return false; // Failed to open the file
    }

    file.write(reinterpret_cast<char*>(&m_dos_header), sizeof(m_dos_header));
    file.write(reinterpret_cast<const char*>(&g_pe_signature), sizeof(g_pe_signature));
    file.write(reinterpret_cast<char*>(&m_file_header), sizeof(m_file_header));
    file.write(reinterpret_cast<char*>(&m_optional_header), sizeof(m_optional_header));

    for (size_t i = 0; i < m_sections.size(); ++i) {
        // Write the section header
        file.write(reinterpret_cast<const char*>(&m_sections[i]), sizeof(m_sections[i]));

        // Assuming m_section_data is a vector of vectors (or similar container) storing data for each section
        const auto& section_data = m_section_data[i];

        // Write the section data
        file.write(reinterpret_cast<const char*>(section_data.data()), section_data.size());
    }

    // Close the file
    file.close();

    m_dos_header.print();
    m_file_header.print();
    m_optional_header.print();
    for(const auto& section : m_sections) {
        section.print();
    }
}

For anyone wondering I've printed the final values (note that they are printed as their respective int counterparts, so hex formatting is missing, although I should also note that the hex values and flags should all be correct as I've created an MVP with them that worked).

dos header:
  magic:    23117
  cblp:     0
  cp:       0
  crlc:     0
  cparhdr:  4
  minalloc: 0
  maxalloc: 0
  ss:       0
  sp:       0
  csum:     0
  ip:       0
  cs:       0
  lfarlc:   0
  ovno:     0
  res:      0 0 0 0
  oemid:    0
  oeminfo:  0
  res2:     0 0 0 0 0 0 0 0 0 0
  lfanew:   64
file header:
  machine:                 34404
  section_count:           1
  time_date_stamp:         0
  pointer_to_symbol_table: 0
  symbol_count:            0
  optional_header_size:    240
  characteristics:         2
optional header (64 bit):
  magic:                   523
  major_linker_version:    14
  minor_linker_version:    0
  code_size:               1
  initialized_data_size:   0
  uninitialized_data_size: 0
  entry_point_address:     4096
  code_base:               4096
  image_base:              5368709120
  section_alignment:       4096
  file_alignment:          512
  major_os_version:        6
  minor_os_version:        0
  major_image_version:     1
  minor_image_version:     0
  major_subsystem_version: 6
  minor_subsystem_version: 0
  win32_version_value:     0
  image_size:              0
  header_size:             0
  check_sum:               0
  subsystem:               3
  DLL_characteristics:     33120
  stack_reserve_size:      1048576
  stack_commit_size:       4096
  heap_reserve_size:       1048576
  heap_commit_size:        4096
  loader_flags:            0
  RVA_and_size_count:      0
  data_directories:
    virtual address: 0, size: 0
    virtual address: 0, size: 0
    virtual address: 0, size: 0
    virtual address: 0, size: 0
    virtual address: 0, size: 0
    virtual address: 0, size: 0
    virtual address: 0, size: 0
    virtual address: 0, size: 0
    virtual address: 0, size: 0
    virtual address: 0, size: 0
    virtual address: 0, size: 0
    virtual address: 0, size: 0
    virtual address: 0, size: 0
    virtual address: 0, size: 0
    virtual address: 0, size: 0
    virtual address: 0, size: 0
section header:
  name:                .text
  virtual_size:        1
  virtual_address:     4096
  raw_data_size:       512
  raw_data_pointer:    512
  relocation_pointer:  0
  line_number_pointer: 0
  relocation_count:    0
  line_number_count:   0
  characteristics:     32

Solution

  • I also wrote an exe exporter once and here are some abnormalities I observed in your output:

    1. Set your pe header Characteristics to 35 instead of 2. You only specified IMAGE_FILE_EXECUTABLE_IMAGE but you don't have a relocation table, so you need IMAGE_FILE_RELOCS_STRIPPED, you still need IMAGE_FILE_EXECUTABLE_IMAGE and I would recommend IMAGE_FILE_LARGE_ADDRESS_AWARE for 64 bit applications.

    2. In the optional pe header set your SizeOfCode to the size of the code section which is a multiple of FileAlign. It does not matter if you only have a single instruction in the code section. The actual amount of instructions is reflected by the VirtualSize of the .text section which sould stay 1 for the single ret instruction.

    3. In the optional pe header the SizeOfImage and SizeOfHeaders should not be zero. For simple testing I would recommend static values! I used 1024 bytes for the SizeOfHeaders, because all headers fit into that space and it is a multiple of FileAlign, which I had set to 512. The SizeOfImage could be: SectionAlign * Numbers of Sections + SectionAlign, so that each section can be up to SectionAlign bytes + the space for the headers rounded up to a multiple of SectionAlign bytes. Be aware that the static values for SizeOfImage are only for testing and should not be used in a production szenario!

    4. Set your NumberOfRvaAndSizes to 16 and write zeroes. The documentation states that the number of data directories is variable but it is quite vague on the ordering. The data directory section seems to provide exact offsets to the different data directories. In my experience the exe export works if you write all 16 data directories and fill the unused with zeroes. This is how the msvc compiler does it!

    5. Set the Characteristics of your .text section to 1610612768 instead of 32. You want IMAGE_SCN_CNT_CODE, IMAGE_SCN_MEM_EXECUTE and IMAGE_SCN_MEM_READ. In the provided code, you set the required windows::section_header_characteristics for the .text section, but this does not seem to be reflected in the final output. Furthermore the IMAGE_SCN_ALIGN_16BYTES does not work for exe files. The documentation states:

    IMAGE_SCN_ALIGN_16BYTES - Align data on a 16-byte boundary. Valid only for object files.

    1. I don't see any kind of zero padding of the sections in your code. You have to ensure that the data pointers line up with your binary file. If the data pointer of the text section header has a value of 512, the section content must be located at byte 512 in your file. This is usually accomplished by zero padding during file export.

    I can update my answer if there is additional input. If possible you can append the exported exe as a binary file and I'll check the structure.