Search code examples
c++windowslinkerwebrtc64-bit

Windows says 64-bit executable is "Unsupported 16-Bit Application"

I have a C++ program that links Google's WebRTC library that compiles and runs successfully when I target 32-bit but doesn't work at all when I target 64-bit. After some trial and error I created the following program:

#include <media/base/adapted_video_track_source.h>

class AdaptedVideoTrackSource : rtc::AdaptedVideoTrackSource
{
public:
  void AddRef() const {}
  rtc::RefCountReleaseStatus Release() const { return rtc::RefCountReleaseStatus::kDroppedLastRef; }
  bool is_screencast() const {return false;}
  absl::optional<bool> needs_denoising() const {return false;}
  bool GetStats(webrtc::VideoTrackSourceInterface::Stats* stats) {return false;}
  webrtc::MediaSourceInterface::SourceState state() const {return webrtc::MediaSourceInterface::kLive;}
  bool remote() const {return false;}
};

int main() {
  AdaptedVideoTrackSource source;
}

This program fails with this error: enter image description here If I remove the super class the program runs fine.

How exactly do I debug a problem like this? I'm at a loss since I can't exactly debug the program. dumpbin appears to think my webrtc library is fine but says warning LNK4048: Invalid format file; ignored for my executable.

There's a lot of steps in the build process and I don't think I can put all of them here. I use CMake with ExternalProject_Add to download and build webrtc. I generate Ninja makefiles to build my code. Here are the ninja rules used to link the exe.

rule CXX_EXECUTABLE_LINKER__Test
  command = cmd.exe /C "$PRE_LINK && "C:\Program Files\CMake\bin\cmake.exe" -E vs_link_exe --intdir=$OBJECT_DIR --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100183~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100183~1.0\x64\mt.exe --manifests $MANIFESTS -- C:\PROGRA~2\MIB055~1\2017\COMMUN~1\VC\Tools\MSVC\1416~1.270\bin\Hostx64\x64\link.exe /nologo $in  /out:$TARGET_FILE /implib:$TARGET_IMPLIB /pdb:$TARGET_PDB /version:0.0  $LINK_FLAGS $LINK_PATH $LINK_LIBRARIES && $POST_BUILD"
  description = Linking CXX executable $TARGET_FILE
  restat = $RESTAT

build utest\Test.exe: CXX_EXECUTABLE_LINKER__Test utest\CMakeFiles\Test.dir\main.cpp.obj | <OTHER LIBARIES> || <OTHER LIBARIES>
  FLAGS = /DWIN32 /D_WINDOWS /GR /EHsc /bigobj /Zi /Ob0 /Od /RTC1 -MTd
  LINK_FLAGS = /machine:x64 /debug /INCREMENTAL /subsystem:console
  LINK_LIBRARIES = <OTHER LIBARIES> webrtc_bundle.lib <OTHER LIBRARIES>
  LINK_PATH = -LIBPATH:D:\Folder\install64\lib
  OBJECT_DIR = utest\CMakeFiles\Test.dir
  POST_BUILD = cd .
  PRE_LINK = cd .
  TARGET_COMPILE_PDB = utest\CMakeFiles\Test.dir\
  TARGET_FILE = utest\Test.exe
  TARGET_IMPLIB = utest\Test.lib
  TARGET_PDB = utest\Test.pdb

It's private code so I renamed the executable I build and replaced the non webrtc libraries with <OTHER LIBARIES>.

This is the args.gn I use to build webrtc:

target_cpu="x64"
rtc_enable_protobuf=true
is_official_build=false
rtc_build_examples=false
rtc_include_tests=false
enable_iterator_debugging=true
is_clang=false

Also, I wrote my own BUILD.gn file to bundle webrtc with other libraries that Google's build system can build.

UPDATE

I found that I can manually link my obj files and make a perfectly good exe. Then I started going into the Ninja Makefiles generated by CMake and playing around with the linker rules. I found that if I remove /debug from the linker flags then everything works great. Of course, I want to be able to debug my debug builds.

Solution

  • I moved on and tried to start building my project with clang-cl and lld-link which provided a little more diagnostic output. I started getting warnings about linking different versions of the run time library. MSVC_RUNTIME_LIBRARY was correctly set and I had the /MTd flag in CMAKE_CXX_FLAGS and CMAKE_LINKER_FLAGS but by running ninja with the verbose setting I could see that my flags were being followed by /MDd which overwrote the previous runtime setting. Eventually I appended /MTd to CMAKE_CXX_FLAGS_DEBUG and CMAKE_CXX_FLAGS_RELEASE and now it's the last runtime flag in the compiler and link commands and I haven't had this problem since.