Android C++ new operator leads to "malloc(4294967295) failed, errno 12"

Question

TD;DR

I'm porting a C++ project of mine to android, but when using the new operator it actually calls malloc(4294967295) for some reason, which results in an out of memory error, even though debugging proved my classes to be of a correct size (and the project works when built for Windows).

This results in the app getting stuck on a black screen on physical android devices, or to simply crash on emulators

Context

I am currently trying to port an SDL project of mine to android.

For this, I followed the instructions from the tutorial located in the official SDL repo. Additionally, I followed this tutorial to add the SDL_image module to my project, along with SDL_ttf.

The issue

Building the app works fine, but once booted, it gets stuck to a black screen:

Figuring out why it happens (debugging)

"Print" debugging

So I have plugged my phone into my computer and tried to debug the issue with adb, using prints, and here is what I have found:

To initialize SDL and other components, my game's main function looks as follows:

int main(int argc, char* argv[]) {
    __android_log_print(ANDROID_LOG_DEBUG, "DEBUG", "Starting");
    srand(time(NULL));

    const int targetFPS = 60;
    const int frameDelay = 1000 / targetFPS;

    // Init
    __android_log_print(ANDROID_LOG_DEBUG, "DEBUG", "Constructing Game class of size: %lX", sizeof(Game));
    Game* game = new Game(argc, argv);
    __android_log_print(ANDROID_LOG_DEBUG, "DEBUG", "Game constructed");

    __android_log_print(ANDROID_LOG_DEBUG, "DEBUG", "Initializing");
    int errInit = game->init("SAE201205", SDL_WINDOWPOS_CENTERED, SDL_WINDOWPOS_CENTERED, 800, 600, SDL_WINDOW_SHOWN | SDL_WINDOW_RESIZABLE);
    __android_log_print(ANDROID_LOG_DEBUG, "DEBUG", "Initialized: %d", errInit);

    if (errInit == 0) {
        while (game->running()) {
            /* Game Logic here */
        }

        game->clean();
    }

    delete game;


    return 0;
}

Notice the __android_log_print calls, they are here to help debugging the function.

When booting the app on my phone and catching my phone's logs using adb logcat, the following lines can be seen:

06-14 08:34:35.471  8262  8326 V SDL     : Running main function SDL_main from library /data/app/~~F1pn5U_H13LZFzCic5LLyA==/org.libsdl.app-AHd89RMgz8bystTw8XwclA==/lib/arm/libmain.so
06-14 08:34:35.471  8262  8326 V SDL     : nativeRunMain()
06-14 08:34:35.471  8262  8326 D DEBUG   : Starting
06-14 08:34:35.471  8262  8326 D DEBUG   : Constructing Game class of size: 98
06-14 08:34:35.472  8262  8326 W libc    : malloc(4294967295) failed: returning null pointer, errno: 12

And then the application breaks. Further prints I have written never get printed.

As can be seen, when using the new operator, it calls malloc. Problem is, instead of calling it with the size of my Game class (0x98, as printed above), it calls it with a size of 4294967295 (=0xFFFFFFFF, or -1 if signed), therefore resulting in malloc throwing error 12 ("Out of memory").

This makes absolutely no sense, since I am obviously not trying to allocate 4 GB of memory, and since the size of my Game class is correctly computed by the compiler when printed, it should try to allocate the correct size.

Disassembling

To make sure that the new operator was indeed called, I opened the compiled .so file containing my code in Ghidra. The instructions corresponding to what we are interested it are as follows:

As can be seen, the new operator is indeed called. It is linked to an external function (since it's a shared library build), but there's no trace of a wrong malloc.

However, the allocated size passed into the new operator seems to be 0x108 instead of the printed 0x98. It is likely that the size Ghidra shows is the correct one, since the android print format seems to have problems inrelated to the described issue here (when using %X, the compiler warns and asks to use %lX, but when using %lXit warns and asks to use %X).

My Windows build gives a size of 0x130. My theory on that size difference is that pointer size, as well as other native structures, might be different.

Either way, no matter the correct size, it is far from being 0xFFFFFFFF, so the malloc error is unlikely related to the size difference.

My configuration

Here is how my project is configured.

First off, it is a Gradle project following the same structure as in the official SDL repo:

My app's Application.mk file (here) is as follows:

    # Uncomment this if you're using STL in your project
    # You can find more information here:
    # https://developer.android.com/ndk/guides/cpp-support
    APP_STL := c++_shared

    APP_ABI := armeabi-v7a arm64-v8a x86 x86_64

    # Min runtime API level
    APP_PLATFORM=android-16

As can be seen, I uncommented the APP_STL line because my project uses a lot of STL functions. It is set to c++_shared by default, but I have tried c++_static and faced the same results. Additionally, gnustl_shared and gnustl_static are deprecated and do not work with SDL projects anyway.

The CMakeLists.txt file located in the same folder (here) is as follows:

    cmake_minimum_required(VERSION 3.6)

    project(GAME)

    # armeabi-v7a requires cpufeatures library
    # include(AndroidNdkModules)
    # android_ndk_import_module_cpufeatures()


    # SDL sources are in a subfolder named "SDL"
    add_subdirectory(SDL)

    # Compilation of companion libraries
    add_subdirectory(SDL_image)
    #add_subdirectory(SDL_mixer)
    add_subdirectory(SDL_ttf)

    # Your game and its CMakeLists.txt are in a subfolder named "src"
    add_subdirectory(src)

The Android.mk file located in my project's C++ source folder (here) is as follows:

    LOCAL_PATH := $(call my-dir)

    include $(CLEAR_VARS)

    LOCAL_MODULE := main

    SDL_PATH := ../SDL

    LOCAL_C_INCLUDES := $(LOCAL_PATH)/$(SDL_PATH)/include $(LOCAL_PATH)/../SDL_image/include $(LOCAL_PATH)/../SDL_ttf/include $(LOCAL_PATH)/../include

    # Add your application source files here...
    LOCAL_SRC_FILES := main.cpp <all my other cpp files here, irrelevant so not including them>

    LOCAL_SHARED_LIBRARIES := SDL2 SDL2_image SDL2_ttf

    LOCAL_LDLIBS := -lGLESv1_CM -lGLESv2 -lOpenSLES -llog -landroid

    include $(BUILD_SHARED_LIBRARY)

I didn't edit any other configuration file as far as I remember (and checked). Feel free to ask if you need the content of a specific file I didn't include here.

Where am I with it now

This kind of problem is just so specific it that it is hard to find anyone who met any error close to this one, so I didn't find anything

As a reminder, the project compiles and works for Windows, the issue I'm encountering here is android-exclusive.

If I can theorize, I'd say that the c++ library simply has an option that I missed to support the new operator, but I couldn't find anything and I'm very new to android development, so I can't say for sure.

Answer 1

I found the issue.

As commenters pointed out, the constructor of the Game class was at fault, not the memory allocation itself.

I thought I had removed everything that could lead to a crash in it, but that was without knowing that members of the class get constructed even if it isn't explicitely stated, and that caused the crashed, as some of them request for files the Windows way.

Sorry!