I need to fix a huge commercial C++ project under Linux. It is build with GCC (specifically g++), under customly written build system in a company. It is built and run under Raspberry Pi 4B.
When it is built dynamically (without -static flag) then everything works fine and program runs correctly and gives correct results.
But when it is built statically (with -static flag) then program crashes on any first exception. To check that program is static I run ldd ./prog and it shows not a dynamic executable. If to do same ldd command for dynamical build then it shows a bunch of dynamical libraries.
I decided to leave whole build config totally same but in main file just replaced main() function with following simple code:
int main() {
std::cout << "Before exc" << std::endl;
try {
throw std::runtime_error("Hello, World!");
} catch (std::exception const & ex) {
std::cout << "Thrown '" << ex.what() << "'." << std::endl;
}
std::cout << "After exc" << std::endl;
}
If whole project with main() from above is built in dynamic then it correctly shows following, as expected:
Before exc
Thrown 'Hello, World!'.
After exc
And if I build exactly same thing statically then output is such:
Before exc
terminate called after throwing an instance of 'std::runtime_error'
terminate called recursively
Aborted
In other words message Before exc is shown, then exception is thrown, but instead of catching it terminate is called and program aborted.
If with same build system I build not the whole project, but statically build just a single file main.cpp with code above, then it outputs expected thing. So the problem here with all the project files and linked libraries, not with the build system (probably).
Maybe some global variables of the project when get initialized modify program in such a way that throwing an exception causes termination. But all global variables were initialized correctly because main() runs correctly till point where exception is thrown. Global variables only made static build exception-unfriendly.
Does anyone have any idea what is happenning and when C++ exception calls terminate at the very point of throw? Does there exist any program config that can set exception throwing to call std::terminate()?
Also anything else is working correctly, only exceptions suffer. In other words statical program runs correctly for some time and does correct actions, but crashes on very first exception.
Build command that was used is below, all names modified intentionally, also only 2-3 object and library files are left, in real example there are dozens of same .a libraries and dozens of .o objects. First goes compile command that compiles .cpp to .o, second goes archive command that aggregates .o into .a, finally goes link command that links all .o and .a files into final single-file program:
"ccache" "g++" "-finput-charset=UTF-8" "-fexec-charset=UTF-8" "-std=c++2a" "-fpic" "-c" "-static" "-static-libgcc" "-static-libstdc++" "-Iinc_dir/" "-I." "-g" "-march=armv6zk+fp" "-ffast-math" "-xc++" "-O0" "-DSOME_DEFINE" "-omisc.o" "misc.cpp"
"ar" "rs" "lib1.a" "lib1.o"
"ccache" "g++" "-finput-charset=UTF-8" "-fexec-charset=UTF-8" "-std=c++2a" "-fpic" "-static" "-static-libgcc" "-static-libstdc++" "-oprog" "main.o" "misc.o" "lib1.a" "lib2.a" "-L./lib_dir/" "-lboost_program_options" "-lboost_system" "-lpthread" "-lc" "-lgcc" "-ldl" "-lstdc++" "-lstdc++fs"
By removing parts of project through binary search just figured out that reason is inside misc.cpp. Somehow if I include it into build then exceptions crash, if I don't include then exceptions don't crash.
As far as I understand the only reason why misc.cpp can make exceptions crash is only due to some global variables inside main.cpp doing such settings of a program that make exceptions crashable. To remind, I included throw.cpp into project that throws toy exception as shown in code above. misc.cpp is not used at all from throw.cpp. Also main.cpp is not included in build. Basically I left only misc.cpp file from project, and only it is causing errors. All the rest of project build config are only external libraries that don't cause crashing.
Right now I made a minimal reproducible example. Now my project consists of throw.cpp and misc.cpp plus a bunch of external linked libraries. This throw.cpp contains just main() with throwing toy code that I mentioned in the beginning of my question, it has nothing else, it doesn't include anything except iostream and stdexcept. misc.cpp has no main(), and misc.cpp is not used at all, just compiled and linked. So now if I remove misc.cpp from linking then exception are thrown in main() of throw.cpp correctly without error. If I link misc.cpp (that I don't use at all, just link) then exceptions thrown in main() of throw.cpp make program crash, only in static link.
After 6 hours of search for problem, I managed to figure out that the reason is not in the project files / libraries / build system.
It appeared that the problem is in combining any C++ code with exceptions and C code with usage of sscanf() (or even maybe any C standard library function).
Probably this happens due to Raspberry Pi 4B having too old GCC/G++ version in default installation, g++ -v produces:
Target: arm-linux-gnueabihf
...
Thread model: posix
gcc version 8.3.0 (Raspbian 8.3.0-6+rpi1)
so version is 8.3.0, while there exists already release 11.2 of GCC.
I created minimal example that crashes:
File z.cpp:
#include <stdexcept>
#include <iostream>
int z2_f();
int main() {
std::cout << "Before exc" << std::endl;
try {
throw std::runtime_error("Hello, World!");
} catch (std::exception const & ex) {
std::cout << "Thrown '" << ex.what() << "'." << std::endl;
} catch (...) {
std::cout << "Thrown 'Unknown'." << std::endl;
}
std::cout << "After exc" << std::endl;
std::cout << z2_f() << std::endl;
}
File z2.cpp:
#include <cstdio>
int z2_f() {
char s[] = "123";
int x = 0;
sscanf(s, "%d", &x);
return x;
}
Compile commands:
"g++" "-std=c++2a" "-c" "-static" "-O0" "-oz.o" "z.cpp"
"g++" "-std=c++2a" "-c" "-static" "-O0" "-oz2.o" "z2.cpp"
"g++" "-std=c++2a" "-static" "-oprog.exe" "z.o" "z2.o"
After compiling and running prog.exe it shows in console:
Before exc
terminate called after throwing an instance of 'std::runtime_error'
terminate called recursively
Aborted
If one compiles and runs only first file z.cpp (with removed use of z2_f()), then everything works as expected without crash:
Before exc
Thrown 'Hello, World!'.
After exc
I managed to solve the issue by installing GCC/G++ version 10.1.0 according to this tutorial (WayBack mirror of page). Basically binary author's release of GCC can be obtained through git clone https://bitbucket.org/sol_prog/raspberry-pi-gcc-binary.git --depth=1.
With this version toy example above runs without crash, original large project also runs without crashes.
To remind, before I had version 8.3.0 of GCC, which was crashing on exception.
My uname -a shows:
Linux raspberrypi 5.4.51-v7l+ #1333 SMP Mon Aug 10 16:51:40 BST 2020 armv7l GNU/Linux
while lsb_release -d shows:
Raspbian GNU/Linux 10 (buster)