I have a Windows fatal exception: code 0xc0000374 - yes there's multiprocessing (wait for the but...). Google says that the exception code 0xc0000374 indicates a heap corruption. Yes, multiprocessing is a must-have. It's apart of the framework I'm working in, as each bot has the potential to have its own core to run in. TL;DR I can't change the fact that there's multiprocessing. However, my bot only runs on one thread, so there shouldn't really be an issue, and in fact, this issue is relatively new.
I think I've found the problem, but it doesn't really make much sense. I'm extending Python with C in order to improve run times, and I think this is where the error is. I've narrowed it down to a function called ground_shot_is_viable, as when I comment it out in Python the error never happens. However, when I try print spamming (in this case I actually wrote to a file as that's more suitable for hundreds of prints) I found that the function successfully completed. I think the error is that the function oversteps it's memory boundaries, which corrupts a portion of data, causing the crash traceback to point elsewhere. (In this case, it's an innocent line in the framework I'm working with - File "G:\RLBotGUIX\venv\lib\site-packages\rlbot\utils\rendering\rendering_manager.py", line 104 in end_rendering which sets a variable to False)
I've also tested this for the other functions, and they don't cause this issue for some reason. There's a slight, small potential that it's because they don't get called as often as ground_shot_is_viable.
The error only happens after a few minutes which prob totals to at least a few hundred times, maybe a thousand. (The bot runs at upwards of 120tps, so the function has the potential to be called 120 times in a second.)
I only managed to get the traceback by setting the environment variable PYTHONFAULTHANLDER to 1 - when I didn't, my program just silently crashed.
I also didn't get a crash dump when I launched the program with python.exe, but with pythonw.exe I did get a crash dump.
Traceback:
Windows fatal exception: code 0xc0000374
Thread 0x00008004 (most recent call first):
File "G:\RLBotGUIX\Python37\lib\threading.py", line 296 in wait
File "G:\RLBotGUIX\Python37\lib\multiprocessing\queues.py", line 224 in _feed
File "G:\RLBotGUIX\Python37\lib\threading.py", line 870 in run
File "G:\RLBotGUIX\Python37\lib\threading.py", line 926 in _bootstrap_inner
File "G:\RLBotGUIX\Python37\lib\threading.py", line 890 in _bootstrap
Current thread 0x000031e0 (most recent call first):
File "G:\RLBotGUIX\venv\lib\site-packages\rlbot\utils\rendering\rendering_manager.py", line 104 in end_rendering
File "G:\RLBotGUIX\venv\lib\site-packages\rlbot\botmanager\bot_manager_struct.py", line 69 in call_agent
File "G:\RLBotGUIX\venv\lib\site-packages\rlbot\botmanager\bot_manager.py", line 250 in perform_tick
File "G:\RLBotGUIX\venv\lib\site-packages\rlbot\botmanager\bot_manager.py", line 206 in run
File "G:\RLBotGUIX\venv\lib\site-packages\rlbot\setup_manager.py", line 617 in run_agent
File "G:\RLBotGUIX\Python37\lib\multiprocessing\process.py", line 99 in run
File "G:\RLBotGUIX\Python37\lib\multiprocessing\process.py", line 297 in _bootstrap
File "G:\RLBotGUIX\Python37\lib\multiprocessing\spawn.py", line 118 in _main
File "G:\RLBotGUIX\Python37\lib\multiprocessing\spawn.py", line 105 in spawn_main
File "<string>", line 1 in <module>
Crash dump:
EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 00007ffb96b7dace (ucrtbase!abort+0x000000000000004e)
ExceptionCode: c0000409 (Security check failure or stack buffer overrun)
ExceptionFlags: 00000001
NumberParameters: 1
Parameter[0]: 0000000000000007
Subcode: 0x7 FAST_FAIL_FATAL_APP_EXIT
PROCESS_NAME: pythonw.exe
ERROR_CODE: (NTSTATUS) 0xc0000409 - The system detected an overrun of a stack-based buffer in this application. This overrun could potentially allow a malicious user to gain control of this application.
EXCEPTION_CODE_STR: c0000409
EXCEPTION_PARAMETER1: 0000000000000007
STACK_TEXT:
000000d4`287ef660 00007ffb`2baf1bb7 : 00000295`00000003 00000000`00000003 00000000`ffffffff 00007ffb`2bc0a3d8 : ucrtbase!abort+0x4e
000000d4`287ef690 00007ffb`2baf17c3 : 000000d4`287ef9a0 000000d4`287ef800 00000000`00000000 00000000`00000000 : python37!Py_RestoreSignals+0x14b
000000d4`287ef6d0 00007ffb`2b9e94a9 : 000000d4`287ef9a0 00000000`00000000 00000295`edd52050 00000000`00000000 : python37!Py_FatalInitError+0x1f
000000d4`287ef700 00007ffb`2b9a09ce : 000000d4`287ef9a0 00000295`edd52050 00000000`00000000 00000000`00000000 : python37!PyErr_NoMemory+0x2ad5d
000000d4`287ef930 00007ffb`2b9a09b6 : 0000ab32`10364489 00007ff7`28481e7e 00000000`00000000 00007ffb`96b29f66 : python37!Py_Main+0x6e
000000d4`287ef960 00007ff7`28481277 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : python37!Py_Main+0x56
000000d4`287efa10 00007ffb`97f07c24 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : pythonw+0x1277
000000d4`287efa50 00007ffb`98f8d4d1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
000000d4`287efa80 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21
SYMBOL_NAME: ucrtbase!abort+4e
MODULE_NAME: ucrtbase
IMAGE_NAME: ucrtbase.dll
STACK_COMMAND: ~0s ; .ecxr ; kb
FAILURE_BUCKET_ID: FAIL_FAST_FATAL_APP_EXIT_c0000409_ucrtbase.dll!abort
Minimal reproducible example (untested, actually, but the full version of my program is ofc tested)
#include <math.h>
#include <Python.h>
#include <string.h>
// Constants
#define simulation_dt 0.05
#define physics_dt 0.008333333333333333333333
#define jump_max_duration 0.2
#define jump_speed 291.6666666666666666666666
#define jump_acc 1458.3333333333333333333333
#define aerial_throttle_accel 66.6666666666666666666666
#define boost_consumption 33.3333333333333333333333
#define brake_force 3500.
#define max_speed 2300.
#define max_speed_no_boost 1410.
#define start_throttle_accel_m -1.02857142857142857
#define start_throttle_accel_b 1600.
#define end_throttle_accel_m -16.
#define end_throttle_accel_b 160.
#define PI 3.14159265358979323846
// simple math stuff
static inline signed char sign(int value)
{
return (value > 0) - (value < 0);
}
// Vector stuff
typedef struct vector
{
double x;
double y;
double z;
} Vector;
static inline double dot(Vector vec1, Vector vec2)
{
return vec1.x * vec2.x + vec1.y * vec2.y + vec1.z * vec2.z;
}
static inline double magnitude(Vector vec)
{
return sqrt(dot(vec, vec));
}
static inline Vector flatten(Vector vec)
{
return (Vector){vec.x, vec.y, 0};
}
Vector normalize(Vector vec)
{
double mag = magnitude(vec);
if (mag != 0)
return (Vector){vec.x / mag, vec.y / mag, vec.z / mag};
return (Vector){0, 0, 0};
}
static inline double angle(Vector vec1, Vector vec2)
{
return acos(cap(dot(normalize(vec1), normalize(vec2)), -1, 1));
}
static inline double angle2D(Vector vec1, Vector vec2)
{
return angle(flatten(vec1), flatten(vec2));
}
// orientation matrix stuff
typedef struct orientation
{
Vector forward;
Vector left;
Vector up;
} Orientation;
static inline Vector localize(Orientation or_mat, Vector vec)
{
return (Vector){dot(or_mat.forward, vec), dot(or_mat.left, vec), dot(or_mat.up, vec)};
}
// hitboxes
typedef struct hitbox
{
double length;
double width;
double height;
Vector offset;
} HitBox;
// other definitions
typedef struct car
{
Vector location;
Vector velocity;
Orientation orientation;
Vector angular_velocity;
_Bool demolished;
_Bool airborne;
_Bool supersonic;
_Bool jumped;
_Bool doublejumped;
unsigned char boost;
unsigned char index;
HitBox hitbox;
} Car;
// extra math functions
double throttle_acceleration(double car_velocity_x)
{
double x = fabs(car_velocity_x);
if (x >= 1410)
return 0;
// use y = mx + b to find the throttle acceleration
if (x < 1400)
return start_throttle_accel_m * x + start_throttle_accel_b;
x -= 1400;
return end_throttle_accel_m * x + end_throttle_accel_b;
}
// physics simulations
_Bool can_reach_target_forwards(double max_time, double jump_time, double boost_accel, double distance_remaining, double car_speed, int car_boost)
{
double *v = &car_speed;
double t = 0;
double b = car_boost;
double *d = &distance_remaining;
double ba_dt = boost_accel * simulation_dt;
double ms_ba_dt = max_speed - ba_dt;
double bc_dt = boost_consumption * simulation_dt;
double bk_dt = brake_force * simulation_dt;
while (*d > 25 && t <= max_time && (*v <= 0 || *d / *v > jump_time))
{
*v += (*v < 0) ? bk_dt : throttle_acceleration(*v) * simulation_dt;
if (b > bc_dt && *v < ms_ba_dt)
{
*v += ba_dt;
if (b <= 100)
b -= bc_dt;
}
*d -= *v * simulation_dt;
t += simulation_dt;
}
double th_dt = aerial_throttle_accel * simulation_dt;
while (*d > 25 && t <= max_time)
{
// yes, this IS max_speed, NOT max_speed_no_boost!
if (*v <= max_speed - th_dt)
*v += th_dt;
if (b > bc_dt && *v < ms_ba_dt)
{
*v += ba_dt;
if (b <= 100)
b -= bc_dt;
}
*d -= *v * simulation_dt;
t += simulation_dt;
}
return *d <= 25;
}
_Bool can_reach_target_backwards(double max_time, double jump_time, double distance_remaining, double car_speed)
{
double *v = &car_speed;
double t = 0;
double *d = &distance_remaining;
double bk_dt = brake_force * simulation_dt;
while (*d > 25 && t <= max_time && (*v >= 0 || *d / (-*v) > jump_time))
{
*v -= (*v > 0) ? bk_dt : throttle_acceleration(*v) * simulation_dt;
*d += *v * simulation_dt;
t += simulation_dt;
}
double th_dt = aerial_throttle_accel * simulation_dt;
double ms_th_dt = max_speed - th_dt;
while (*d > 25 && t <= max_time)
{
// yes, this IS max_speed, NOT max_speed_no_boost!
if (-*v <= ms_th_dt)
*v -= th_dt;
*d += *v * simulation_dt;
t += simulation_dt;
}
return *d <= 25;
}
// Parsing shot slices
_Bool generic_is_viable(double *T, double jump_time, double *boost_accel, Car *me, Vector *direction, double *distance_remaining)
{
if (*T <= 0 || *distance_remaining / *T > 2300)
return 0;
double forward_angle = angle2D(*direction, me->orientation.forward);
double backward_angle = PI - forward_angle;
double forward_time = *T - (forward_angle * 0.418);
double backward_time = *T - (backward_angle * 0.318);
double true_car_speed = dot(me->orientation.forward, me->velocity);
double car_speed = magnitude(me->velocity) * sign((int)true_car_speed);
jump_time *= 1.05;
_Bool forward = forward_time > 0 && can_reach_target_forwards(forward_time, jump_time, *boost_accel, *distance_remaining, car_speed, me->boost);
_Bool backward = backward_time > 0 && forward_angle >= 1.6 && true_car_speed < 1000 && can_reach_target_backwards(backward_time, jump_time, *distance_remaining, car_speed);
return forward || backward;
}
_Bool ground_shot_is_viable(double *T, double *boost_accel, Car *me, double *offset_target_z, Vector *direction, double *distance_remaining)
{
if (*offset_target_z >= (92.75 + (me->hitbox.height / 2)) || me->airborne)
return 0;
return generic_is_viable(T, 0, boost_accel, me, direction, distance_remaining);
}
// Linking the C functions to Python methods
static PyObject *method_ground_shot_is_viable(PyObject *self, PyObject *args)
{
double T, boost_accel, offset_target_z, distance_remaining;
Vector direction;
Car me;
_Bool shot_viable = 0;
// args are for >= 1.11
if (!PyArg_ParseTuple(args, "dd((ddd)(ddd)((ddd)(ddd)(ddd))(ddd)bbbbbbb(ddd)(ddd))d(ddd)d", &T, &boost_accel, &me.location.x, &me.location.y, &me.location.z, &me.velocity.x, &me.velocity.y, &me.velocity.z, &me.orientation.forward.x, &me.orientation.forward.y, &me.orientation.forward.z, &me.orientation.left.x, &me.orientation.left.y, &me.orientation.left.z, &me.orientation.up.x, &me.orientation.up.y, &me.orientation.up.z, &me.angular_velocity.x, &me.angular_velocity.y, &me.angular_velocity.z, &me.demolished, &me.airborne, &me.supersonic, &me.jumped, &me.doublejumped, &me.boost, &me.index, &me.hitbox.length, &me.hitbox.width, &me.hitbox.height, &me.hitbox.offset.x, &me.hitbox.offset.y, &me.hitbox.offset.z, &offset_target_z, &direction.x, &direction.y, &direction.z, &distance_remaining))
{
return NULL;
// removed legacy code stuff
}
else
{
shot_viable = ground_shot_is_viable(&T, &boost_accel, &me, &offset_target_z, &direction, &distance_remaining);
}
return (shot_viable) ? Py_True : Py_False;
}
static PyMethodDef methods[] = {
{"ground_shot_is_viable", method_ground_shot_is_viable, METH_VARARGS, "Check if a ground shot is viable"},
{NULL, NULL, 0, NULL}};
static struct PyModuleDef module = {
PyModuleDef_HEAD_INIT,
"gstest",
"Test thing",
-1,
methods};
PyMODINIT_FUNC PyInit_test(void)
{
return PyModule_Create(&module);
};
This code is used in conjunction with Python (ofc...) and the RLBot python package. The bot runs in Rocket League, and it plays the game. The RLBot framework itself doesn't crash, just the bot. I've tested this by running 2 bots at once, and only my bot crashed. The other bot (or bots, actually) were unaffected.
My system has 20GB of ram and the issue was happening with me at 50%-75% capacity, so my system's amount of ram isn't the issue. I'm not the best at testing for memory leaks, but it does appear to go up at about 0.1MB every 30 seconds to a minute. This isn't much as the bot starts out by taking up around about 30MB of ram.
This issue has been bugging me for almost a month and I'm trying to bring it before the firing squad but the damn thing is annoying af.
I was very hesitant in posting this issue on SO because I wasn't quite sure what to put. I hope I've provided everything that's required! If you want the full C program, or if you want the setup.py file and stuff, just ask and I'll provide a hastebin or something. Or if you want the full bot I can prob upload a zip file to dropbox, as the bot is fairly large at multiple thousands of lines of code and multiple files.
The bug is most likely in this line of method_ground_shot_is_viable:
return (shot_viable) ? Py_True : Py_False;
Functions registered using PyMethodDef must return a new reference, which means they must explicitly increment the reference count when returning a global object like Py_True or Py_False. Failing to do so results in the caller decreasing the reference count of the returned object without it being previously increased. After method_ground_shot_is_viable has been called enough times, the reference count of True or False drop to zero and it get deallocated, leading to use-after-free.
Incrementing the reference count can be done by applying the Py_INCREF macro to Py_True or Py_False as appropriate before returning them. You can also use the boolean-returning convenience macros that take care of reference counting:
if (shot_viable)
Py_RETURN_TRUE;
else
Py_RETURN_FALSE;