Search code examples
c++assemblymemory64-bitgame-engine

Values of a vector are changing when they shouldn't be


I am writing a game engine from scratch as a free-time learning exercise. I am currently implementing a rendering queue, but the values of the vector in charge of the queue keep changing. (always to the same value of -107374176 when it should be 10.0f) The vector objRID is of type OBJR*, where OBJR is a struct containing position information, as well as a pointer to a bitmap. The bitmap library I am using doesn't seem to be the culprit, but it can be found at: http://partow.net/programming/bitmap/index.html.

The overarching exception is a read access violation of 0x1CCCCCCCC. I have stepped through the program and have found that the values of the struct change one by one every iteration of the "rep stos" after the 19th iteration. I have no real Idea as to how the "rep stos" could affect something which is seemingly unrelated. (A don't have a great grasp on assembler in the first place) I am very open to suggestions besides the error at hand.

If someone could explain how the following assembly affects the vector objRID I think I would be able to solve this problem myself in the future.

163: int loop()
164: {
00007FF7AC74D580 40 55                push        rbp  
00007FF7AC74D582 57                   push        rdi  
00007FF7AC74D583 48 81 EC A8 01 00 00 sub         rsp,1A8h  
00007FF7AC74D58A 48 8D 6C 24 20       lea         rbp,[rsp+20h]  
00007FF7AC74D58F 48 8B FC             mov         rdi,rsp  
00007FF7AC74D592 B9 6A 00 00 00       mov         ecx,6Ah  
00007FF7AC74D597 B8 CC CC CC CC       mov         eax,0CCCCCCCCh  
00007FF7AC74D59C F3 AB                rep stos    dword ptr [rdi] <---- 19th - 26th iteration here

I hate to just throw the whole program in here, but I believe it is much less confusing this way.

The program is structured as such:

#include "stdafx.h"
#include <Windows.h>
#include "bitmap_image.hpp"
#define maxObjects 1024




struct VEC2_f {
    float x, y;

    VEC2_f(float x, float y)
    {
        VEC2_f::x = x;
        VEC2_f::y = y;
    }

    VEC2_f()
    {
        VEC2_f::x = 0.0f;
        VEC2_f::y = 0.0f;
    }
};

struct OBJR {
    VEC2_f pos, vel;
    int ID = -1;
    bitmap_image* Lbmp;

    OBJR(bitmap_image* Lbmp, VEC2_f pos, VEC2_f vel)
    {
        OBJR::Lbmp = Lbmp;
        OBJR::pos = pos;
        OBJR::vel = vel;
    }

    OBJR(bitmap_image* Lbmp, float x, float y, float vx, float vy)
    {
        OBJR::Lbmp = Lbmp;
        OBJR::pos = VEC2_f(x, y);
        OBJR::vel = VEC2_f(vx, vy);
    }

    //if -1 then ID isn't set yet
    int getID()
    {
        return ID;
    }
};



std::vector<OBJR*> objRID;
int IDCOUNTER = 0;
bool running = true;
HWND con;
HDC dc;
COLORREF color;



void objInit(OBJR* Lobj)
{
    if (objRID.size() > maxObjects)
    {
        objRID.pop_back();
        Lobj->ID = maxObjects;          }
    Lobj->ID = IDCOUNTER++;
    objRID.push_back(Lobj);
}

void input()
{

}

void update()
{

}

VEC2_f interpolate(float interpolation, VEC2_f pos, VEC2_f vel)
{
    return VEC2_f(pos.x + (vel.x * interpolation), pos.y + (vel.y * interpolation));
}




void renderBitmap(bitmap_image* Lbmp, VEC2_f Ipos)
{
    unsigned int h, w;
    rgb_t colorT;
    h = Lbmp->height();  <--- Read access violation here
    w = Lbmp->width();

    for (unsigned int y = 0; y < h; y++)
    {
        for (unsigned int x = 0; x < w; x++)
        {
            colorT = Lbmp->get_pixel(x, y);
            color = RGB(colorT.red, colorT.green, colorT.blue);
            SetPixelV(dc, x + Ipos.x, y + Ipos.y, color);
        }
    }
}

void renderOBJR(float interpolation, OBJR* obj)
{
    renderBitmap(obj->Lbmp, interpolate(interpolation, obj->pos, obj->vel));
}


void render(float interpolation)
{
    for (int i = 0; i < objRID.size(); i++)
    {
        renderOBJR(interpolation, objRID[i]);
    }
}


void resizeWindow()
{
    RECT r;
    GetWindowRect(con, &r);
    MoveWindow(con, r.left, r.top, 800, 600, true);
}


int init()
{
    con = GetConsoleWindow();
    dc = GetDC(con);
    resizeWindow(); 
    return 0;
}


int loop()
{  //<--- this is where the disassembly was taken from and is where the Lbmp becomes invalid
    const int TPS = 60;
    const int SKIP_TICKS = 1000 / TPS;
    const int FRAMESKIP = 1;

    DWORD next_tick = GetTickCount();
    float interpolation;
    int loop;

    while (running)
    {
        loop = 0;
        while (GetTickCount() > next_tick && loop < FRAMESKIP)
        {
            input();
            update();

            next_tick += SKIP_TICKS;
            loop++;
        }
        interpolation = float(GetTickCount() + SKIP_TICKS - next_tick) / float(SKIP_TICKS);
        render(interpolation);

    }
    return 0;
}

int deInit()
{

    ReleaseDC(con, dc);
    return 0;
}



void test() 
{
    bitmap_image bitmap = bitmap_image("testBW.bmp");
    VEC2_f pos = VEC2_f(10.f, 10.f);
    VEC2_f vel = VEC2_f();
    OBJR test1 = OBJR(&bitmap, pos, vel);
    objInit(&test1);
    renderBitmap(&bitmap, pos);

}

int main()
{

    init();
    test();
    loop();
    deInit();
    return 0;
}

Solution

  • rep stosd is one way to implement memset. Some compilers will inline it.

    It looks like your compiler is using it to init stack memory with a poison value of 0xCCCCCCCC, for 4 * 0x6A bytes. (Notice that there's a mov rdi, rsp right before it.) I assume this is a debug build, because optimized builds wouldn't do that extra work.

    The overarching exception is a read access violation of 0x1CCCCCCCC.

    Looks like you read a pointer from an uninitialized object, and the "poison" value did its job of creating a pointer value that you can easily see is bogus, and which faults instead of silently continuing until some later fault farther away from the real problem.

    The 0x00000001 in the high half of the pointer is suspicious. Perhaps you partially overwrote this object? Set a watch point on the memory where the pointer value is stored, and find out what else modifies it.


    Or you're keeping pointers to local variables after they've gone out of scope, and then the next function call reuses that stack memory for its stack frame. The debug-mode code poisons its whole stack frame, overwriting some of the objects pointed to by your std::vector<OBJR*> objRID;.

    So it's not the vector contents or the vector object itself that are changing, it's the objects pointed to by the vector contents.

    Again, the 0xCCCCCCCC poison is doing its job of finding bugs in your program.

    (@molbdnilo spotted this. I didn't take the time to read your whole C++ code that carefully the first time.)