I was working on a shared pointer (called Handle) implementation for my student project's game engine, and we ran into a bug that we couldn't explain. For some reason, at a certain point in our factory, there was an invalid internal pointer being passed to a factory through a handle, which was causing a crash during our archetype loading. We debugged through the process for hours, and broke any complex statements down to their most simple versions for easier debugging. I finally isolated the problem down to the Handle class' copy constructor. However, it still seemed like there was some intermediate step where the internal pointer was being freed. I read every article I could find on what could cause this problem, and didn't find anything. Finally, I decided to look at the disassembly and see if I could figure out what was happening. Here's the copy constructor without the disassembly.
template <class TYPE>
Handle<TYPE>::Handle(const Handle<TYPE>& rhs, bool weak) : m_weak(weak), m_ref_count(rhs.m_ref_count), m_ptr(rhs.m_ptr)
{
if(!m_weak)
++(*m_ref_count);
}
This is the copy constructor with the disassembly.
template <class TYPE>
Handle<TYPE>::Handle(const Handle<TYPE>& rhs, bool weak) : m_weak(weak), m_ref_count(rhs.m_ref_count), m_ptr(rhs.m_ptr)
{
013FFF50 push ebp
013FFF51 mov ebp,esp
013FFF53 sub esp,0CCh
013FFF59 push ebx
013FFF5A push esi
013FFF5B push edi
013FFF5C push ecx
013FFF5D lea edi,[ebp-0CCh]
013FFF63 mov ecx,33h
013FFF68 mov eax,0CCCCCCCCh
013FFF6D rep stos dword ptr es:[edi]
013FFF6F pop ecx
013FFF70 mov dword ptr [this],ecx
013FFF73 mov eax,dword ptr [this]
013FFF76 mov cl,byte ptr [weak]
013FFF79 mov byte ptr [eax],cl
013FFF7B mov eax,dword ptr [this]
013FFF7E mov ecx,dword ptr [rhs]
013FFF81 mov edx,dword ptr [ecx+4]
013FFF84 mov dword ptr [eax+4],edx
013FFF87 mov eax,dword ptr [this]
013FFF8A mov ecx,dword ptr [rhs]
013FFF8D mov edx,dword ptr [ecx+8]
013FFF90 mov dword ptr [eax+8],edx
if(!m_weak)
013FFF93 mov eax,dword ptr [this]
013FFF96 movzx ecx,byte ptr [eax]
013FFF99 test ecx,ecx
013FFF9B jne Handle<Component>::Handle<Component>+60h (013FFFB0h)
++(*m_ref_count);
013FFF9D mov eax,dword ptr [this]
013FFFA0 mov ecx,dword ptr [eax+4]
013FFFA3 mov edx,dword ptr [ecx]
013FFFA5 add edx,1
013FFFA8 mov eax,dword ptr [this]
013FFFAB mov ecx,dword ptr [eax+4]
013FFFAE mov dword ptr [ecx],edx
}
The problem (as far as I can tell) is here:
013FFF6D rep stos dword ptr es:[edi]
As everyone knows, this is used to clear memory quickly. However, this instruction is clearing the pointer that the rhs Handle's internal pointer-to-pointer points to, and not only that, but it's violating const, as the passed in Handle is a const reference. I'm not sure if it's a VS issue, some kind of flag that's set in our build settings that's causing it to happen, but the copy constructor works fine until we try to make archetypes with it.
Any help would be greatly appreciated!
EDIT 1: I have further researched the issue after some of the comments brought to light that I might be returning a temporary variable somewhere in my code. I wasn't able to find anything that would suggest I was doing so, but I did sift through some more disassembly to see if I could find any hints. Here is our CreateGenericArchetype function, which is where the problem is cropping up.
Handle<Object> ObjectFactory::CreateGenericArchetype(const std::wstring &ArchetypeName)
{
/* Create The Object */
auto archetype = mComponentFactoryMap.find(std::wstring(L"Object"));
Handle<Component> created_component = archetype->second->CreateArchetypeComponent();
Handle<Object> retVal = static_handle_cast<Object>(created_component);
retVal->mArchetypeName = ArchetypeName;
mArchetypeMap.insert(std::pair<std::wstring, Handle<Object>>(ArchetypeName, retVal));
return Handle<Object>(retVal, true);
}
The line that's causing all of our problems (at the moment) is this:
Handle<Component> created_component = archetype->second->CreateArchetypeComponent();
Naturally, I extended my debugging to CreateArchetypeComponent as well, so here is that too:
Handle<Component> ComponentTypeFactory<Object>::CreateArchetypeComponent() const
{
Object* created_object = new Object;
Component* casted_object = static_cast<Component*>(created_object);
Handle<Component> newComponent(casted_object);
newComponent->mType = mType;
newComponent->mID = GetNewID();
return newComponent;
}
Object inherits from Component, so the static_cast downcast of the Object to Component is valid, and the constructor successfully constructs the Handle from the Component*. The issue crops up when we try to return the Handle we constructed inside the function. Here is the disassembly of that interaction:
return newComponent;
005602AD push 0
005602AF lea eax,[newComponent]
005602B2 push eax
005602B3 mov ecx,dword ptr [ebp+8]
005602B6 call Handle<Component>::Handle<Component> (04EA050h)
005602BB mov ecx,dword ptr [ebp-110h]
005602C1 or ecx,1
005602C4 mov dword ptr [ebp-110h],ecx
005602CA mov byte ptr [ebp-4],0
005602CE lea ecx,[newComponent]
005602D1 call Handle<Component>::~Handle<Component> (04F2E7Bh)
005602D6 mov eax,dword ptr [ebp+8]
}
00EB02D9 push edx
00EB02DA mov ecx,ebp
00EB02DC push eax
00EB02DD lea edx,ds:[0EB0318h]
00EB02E3 call @_RTC_CheckStackVars@8 (0E4BA9Eh)
00EB02E8 pop eax
00EB02E9 pop edx
00EB02EA mov ecx,dword ptr [ebp-0Ch]
00EB02ED mov dword ptr fs:[0],ecx
00EB02F4 pop ecx
00EB02F5 pop edi
00EB02F6 pop esi
00EB02F7 pop ebx
00EB02F8 mov ecx,dword ptr [ebp-10h]
00EB02FB xor ecx,ebp
00EB02FD call @__security_check_cookie@4 (0E4E9BAh)
00EB0302 add esp,130h
00EB0308 cmp ebp,esp
00EB030A call __RTC_CheckEsp (0E41C3Dh)
00EB030F mov esp,ebp
00EB0311 pop ebp
00EB0312 ret 4
*******back to CreateGenericArchetype*******
005C2639 call __RTC_CheckEsp (04F1C3Dh)
005C263E mov dword ptr [ebp-1D4h],eax
005C2644 mov ecx,dword ptr [ebp-1D4h]
005C264A mov dword ptr [ebp-1D8h],ecx
005C2650 mov byte ptr [ebp-4],4
005C2654 mov edx,dword ptr [ebp-1D8h]
005C265A push edx
005C265B lea ecx,[created_component]
005C265E call Handle<Component>::Handle<Component> (04EA050h)
*******copy constructor disassembly from above here*******
005C2663 mov byte ptr [ebp-4],6
005C2667 lea ecx,[ebp-174h]
005C266D call Handle<Component>::~Handle<Component> (04F2E7Bh)
Handle<Object> retVal = static_handle_cast<Object>(created_component);
Unless my copy constructor being called with the "created_component" as the rhs Handle is considered returning a local variable, I can't see where it could possibly be going wrong, unless it's because the returned Handle from CreateArchetypeComponent is on the stack when it's passed to the copy constructor and is then being cleared.
Thanks for all the great help with this issue, but we figured out what was going on. What was happening is that in some of the Handle constructors we needed to allocate memory for the pointer that the Handle's internal pointer-to-pointer was going to be pointing at. So what was happening was that the Handle itself was using a pointer that was held on the stack, and when the rep stos opcode was called, it was clearing out said pointer.