I want to search for a pattern in memory of current process. I query for memory pages using VirtualQuery to extract ranges of interest. The problem is that after 5 minutes of continous looping the program crashes by a stackoverflow. The size of allocated memory by the process gets bigger.
The problem resides in this loop:
valid_range:
pushad
mov eax, [ebp]
mov esi, [eax] ; memory range start
mov ecx, [eax + 12] ; memory range size
xor eax, eax
loopmem:
lodsb ; this causes stackoverflow after certain time ????
dec ecx
cmp ecx, 0 ; we loop trough memory until we finish it
; reduced the code to minimum
je finish_range
jmp loopmem
finish_range:
popad
ret
You are probably probing pages that are reserved for stack. That may cause new stack pages to be allocated and eventually result in stack overflow. From Microsoft Support:
In the Microsoft Windows NT operating system, stack overflow is detected by hardware and software working together, using the page protection mechanisms. Each new Windows NT process has a maximum reserved stack size and an initial committed stack allocation. Committed memory is physically allocated to the process and is backed by the page file; it is a relatively "expensive" resource. Reserved memory is address space that is not mapped to real memory; it is a relatively "cheap" resource.
As the stack grows, it moves from the committed portion of stack memory into the reserved or uncommitted memory. When this happens, a page fault occurs and the operating system commits another page of memory to the stack. If a page fault occurs when the stack has already grown to its maximum specified size, the system reports a stack overflow exception.
This automatic growth method uses a guard page, a reserved, uncommitted, memory page that is contiguous with the committed portion of memory. When the application touches the guard page, the operating system commits that page and the next uncommitted page becomes the new guard page. Automatic stack growth works only for the guard page and stack memory must grow in 4K, or one page, increments. If the application touches another reserved but uncommitted page of stack memory before it touches the guard page, a normal page fault exception occurs and unpredictable behavior can result.