Sieve of Eratosthenes x86 Assembly

I am new to assembly, trying to implement Sieve of Erathothenes, I have the code but it only works between 1 and 600, for some reason it goes haywire when I put n=1000, here is the complete code, any help will do.

Include Irvine32.inc
n=1000

.data
prime DWORD n DUP(?)

.code
main PROC
        mov ecx,n       ;initialize ecx to 1000
        mov esi, OFFSET prime
;--------------------------------------------------------
;   Initialize Array with 1's
;--------------------------------------------------------
FillArray:  mov eax,1
        mov [esi],eax
        mov eax,[esi]
        add esi,TYPE DWORD              ;4 to jump to the next index
        loop FillArray

    mov ebx,8
    mov edx,0
    mov ecx,n
    mov ebp,2
    mov esi,0
;----------------------------------------------------------------
;   Sieve
;----------------------------------------------------------------
SIEVE:      mov eax,prime[ebx]
        mov edx,ebx
        mov edi,ebx
        add edx,edi                 ;j=i+i
        mov esi,ebp
        mov ecx,n
        sub ecx,ebp
        add ecx,2
        add ebx,TYPE DWORD
        cmp eax,1
        je FLIP
        inc ebp
        loop SIEVE
        jmp K

FLIP:       mov eax,0
        mov prime[edx],eax
        add edx,edi
        cmp esi,n
        jg SIEVE
        add esi,ebp
        loop FLIP

K:

    mov esi, OFFSET prime
    mov ecx,n
    sub ecx,2   
    mov ebx,1
    mov edx,8           ;Start checking from index of second element

PRINT:  mov eax,prime[edx]          ;
        add edx,TYPE DWORD
        inc ebx
        cmp eax,1
        jne H
        mov eax,ebx
        call WriteDec
        call Crlf
        loop PRINT
        jmp D
H: loop PRINT
D:
    exit

main ENDP
END main

Solution

You were not specific enough with what "going haywire" means, and you probably didn't debug your code, nor you commented it enough to make it easy to follow, so after few minutes looking at it I followed my instinct, which was telling me to start from scratch and write it in regular way.

Whether this will help to you - I have no idea, probably not, at least not without hefty "analyse this" effort.

But I believe this is valid answer for a question with title "Sieve of Eratosthenes x86 Assembly" and "any help will do".

So here you go, try to understand completely foreign source and try to grasp the ideas used; to a point, where it will start to help you when writing your own code. Good luck. :)

First thing ahead of some code, you should clean up your task in terms of math first.

If you would, you would know that there's no point to "sieve" further beyond sqrt(n) value.

Because if sqrt(n) is prime, then its multiplies 2*sqrt(n), 3*sqrt(n), ..., previous_prime*sqrt(n) are already "sieved" from those smaller prime numbers sieving earlier in the loop.

The first "yet unsieved" number is sqrt(n) * sqrt(n), that's n itself, and that's out of array bounds. So after reaching number >= sqrt(n) you can end your algorithm main loop (I'm using sharp number > sqrt(n) test, because the sqrt(n) is truncated to integer, so for special "n" (with integer sqrt(n)) I will try to "sieve" n itself once, but that will be detected by the test inside flip_loop and terminated anyway.

Include Irvine32.inc
n=1000
n_terminal=31
    ; terminal compare value is min(sqrt(n), 0FFFFh) (last value to process)
    ; no point to sieve beyond sqrt(n), because first unset multiply is
    ; sqrt(n)*sqrt(n) = n => beyond array
    ; and no point to sieve 10000h, because 10000h*10000h is out of 32b

.data
prime BYTE n DUP(?)

.code
main PROC
    ; fill array with "1" (from number 2 to n)
    mov   edi,OFFSET prime+2
    mov   ecx,n-2
    mov   eax,1         ; eax=1 will be used also later (also ah=0)
    rep stosb
    ; set "0" and "1" as non-prime too
    mov   WORD PTR [prime],cx   ; (cx == 0 after "rep stosb")
    ; set "0" to all other non-primes by sieving
    mov   esi,eax       ; esi=1, start the loop as if "1" was handled last
sieve_loop:
    inc   esi           ; next number to test
    ; sqrt(n) is enough to process, beyond that n <= number*number => out of array
    cmp   esi,n_terminal
    ja    sieve_end     ; out of table, end sieving
    cmp   prime[esi],al ; compare with "1"
    jne   sieve_loop    ; not a prime, don't flip the rest of sieve
    ; esi is prime, flip the other multiplies of it, starting from esi*esi (!)
    mov   edi,esi       ; esi*esi as smaller multiples are already covered by
    imul  edi,edi       ; smaller prime numbers sieving the array before
flip_loop:
    cmp   edi,n         ; check if the multiply points beyond table
    jae   sieve_loop    ; if yes, continue main loop with next "prime"
    mov   prime[edi],ah ; set this number to "0" (not prime)
    add   edi,esi       ; edi = next multiply of esi prime
    jmp   flip_loop

sieve_end:
    ; print all primes starting from 2
    mov   esi,eax       ; esi = 1, as if "1" was handled last
print_loop:
    inc   esi           ; while (++number < n)
    cmp   esi,n
    jae   end_print_loop
    cmp   BYTE PTR prime[esi],1 ; eax/al is modified from "WriteDec"
    jne   print_loop    ; not a prime number, loop
    ; esi is prime number, print it
    mov   eax,esi
    call  WriteDec      ; these two calls must preserve esi
    call  Crlf
    jmp   print_loop

end_print_loop:
    exit

main ENDP
END main

I did test it under linux with NASM, so I had to change the syntax a bit to compile it, and then change the syntax back for MASM/TASM + irvine, so maybe some tiny bug slipped there during conversion, let me know if it doesn't work.