How to make smudge tool algorithm?

Question

I am creating an algorithm for smudge tool but it has to be done pixel by pixel.

The concept of smudge tool is simple

onMouseMove - Copy pixels of old point to new point using a brush template

I am having problem in bitwise operations. The algorithm is not drawing pixels properly.(I am creating this algorithm from scratch so there might be some silly mistakes)

diameter = brush.size;
_bitData = _canvas.bitmapData;
_bitwidth = _bitData.rect.width;//width of canvas

_bitVector = _bitData.getVector();//1d vector of uint

_brushVector = brush.bitmapData.getVector();//1d vector of uint

brushVectorIndex = 0;
for(yIndex = 0; yIndex < diameter; yIndex++)
{
    for(xIndex = 0; xIndex < diameter; xIndex++)
    {
        yCor = yIndex + oldY;
        xCor = xIndex + oldX;

        if(_bitData.rect.contains(xCor,yCor))
        {
            bitVectorIndex_old      = (yCor * _bitWidth)        + xCor;
            bitVectorIndex_new      = ((Y+yIndex) * _bitWidth)  + X+xIndex;

            //Creating alpha map of brush and old mouse point's pixel
            brushPixelAlpha = (_brushVector[brushVectorIndex] & _bitVector[bitVectorIndex_old] & 0xFF000000);

            //Adding colors to the aplha map according to old mouse point's pixel
            brushPixel = brushPixelAlpha | (_bitVector[bitVectorIndex_old] & 0x00FFFFFF);

            //Create alpha map for new pixel
            pixelAlpha = ((brushPixel | _bitVector[bitVectorIndex_new]) & 0xFF000000)

            //Adding color to pixel alpha map using brush's stamp 
            pixel =  pixelAlpha | (brushPixel & 0x00FFFFFF);

            _bitVector[bitVectorIndex_new] = pixel;
        }
        brushVectorIndex++;
    }
}
_bitData.setVector(_bitVector);

If you could suggest how to optimise this code then that will be helpful too because this code will run 10000s time every frame.

Edit: Made a solution which works. The bitwise operations above are very wrong. The code below is not optimised but it works.

private function smudgeIt(brush:uint,oldMouse:uint,newMouse:uint):uint
        {
            var pixel:uint;
            var bA:uint = (brush>>24)&0xff;
            var oA:uint = (oldMouse>>24)&0xff;

            var oldAlpha:uint = oA<bA?oA:bA;

            var rOld:uint,gOld:uint,bOld:uint;
            var rNew:uint,gNew:uint,bNew:uint;

            rOld = (oldMouse >>16) & 0xff;
            gOld = (oldMouse >>8)  & 0xff;
            bOld = (oldMouse       & 0xff);

            rNew = (newMouse >>16) & 0xff;
            gNew = (newMouse >>8)  & 0xff;
            bNew = (newMouse       & 0xff);

            var newAlpha:uint = ((newMouse>>24)&0xff)-oldAlpha;
            newAlpha = newAlpha<0?0:newAlpha;

            rNew = (rNew*newAlpha + rOld*oldAlpha)/255;
            gNew = (gNew*newAlpha + gOld*oldAlpha)/255;
            bNew = (bNew*newAlpha + bOld*oldAlpha)/255;

            var finalAlpha:uint = ((newMouse >> 24) & 0xff)+oldAlpha;
            finalAlpha = finalAlpha>255?255:finalAlpha;

            pixel = finalAlpha<<24 | rNew << 16 | gNew << 8 | bNew;

            return pixel;
        }

Answer 1

There is some confusion with the meaning of alpha channel.
The code you are showing if overwriting the RGB channel of new position with a mixture of old and brush:

RGB_new = RGB_old | RGB_brush;
alpha_new = alpha_new | (alpha_old & alpha_brush);

This is not what you want, you lost all the color information at new position (thus did not mixed it), and at worse, if brush really has RGB information, you'll saturate all the channels until you obtain pure white.
Let's forget about alpha channels for a while, and let us start with plain opaque RGB forms.
The brush as you describe it, operates as a bit mask: if a bit is set to 1, take pixel from old position, else keep pixel from new position. Assuming a 8 bits mask8 = mask1*16rFF), applied on 1-8bits channel of 1 pixel:

R_new = (R_new & (256-mask8)) | (R_old & mask8);

Now assuming a 24 bits mask (8 bit mask repeated 3 times, (mask8<<16)|(mask8<<8)|mask8 or mask8*0x10100 or mask1*16rFFFFFF), this can be applied to the 3 channels simultaneously:

RGB_new = (RGB_new & (~mask24)) | (R_old & mask24);

If you want to complexify the thing with alpha channel (that is some sort of gradual transparency rather than all-or-nothing), then you will have to describe how to mix the old and new bits, but I think that the brush would then be the alpha channel itself.

One classical algorithm (alpha blending) is having R_new = R_new * (1-alpha) + R_old * alpha where alpha is a fraction between 0 and 1. It can be performed in integer arithmetic with alpha between 0 (transparent) and 0xFF (opaque):

R_new = (R_new * (255-alpha) + R_old*alpha) / 255;

With some bit trick, you can multiplex operations on R_B channels and A_G_ channels. You can find some of these alpha blending bit tricks on this site. I did apply some of them to Squeak VM http://bugs.squeak.org/view.php?id=7803, after all BitBlt operations where invented by Dan Ingalls in the Smalltalk context https://en.wikipedia.org/wiki/Bit_blit, so we must give back to caesar.

Here, we did not take into account intrinsic transparency of old and new pixels, we could define more complex operations to take those into account, but what would that mean? If they are transparent, what do we see behind?