I am looking for an efficient algorithm that will look at a four channel image (RGBA), find an object (a group of pixels with Alpha >0 surrounded by pixels with alpha = 0) and then attempt to antialias the edges of the object. Suggestions?
I am guessing that you're trying to describe a situation where the object's image with alpha has been pasted onto an alphaless background, replacing pixels wherever it had alpha > 0, rather than blending properly. In the resulting image, every pixel with alpha=0 is background, every pixel with alpha > 0 is from the object. There's been no blending, just over pasting.
So - an algorithm for guessing what color the pixels where alpha is now non zero were before, would solve the problem. If you had that, you could do the blending properly, at every place where 0% < alpha < 100% to get a proper anti-aliased edge.
One way to guess the missing pixel values is to make each unknown pixel the average of the pixels around it, and keep repeating this until there is little change. This is pretty much the algorithm used for repairing scratches in old photographic images using pixels nearby. You asked for an efficient method. A multigrid method will get you there a little faster when there are larger unknown pixel areas.
Once you have a plausible unknown background, blend back in the pixels with alpha > 0, this time properly taking into account their alpha.