I am making an android app, wherein there is a simple camera UI along with a filter list to manipulate pictures clicked by the camera. I am using the Processing library (https://processing.org/) in Java along with the Ketai library (http://ketai.org/) in the processing IDE.
Following is how my program primarily runs:
--- I have a camera object and an empty image object. The image object has the same resolution as that of the camera.
--- Reading from the camera also returns an image object. Lets call the image returned by the camera as cameraImage and the empty image as dummyImage.
--- At each frame, I read from the camera using the pixel array of the cameraImage, I copy each pixel individually to the dummyImage at the respective coordinate position.
--- It is at this point of copying from the cameraImage to the dummyImage that I manipulate the pixels according to the options selected by the user.
--- I take the red, green and blue values of each pixel while copying it using bit shifting (to gain maximum efficiency) and then manipulate it.
--- After copying to the dummyImage is done, I display the dummyImage to the screen as a normal Image.
Now the thing is the when I display the cameraImage directly, I am easily able to get around 50 to 60 fps. But when I display the dummyImage after copying, I get approximately 1.5 fps.
The following code demonstrates how I am copying from cameraImage to dummyImage. It doesn't have any filters but would take as much time as with any filters.
cameraImage.read();
cameraImage.loadPixels();
dummyImage.loadPixels();
for (int x = 0; x < cameraImage.width; x++){
for (int y = 0; y < cameraImage.height; y++){
int index = x + (y * cameraImage.width);
color currentPixel = cameraImage.pixels[index];
float r = currentPixel >> 16 & 0xFF;
float g = currentPixel >> 8 & 0xFF;
float b = currentPixel & 0xFF;
dummyImage[index] = color(r, g, b)
}
}
dummyImage.updatePixels();
image(dummyImage);
I would like to know how I can increase the frames upto atleast 20 per second. Please comment for extra information, I would reply as soon as possible.
A few possible optimisations:
Replace = color(r, g, b)
with = 0xff000000 | ((int) (r) << 16 | (int) (g) << 8 | (int) (b))
.
You can't rasterize graphics in Processing in parallel but you can read/write to the pixels[]
array in parallel (then updatePixels()
afterwards), so use thread()
to split pixel iteration over multiple CPU threads.
Rather than calling image(dummyImage)
, which involves a second pass over the pixels to copy them into the PApplet
, write into the PApplet directly within the loop (remove all references to dummyImage
and use pixels[index] = ...
instead).
As pixel-by-pixel computation is embarrassingly parallel, you could even consider writing a glsl shader (there's lots of Processing examples of this) or use Aparapi, which converts Java code to GPU-ready OpenCL (I've had success using this with Processing).