Search code examples
javaandroidimagecameraprocessing

How to make pixel by pixel PImage manipulation faster?


I am making an android app, wherein there is a simple camera UI along with a filter list to manipulate pictures clicked by the camera. I am using the Processing library (https://processing.org/) in Java along with the Ketai library (http://ketai.org/) in the processing IDE.

Following is how my program primarily runs:

--- I have a camera object and an empty image object. The image object has the same resolution as that of the camera.

--- Reading from the camera also returns an image object. Lets call the image returned by the camera as cameraImage and the empty image as dummyImage.

--- At each frame, I read from the camera using the pixel array of the cameraImage, I copy each pixel individually to the dummyImage at the respective coordinate position.

--- It is at this point of copying from the cameraImage to the dummyImage that I manipulate the pixels according to the options selected by the user.

--- I take the red, green and blue values of each pixel while copying it using bit shifting (to gain maximum efficiency) and then manipulate it.

--- After copying to the dummyImage is done, I display the dummyImage to the screen as a normal Image.

Now the thing is the when I display the cameraImage directly, I am easily able to get around 50 to 60 fps. But when I display the dummyImage after copying, I get approximately 1.5 fps.

The following code demonstrates how I am copying from cameraImage to dummyImage. It doesn't have any filters but would take as much time as with any filters.

  cameraImage.read();
  cameraImage.loadPixels();
  dummyImage.loadPixels();
  for (int x = 0; x < cameraImage.width; x++){
    for (int y = 0; y < cameraImage.height; y++){
      int index = x + (y * cameraImage.width);
      color currentPixel = cameraImage.pixels[index];
      
      float r = currentPixel >> 16 & 0xFF;
      float g = currentPixel >> 8 & 0xFF;
      float b = currentPixel & 0xFF;
      
      dummyImage[index] = color(r, g, b)
    }
  }
  dummyImage.updatePixels();
  image(dummyImage);

I would like to know how I can increase the frames upto atleast 20 per second. Please comment for extra information, I would reply as soon as possible.


Solution

  • A few possible optimisations:

    • Replace = color(r, g, b) with = 0xff000000 | ((int) (r) << 16 | (int) (g) << 8 | (int) (b)).

    • You can't rasterize graphics in Processing in parallel but you can read/write to the pixels[] array in parallel (then updatePixels() afterwards), so use thread() to split pixel iteration over multiple CPU threads.

    • Rather than calling image(dummyImage), which involves a second pass over the pixels to copy them into the PApplet, write into the PApplet directly within the loop (remove all references to dummyImage and use pixels[index] = ... instead).

    • As pixel-by-pixel computation is embarrassingly parallel, you could even consider writing a glsl shader (there's lots of Processing examples of this) or use Aparapi, which converts Java code to GPU-ready OpenCL (I've had success using this with Processing).