pythonimage-processingpython-imaging-library

Python - Quick batch modification of PNGs


I wrote a python script with combines images in unique ways for an OpenGL shader. The problem is that I have a large number of very large maps and it takes a long time to process. Is there a way to write this in a quicker fashion?

import numpy as np

map_data = {}
image_data = {}
for map_postfix in names:
file_name = inputRoot + '-' + map_postfix + resolution + '.png'
print 'Loading ' + file_name
image_data[map_postfix] = Image.open(file_name, 'r')
map_data[map_postfix] = image_data[map_postfix].load()


color = mapData['ColorOnly']
ambient = mapData['AmbientLight']
shine = mapData['Shininess']

width = imageData['ColorOnly'].size[0]
height = imageData['ColorOnly'].size[1]

arr = np.zeros((height, width, 4), dtype=int)

for i in range(width):
    for j in range(height):
        ambient_mod = ambient[i,j][0] / 255.0
        arr[j, i, :] = [color[i,j][0] * ambient_mod , color[i,j][1] * ambient_mod , color[i,j][2] * ambient_mod , shine[i,j][0]]

print 'Converting Color Map to image'
return Image.fromarray(arr.astype(np.uint8))

This is just a sample of a large number of batch processes so I am more interested in if there is a faster way to iterate and modify an image file. Almost all the time is being spent on the nested loop vs loading and saving.


Solution

  • Vectorised-code example -- test effect on yours in timeit or zmq.Stopwatch()

    Reported to have 22.14 seconds >> 0.1624 seconds speedup!

    While your code seems to loop just over RGBA[x,y], let me show a "vectorised"-syntax of a code, that benefits from numpy matrix-manipulation utilities ( forget the RGB/YUV manipulation ( originally based on OpenCV rather than PIL ), but re-use the vectorised-syntax approach to avoid for-loops and adapt it to work efficiently for your calculus. Wrong order of operations may more than double yours processing time.

    And use a test / optimise / re-test loop for speeding up.

    For testing, use standard python timeit if [msec] resolution is enough.

    Go rather for zmq.StopWatch() if you need going into [usec] resolution.

    # Vectorised-code example, to see the syntax & principles
    #                          do not mind another order of RGB->BRG layers
    #                          it has been OpenCV traditional convention
    #                          it has no other meaning in this demo of VECTORISED code
    
    def get_YUV_U_Cb_Rec709_BRG_frame( brgFRAME ):  # For the Rec. 709 primaries used in gamma-corrected sRGB, fast, VECTORISED MUL/ADD CODE
        out =  numpy.zeros(            brgFRAME.shape[0:2] )
        out -= 0.09991 / 255 *         brgFRAME[:,:,1]  # // Red
        out -= 0.33601 / 255 *         brgFRAME[:,:,2]  # // Green
        out += 0.436   / 255 *         brgFRAME[:,:,0]  # // Blue
        return out
    # normalise to <0.0 - 1.0> before vectorised MUL/ADD, saves [usec] ...
    # on 480x640 [px] faster goes about 2.2 [msec] instead of 5.4 [msec]
    

    In your case, using dtype = numpy.int, guess it shall be faster to MUL first by ambient[:,:,0] and finally DIV to normalisearr[:,:,:3] /= 255

    # test if this goes even faster once saving the vectorised overhead on matrix DIV
    arr[:,:,0] = color[:,:,0] * ambient[:,:,0] / 255  # MUL remains INT, shall precede DIV
    arr[:,:,1] = color[:,:,1] * ambient[:,:,0] / 255  # 
    arr[:,:,2] = color[:,:,2] * ambient[:,:,0] / 255  # 
    arr[:,:,3] = shine[:,:,0]                         # STO alpha
    

    So how it may look in your algo?

    One need not have Peter Jackson's impressive budget and time once planned, spanned and executed immense number-crunching over 3 years in a New Zealand hangar, overcrowded by a herd of SGI workstations, as he was producing "The Lord of The Rings" fully-digital mastering assembly-line, right by the frame-by-frame pixel manipulation, to realise that miliseconds and microseconds and even nanoseconds in the mass-production pipe-line simply do matter.

    So, take a deep breath and test and re-test so as to optimise your real-world imagery processing performance to levels that your project needs.

    Hope this may help you on this:

    # OPTIONAL for performance testing -------------# ||||||||||||||||||||||||||||||||
    from zmq import Stopwatch                       # _MICROSECOND_ timer
    #                                               # timer-resolution step ~ 21 nsec
    #                                               # Yes, NANOSECOND-s
    # OPTIONAL for performance testing -------------# ||||||||||||||||||||||||||||||||
    arr        = np.zeros( ( height, width, 4 ), dtype = int )
    aStopWatch = zmq.Stopwatch()                    # ||||||||||||||||||||||||||||||||
    # /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\# <<< your original code segment          
    #  aStopWatch.start()                           # |||||||||||||__.start
    #  for i in range(     width  ):
    #      for j in range( height ):
    #          ambient_mod  = ambient[i,j][0] / 255.0
    #          arr[j, i, :] = [ color[i,j][0] * ambient_mod, \
    #                           color[i,j][1] * ambient_mod, \
    #                           color[i,j][2] * ambient_mod, \
    #                           shine[i,j][0]                \
    #                           ]
    #  usec_for = aStopWatch.stop()                 # |||||||||||||__.stop
    #  print 'Converting Color Map to image'
    #  print '           FOR processing took ', usec_for, ' [usec]'
    # /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\# <<< proposed alternative
    aStopWatch.start()                              # |||||||||||||__.start
    # reduced numpy broadcasting one dimension less # ref. comments below
    arr[:,:, 0]  = color[:,:,0] * ambient[:,:,0]    # MUL ambient[0]  * [{R}]
    arr[:,:, 1]  = color[:,:,1] * ambient[:,:,0]    # MUL ambient[0]  * [{G}]
    arr[:,:, 2]  = color[:,:,2] * ambient[:,:,0]    # MUL ambient[0]  * [{B}]
    arr[:,:,:3] /= 255                              # DIV 255 to normalise
    arr[:,:, 3]  = shine[:,:,0]                     # STO shine[  0] in [3]
    usec_Vector  = aStopWatch.stop()                # |||||||||||||__.stop
    print 'Converting Color Map to image'
    print '           Vectorised processing took ', usec_Vector, ' [usec]'
    return Image.fromarray( arr.astype( np.uint8 ) )