multiprocessing in python moviepy

Question

working on writing video clips as video files in parallel simultaneously on moviepy, without having to wait for the process to complete,

I therefore divide my video into 5-second clips

     n=0
     p = 5
     clip = mp.VideoFileClip(videofile).subclip(n, n+p)

I then add subtitles to the video,

    x = 0
    text2 = 'hello'+str(x)
    text[x] = TextClip(text2, font='Amiri-regular',color='white',fontsize=24).set_duration(p).set_start(0)

I then do this again and again for the first five clips, and on the fifth one, I write the clip as a video file

I want to continue processing the rest of the video as the writing continues in the background, so I use multiprocessing, after editing the code suggested by @Roland Smith, I use:

if float.is_integer(float(x)/5.0) == True and x != 0:
    text2 = concatenate(text.values())
    textd = text2.on_color(size=(clip.w ,text2.h),color=(0,0,0), col_opacity=0.6).set_pos('bottom')
    video3[n] = CompositeVideoClip([VideoFileClip(videofile).subclip(n,5+n), textd])



def audioclip(data):
    outname = str(data)[-10:].strip('>') + '.mp4'
    data.write_videofile(outname,fps=24, codec='libx264')       
    return outname

names = video3.values()
h = multiprocessing.Pool()
audiofiles = h.map(audioclip, names)
gc.collect()
n = n+p
x = x+1

I had imported

    from moviepy.editor import *
    import moviepy.editor as mp
    import os 
    import multiprocessing
    from multiprocessing import pool

however, I get this error:

     Traceback (most recent call last):
      File "p2 (copy).py", line 128, in <module>
audiofiles = h.map(audioclip, names)
      File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
return self.map_async(func, iterable, chunksize).get()
      File "/usr/lib/python2.7/multiprocessing/pool.py", line 558, in get
raise self._value
    cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

please help

Answer 1

Your question isn't completely clear, but I thought I'd show you how to do the audio extraction in parallel.

The first code fragment you give chould be reworked into a function.

import moviepy.editor as mp
from multiprocessing import pool

def audioclip(input):
    clip = mp.VideoFileClip(input).subclip(0,20)
    outname = input[:-3] + 'mp3'
    clip.audio.write_audiofile(outname)
    return outname

Furthermore, you'll need a list of input filenames. Then using the map method of the multiprocessing.Pool object, you apply the abovementioned function to all videos.

# You should probably take the names from the command line...
names = ['foo.mp4', 'bar.mp4', 'spam.mp4', 'eggs.mp4']

p = multiprocessing.Pool()
audiofiles = p.map(audioclip, names)

This will extract audio from the clips in parallel, using as many worker processes as your CPU has cores by default.

Edit: Note that if you use map, the items of the iterable have to be pickled and sent to the worker process. To prevent this from using a lot of resources it is better to e.g. send the name of a big file to the worker process (so it can read the file itself) rather than the contents of that file. Otherwise this would fast become a bottleneck in the process. This might look wasteful if multiple workers all have to read the same file, but the filesystem caching that all modern operating systems do should mitigate that.