How can tornado use pipe to async send data to client?

Question

With the tornado I use multiprocessing to create a child process to do the compressing work. And the child process directly send data to main process via os.pipe

Right, now the problem is self.stream.read_until_close block the call, and it does not tar_worker close the other side of the pipe.

My code list below:

def tar_worker(fd, path):
    '''Tar worker to do the compressing work, directly write data to pipe.'''
    fp = os.fdopen(fd, 'w')
    tar = tarfile.open(mode='w|', fileobj=fp, dereference=True)
    tar.add(path)
    tar.close()
    fp.close()


class getTarFileHandler(tornado.web.RequestHandler):

    @tornado.web.asynchronous
    @gen.coroutine
    def get(self):
        recv, sender = os.pipe()
        self.p = multiprocessing.Process(target=tar_worker, args=(sender, '/boot/'))
        self.p.start()
        self.fd = recv

        # Create PipeIOStream
        self.stream = tornado.iostream.PipeIOStream(self.fd)
        self.stream.read_until_close(callback=self.f, streaming_callback=self.send_data)


    def f(self, s):
        print 'done send work'
        self.finish() #close connection

    def send_data(self, chunk):
        self.write(chunk)
        self.flush()

Answer 1

File handles can be shared across multiple processes; in this case both the parent and child processes have file descriptors pointing to both ends of the same underlying pipe. Both the parent and child process must close sender for it to be truly closed. After starting the multiprocessing.Process, the parent process must do os.close(sender) to close its copy of the writing end of the pipe. This will not prevent the child process from writing to its copy; once both parent and child have closed their copies of the writing end of the pipe, the reading end of the pipe will report EOF and the IOStream read methods will raise StreamClosedError.