How to use gevent for a task "read - search - write"?

Question

I am trying deal with a task as following:

1. read a line in a local file
2. send the line to a web service for query
3. write the results to a local file

`

def read(fin):
    for query in fin:
        return query

def search(query):
    # Send the query to something like Google Search
    result = google(query)
    return result

def write(fout, result):
    fout.write(result)

`

I refer to the Gevent tutorial but could not figure out how to put the three functions into gevent.

Answer 1

I have no idea about the point of your example related to gevent. Anyway, it still can be processed through example code below.

import gevent

def read(fin):
    for query in fin:
        return query

def search(query):
    # Send the query to something like Google Search
    result = google(query)
    return result

def write(fout, result):
    fout.write(result)

# read from different files
input_files = ["input1.json", "input2.json" ... ] 

# write result into output file
fout = open("output.json", 'wb')

# get queries from input files
queries = []
for input_file in input_files:
    with open(input_file, 'rb') as fin:
        queries.append(gevent.joinall([gevent.spawn(read, fin)]))

# store results of queries
results = []
for item in [query for query in queries]:
    results.append(gevent.joinall([gevent.spawn(search, item[0].value)]))

# save results into output file
for item in [result for result in results]:
    gevent.joinall([gevent.spawn(write, fout, item[0].value)])

The thing is both read-to-search and search-to-write have dependences, which is not able to use asynchronous way to achieve this. Thus in the example, it just can process each function and save return one by one.