Search code examples
pythonmultithreadingqueuepython-multithreadingfifo

Python script with multi-threading showing wrong sequence from that of the sequence of list passed


I am using fetch_parallel method to download fits files and want them to be in sequence so that I can plot them using matplotlib but each time I am running the script there is a separate sequence of returned objects meaning if i pass

urls=['foo1','foo2','foo3']

# the return sequence should be:
return_objects = ['obj1','obj2','obj3']

# but i am getting random sequence like:
return_objects = ['obj3','obj1','obj2']
  1. Is it because of the thread joining early and latter due to download progress?
  2. How do I return in the same sequence as passsed?
def fetch_parallel(urls):
  result = queue.Queue()
  threads = [threading.Thread(target=read_url, args = (url,result)) for url in urls]
  for t in threads:
      t.start()
  for t in threads:
      t.join()
  return result

update : read_url method


def read_url(url, queue):
  data = urllib.request.urlopen(url, timeout=300)
  fget = fts.getdata(data, header=True)
  print('Fetched %s from %s' % (len(fget), url))

  queue.put(fget)

Solution

  • Try this changes, result should be a "shared" list and write result on the same order as url in urls

    result = list(xrange(len(urls)))
    threads = [threading.Thread(target=read_url, args = (urls[i_url], i_url, result)) for i_url in xrange(len(urls))]
    
    
    def read_url(url, i_url, queue):
        ...
    
        #queue.put(fget)
        queue[i_url] = fget