Is python's queue.Queue.put() method asynchronous?

Question

If I run a thread with the following function as the worker,

q = queue.Queue()

def worker():
    while True:
        t = {}
        for i in range(3):
            t['a'] = i
            q.put(t)

the queue is populated with dictionaries that are all the same, i.e., {'a': 2} instead of the sequence {'a': 0}, {'a': 1}, {'a': 2}. I assume this is because the put() method runs after the for loop has finished and the last value of i was 2. Am I interpreting that right?

Now, if I move the instantiation of the dictionary inside the for loop,

def worker():
    while True:
        for i in range(3):
            t = {'a': i}
            q.put(t)

the queue is populated with the desired sequence. My interpretation is that in the first instance, I create a dictionary object in memory, then begin a for loop and reassign its value 3 times but the put() calls happen after the loop has finished. In the second instance, I create a new dictionary object every iteration of the for loop and so when the put() calls occur after the loop, they access 3 distinct instances of the dictionary with their own key-value pairs.

Can anyone shed some light on what's happening behind the curtain here?

Answer 1

Am I interpreting that right?

You observe such behavior because you’re modifying the same object all the time

Lets put aside queues / threads and run a simplified equivalent of your code with some prints to understand what’s happening

t = {}
l = []
for i in range(3):
    t['a'] = i
    l.append(t)
print(l)
t['a'] = 20
print(l)
print(map(id, l))

[{'a': 2}, {'a': 2}, {'a': 2}]
[{'a': 20}, {'a': 20}, {'a': 20}]
# they are all the same!
[4474861840, 4474861840, 4474861840]

So it has nothing to do we threads/queues - you’re just adding the same object 3 times.

Now, if I move the instantiation of the dictionary inside the for loop

In this case you create a new object every time like in the following code:

l = []
for i in range(3):
    t = {}
    t['a'] = i
    l.append(t)
print(l)
t['a'] = 20
print(l)
print(map(id, l))

[{'a': 0}, {'a': 1}, {'a': 2}]
[{'a': 0}, {'a': 1}, {'a': 20}]
# they are all different!
[4533475600, 4533502592, 4533502872]

So no magic here

back to your question

This is what could be of interest for you: “Is python’s queue.Queue.put() thread safe?” meaning that the global variable q could be accessed by multiple concurrent threads safely. The answer is yes - it is thread safe

The Queue module implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be exchanged safely between multiple threads. The Queue class in this module implements all the required locking semantics