Special case when += for string concatenation is more efficient than =

Question

I have this code using python 3.11:

import timeit

code_1 = """
initial_string = ''
for i in range(10000):
    initial_string = initial_string + 'x' + 'y'
"""

code_2 = """
initial_string = ''
for i in range(10000):
    initial_string += 'x' + 'y'
"""

time_1 = timeit.timeit(code_1, number=100)
time_2 = timeit.timeit(code_2, number=100)

print(time_1)
# 0.5770808999950532
print(time_2)
# 0.08363639999879524

Why += is more efficient in this case? As far as I know, there is the same number of concatenation, and the order of execution doesn't change the result.

Since strings are immutable, it's not because of inplace shinanigans, and the only thing I found about string concat is about .join efficiency, but I don't want the most efficient, just understand why += seems more efficient than =.

With this code, performances between forms almost equals:

import timeit

code_1 = """
initial_string = ''
for i in range(10000):
    initial_string = initial_string + 'x'
"""

code_2 = """
initial_string = ''
for i in range(10000):
    initial_string += 'x'
"""

time_1 = timeit.timeit(code_1, number=100)
time_2 = timeit.timeit(code_2, number=100)

print(time_1)
# 0.07953230000566691
print(time_2)
# 0.08027460001176223

---

I noticed a difference using different Python version ('x' + 'y' form):

Python 3.7 to 3.9:

print(time_1)
# ~0.6
print(time_2)
# ~0.3

Python 3.10:

print(time_1)
# ~1.7
print(time_2)
# ~0.8

Python 3.11 for comparison:

print(time_1)
# ~0.6
print(time_2)
# ~0.1

---

Similar but not answering the question: How is the s=s+c string concat optimization decided?

If s is a string, then s = s + 'c' might modify the string in place, while t = s + 'c' can't. But how does the operation s + 'c' know which scenario it's in?

In a nutshell: Optimization occur when s = s + 'c', not when t = s + 'c' because python need to keep a ref to the first string and can't concatenate in-place.

Here, we are always assigning using simple assignment or augmented assignment to the original string, so in-place concatenation should apply in both cases.

Answer 1

For a while now, CPython has had an optimization that tries to perform string concatenation in place where possible. The details vary between Python versions, sometimes a lot - for example, it doesn't work for globals on Python 3.11, and it used to be specific to bytestrings on Python 2, but it's specific to Unicode strings on Python 3.

On Python 3.10, the optimization starts in unicode_concatenate, and it eventually hits a PyObject_Realloc inside resize_compact or resize_inplace, attempting to resize the left-hand operand in place.

One fairly consistent thing about the optimization across Python versions is that it only works if the left-hand side of the concatenation has no other references, or if the only reference is a variable that the result of the concatenation will be assigned to. In your slow case:

initial_string = initial_string + 'x' + 'y'

the LHS of initial_string + 'x' is initial_string, and you're not going to assign the result back to initial_string - you're going to concatenate 'y' to the result first. Thus, the optimization can't kick in for initial_string + 'x'. (It kicks in for the + 'y' part, though.)

For your other cases, the optimization works. For example, in

initial_string += 'x' + 'y'

instead of concatenating initial_string and 'x' and then appending 'y', you concatenate 'x' and 'y' and then concatenate initial_string and the result. The changed order of operations means that you're assigning the result of the initial_string concatenation back to initial_string, so the optimization applies. (Also the 'x' + 'y' gets constant-folded away, which helps a little but isn't the primary cause of the performance difference.)