removing list elements starting with octothorpe in python omits some elements

Question

This is the text file (pathProtocol.txt):

# 2018-09-30
# incubator at 33C
# sample compartment at 32C
# hold sample in for 30sec before
# measurement

samp-2
reps-3

#Temp   Humid   Press
# 24    42  980

# background
MilliQ_MilliQ_0 000-005

Q_Prp_62mM  006-011
Q_Ah6_62mM  012-017
Q_Eth_62mM  018-023
Q_AcA_62mM  024-029
Q_Imd_62mM  030-035

# background
MilliQ_MilliQ_0 036-041
# 24    43  977

that I'm parsing and from which I'm trying to remove blank and lines starting with a #. Using this answer my code starts with:

# PROTOCOL PARSING
# READ FILE & EXCLUDE BLANKS
with open(os.path.join(*pathProtocol)) as f:
    content = (line.rstrip() for line in f)
    # Non-blank lines in a list
    content = list(line for line in content if line)
#
print(content)
print(type(content))
print(len(content))
print('')

yielding the desired output:

['# 2018-09-30', '# incubator at 33C', '# sample compartment at 32C', '# hold sample in for 30sec before', '# measurement', 'samp-2', 'reps-3', '#Temp   Humid\tPress', '# 24\t42\t980', '# background', 'MilliQ_MilliQ_0\t000-005', 'Q_Prp_62mM\t006-011', 'Q_Ah6_62mM\t012-017', 'Q_Eth_62mM\t018-023', 'Q_AcA_62mM\t024-029', 'Q_Imd_62mM\t030-035', '# background', 'MilliQ_MilliQ_0\t036-041', '# 24\t43\t977']
<class 'list'>
19

The interesting part starts when I try to remove lines starting with octothrope from the above created list (this answer):

# DELETE COMMENTS
for i, line in enumerate(content):
    print(str(i), line, '\tfirst char:', line[0])
    if line.startswith('#'):
        content.remove(line)
    #
#
print(content)

I get the following output:

0 # 2018-09-30  first char: #
1 # sample compartment at 32C   first char: #
2 # measurement     first char: #
3 reps-3    first char: r
4 #Temp   Humid Press   first char: #
5 # background  first char: #
6 Q_Prp_62mM    006-011     first char: Q
7 Q_Ah6_62mM    012-017     first char: Q
8 Q_Eth_62mM    018-023     first char: Q
9 Q_AcA_62mM    024-029     first char: Q
10 Q_Imd_62mM   030-035     first char: Q
11 # background     first char: #
12 # 24 43  977     first char: #
['# incubator at 33C', '# hold sample in for 30sec before', 'samp-2', 'reps-3', '# 24\t42\t980', 'MilliQ_MilliQ_0\t000-005', 'Q_Prp_62mM\t006-011', 'Q_Ah6_62mM\t012-017', 'Q_Eth_62mM\t018-023', 'Q_AcA_62mM\t024-029', 'Q_Imd_62mM\t030-035', 'MilliQ_MilliQ_0\t036-041']

that omits some items from the initial list but shows them in the subsequent list output. I just can't wrap my head around what's going on in line-by-line output (as a python beginner). What is the mistake I'm making ? How can I correctly remove lines starting with # ?

Answer 1

You're changing content while you're iterating over it, which generally doesn't work. Instead, iterate over a copy of content. That is, change enumerate(content) to enumerate(content[:]).