program to read a file contents

Question

I have a snort log file named "logs" and want to extract IP addresses from it and store them to another file named "blacklist". it can extract unique IP Addresses but if I run the program again, it adds the previous IPs as well. I want the program to first check whether IP is already in blacklist file? if so, just ignore it otherwise add unique IPs from logs file to blacklist. code:

#!/usr/bin/python
import re
mylist1 = []
mylist2 = []
mylist3 = []
mylist4 = []
logfile = open('/var/log/snort/logs', 'r')
blklist = open('blacklist', 'ab+')

for line in open ('blacklist', 'r').readlines():
  mylist4.append(line)

for l in logfile.readlines():
  l = l.rstrip()
  ip = re.findall(r'[0-9]+(?:\.[0-9]+){3}',l)
  if ip is not None and ip not in mylist1:
    mylist1.append(ip)
for ip in mylist1:
  addr = ",".join(ip)
  if ',' in addr:
    a = addr.split(',')
    for ip in a:
        addr = "".join(ip)
        if addr is not '':
            mylist2.append(addr)
        else:
            mylist3.append(addr)
for x in blklist:
  mylist2.append(x.strip())
for x in mylist2:
  if x not in mylist3 and x not in mylist4:
    blklist.write(x+'\n')
    mylist3.append(x)

Logs file is:

12/16-10:34:27.070967 [**] [1:10000001:1] snort alert [1:0000001] [**][classification ID: 0] [Priority ID: 0] {ICMP} 192.168.40.19 -> 192.168.50.29

12/16-10:34:27.070967 [**] [1:10000001:1] snort alert [1:0000001] [**][classification ID: 0] [Priority ID: 0] {ICMP} 192.168.50.29 -> 192.168.30.20

Output of blacklist file after first program run:

192.168.30.20
192.168.50.29
192.168.40.19

Output of blacklist file after second program run:

192.168.30.20
192.168.50.29
192.168.40.19
192.168.30.20
192.168.50.29
192.168.40.19

any help please?

Answer 1

You can read everything in from your blacklist file and log into lists. Join those list and then ouput a set back to the blacklist file (sets are unique values) since the read empties the file your will have a unique list of all new and old IPs. If the order matters (doubt it does) then a set will cause issues. Let me know and I can revamp the below.

if __name__ == '__main__':
    import re
    blacklist = list(open("blacklist", 'r').read().split('\n'))
    logfile = list(open("/var/log/snort/logs", 'r').read().split('\n'))

    newentry = []
    for entry in logfile:
        ips = re.findall( r'[0-9]+(?:\.[0-9]+){3}', entry)
        for ip in ips:
            newentry.append(ip)

    newblacklist = blacklist + newentry

    with open("blacklist", 'w+') as f:
        f.write('\n' .join(set(newblacklist)))
        f.close()