I'm collecting data from a few thousand network devices every few minutes in Python 2.7.8 via package netsnmp. I'm also using fastsnmpy so that I can access the (more efficient) Net-SNMP command snmpbulkwalk.
I'm trying to cut down how much memory my script uses. I'm running three instances of the same script which sleeps for two minutes before re-querying all devices for data we want. When I created the original script in bash they would use less than 500MB when active simultaneously. As I've converted this over to Python, however, each instance hogs 4GB each which indicates (to me) that my data structures need to be managed more efficiently. Even when idle they're consuming a total of 4GB.
My script begins with creating a list where I open a file and append the hostname of our target devices as separate values. These usually contain 80 to 1200 names.
expand = []
f = open(self.deviceList, 'r')
for line in f:
line = line.strip()
expand.append(line)
From there I set up the SNMP sessions and execute the requests
expandsession = SnmpSession ( timeout = 1000000 ,
retries = 1, # I slightly modified the original fastsnmpy
verbose = debug, # to reduce verbose messages and limit
oidlist = var, # the number of attempts to reach devices
targets = expand,
community = 'expand'
)
expandresults = expandsession.multiwalk(mode = 'bulkwalk')
Because of how both SNMP packages behave, the device responses are parsed up into lists and stored into one giant data structure. For example,
for output in expandresults:
print ouput.hostname, output.iid, output.val
#
host1 1 1
host1 2 2
host1 3 3
host2 1 4
host2 2 5
host2 3 6
# Object 'output' itself cannot be printed directly; the value returned from this is obscure
...
I'm having to iterate through each response, combine related data, then output each device's complete response. This is a bit difficult For example,
host1,1,2,3
host2,4,5,6
host3,7,8,9,10,11,12
host4,13,14
host5,15,16,17,18
...
Each device has a varying number of responses. I can't loop through expecting every device having a uniform arbitrary number of values to combine into a string to write out to a CSV.
I believe it is here where I'm consuming a lot of memory but I cannot resolve how to simplify the process while simultaneously removing visited data.
expandarrays = dict()
for output in expandresults:
if output.val is not None:
if output.hostname in expandarrays:
expandarrays[output.hostname] += ',' + output.val
else:
expandarrays[output.hostname] = ',' + output.val
for key in expandarrays:
self.WriteOut(key,expandarrays[key])
Currently I'm creating a new dictionary, checking that the device response is not null, then appending the response value to a string that will be used to write out to the CSV file.
The problem with this is that I'm essentially cloning the existing dictionary, meaning I'm using twice as much system memory. I'd like to remove values that I've visited in expandresults when I move them to expandarrays so that I'm not using so much RAM. Is there an efficient method of doing this? Is there also a better way of reducing the complexity of my code so that it's easier to follow?
Thanks to those who answered. For those in the future that stumble across this thread due to experiencing similar issues: the fastsnmpy package is the culprit behind the large use of system memory. The multiwalk() function creates a thread for each host but does so all at once rather than putting some kind of upper limit. Since each instance of my script would handle up to 1200 devices that meant 1200 threads were instantiated and queued within just a few seconds. Using the bulkwalk() function was slower but still fast enough to suit my needs. The difference between the two was 4GB vs 250MB (of system memory use).
If the device responses are in order and are grouped together by host, then you don't need a dictionary, just three lists:
last_host = None
hosts = [] # the list of hosts
host_responses = [] # the list of responses for each host
responses = []
for output in expandresults:
if output.val is not None:
if output.hostname != last_host: # new host
if last_host: # only append host_responses after a new host
host_responses.append(responses)
hosts.append(output.hostname)
responses = [output.val] # start the new list of responses
last_host = output.hostname
else: # same host, append the response
responses.append(output.val)
host_responses.append(responses)
for host, responses in zip(hosts, host_responses):
self.WriteOut(host, ','.join(responses))