I'm new to Python. I just wrote a new script to export some data from multiple Google analytics profiles. It's working perfectly fine, however I'm sure that it's written very badly.
I don't really know where to start to improve it so here is my a first question.
I'm looping through a list of profile id. For each profil ID, I've several actions, where I'm using the append
method. So I'm building some list step-by-step, but at the end I need to reset these lists. So I have something like this at the beginning and at the end of my code :
fullurllist = []
urllist = []
share = []
sharelist = []
sharelist1 = []
end_list = []
I guess I should avoid this. Do I need to change all the logic of my code. Is there something else I can do to improve this aspect.
Here is the code :
# Loop through the profiles_list and get the best pages for each profile
for profile in profiles_list:
response = service.data().ga().get(
ids='ga:' + profile,
start_date='1daysAgo',
end_date='today',
metrics='ga:sessions',
dimensions='ga:pagePath',
sort='-ga:sessions',
filters='ga:sessions>400').execute()
# Catch response.
rawdata = response.get('rows', [])
# Flatten response (which is a list of lists)
for row in rawdata:
urllist.append(row[0])
# Building a list of full url (Hostname + Page path)
fullurllist = [urljoin(base, h) for h in urllist]
# Scraping some data from the url list
for url in fullurllist:
try:
page = urllib2.urlopen(url)
except urllib2.HTTPError as e:
if e.getcode() == 404: # eheck the return code
continue
soup = BeautifulSoup(page, 'html.parser')
# Take out the <div> of name and get its value
name_box = soup.find(attrs={'class': 'nb-shares'})
if name_box is None:
continue
share = name_box.text.strip() # strip() is used to remove starting and trailing
# save the data in tuple
sharelist.append(url)
sharelist1.append(share)
# Format the data scraped
end_list = [int(1000*float(x.replace('k', ''))) if 'k' in x else int(x) for x in sharelist1]
#export in csv
csv_out = open(response.get('profileInfo').get('profileName') + '.csv', 'wb')
mywriter = csv.writer(csv_out)
for row in zip(sharelist, end_list):
mywriter.writerow([row])
csv_out.close()
#reset list
fullurllist = []
urllist = []
share = []
sharelist = []
sharelist1 = []
end_list = []
Thanks a lot !
More proper would be to do the declarations ( fullurlllist = []
) at the top of the for loop and not outside.
They should live only inside the loop anyways