Search code examples
pythongoogle-analytics-api

Is there a better way than empty my list after each loop ?


I'm new to Python. I just wrote a new script to export some data from multiple Google analytics profiles. It's working perfectly fine, however I'm sure that it's written very badly.

I don't really know where to start to improve it so here is my a first question.

I'm looping through a list of profile id. For each profil ID, I've several actions, where I'm using the append method. So I'm building some list step-by-step, but at the end I need to reset these lists. So I have something like this at the beginning and at the end of my code :

fullurllist = []
urllist = []
share = []
sharelist = []
sharelist1 = []
end_list = []

I guess I should avoid this. Do I need to change all the logic of my code. Is there something else I can do to improve this aspect.

Here is the code :

  # Loop through the profiles_list and get the best pages for each profile 
  for profile in profiles_list:
    response = service.data().ga().get(
      ids='ga:' + profile,
      start_date='1daysAgo',
      end_date='today',
      metrics='ga:sessions',
      dimensions='ga:pagePath',
      sort='-ga:sessions',
      filters='ga:sessions>400').execute()

    # Catch response.
    rawdata = response.get('rows', [])

    # Flatten response (which is a list of lists)
    for row in rawdata:
      urllist.append(row[0])

    # Building a list of full url (Hostname + Page path)
    fullurllist = [urljoin(base, h) for h in urllist]

    # Scraping some data from the url list
    for url in fullurllist:  

      try:
          page = urllib2.urlopen(url)
      except urllib2.HTTPError as e:
              if e.getcode() == 404: # eheck the return code
                  continue
      soup = BeautifulSoup(page, 'html.parser')

      # Take out the <div> of name and get its value
      name_box = soup.find(attrs={'class': 'nb-shares'})
      if name_box is None:
        continue
      share = name_box.text.strip() # strip() is used to remove starting and trailing

      # save the data in tuple
      sharelist.append(url)
      sharelist1.append(share)

      # Format the data scraped
      end_list = [int(1000*float(x.replace('k', ''))) if 'k' in x else int(x) for x in sharelist1]

    #export in csv
    csv_out = open(response.get('profileInfo').get('profileName') + '.csv', 'wb')
    mywriter = csv.writer(csv_out)
    for row in zip(sharelist, end_list):
      mywriter.writerow([row])
    csv_out.close()

    #reset list
    fullurllist = []
    urllist = []
    share = []
    sharelist = []
    sharelist1 = []
    end_list = []

Thanks a lot !


Solution

  • More proper would be to do the declarations ( fullurlllist = [] ) at the top of the for loop and not outside. They should live only inside the loop anyways