Search code examples
pythongoogle-analyticsgoogle-analytics-api

Google Analytics - Extract User Activity Historical and Daily Data

I've been trying to extract and loop through the User Activity API to extract historical data and create a daily feed for each GA profile, but run into the "10,000 requests per view (profile) per day" quota limit as I have ~50,000 users per day.

My current script loops through each CLIENT_ID one by one and extracts the user_activity as shown below and following this documentation: https://developers.google.com/analytics/devguides/reporting/core/v4/rest/v4/userActivity/search

## List of CLIENT_IDs extracted from batch.get() reports API
CLIENT_ID = ['x','y','z']
START_DATE = '2020-01-01'
END_DATE = '2022-01-01'

    for client_id in CLIENT_ID:

        data = analytics.userActivity().search(
            body={
                "viewId": VIEW_ID,
                "pageSize": 1000000000,
                "user": {
                    "type": "CLIENT_ID",
                    "userId": str(client_id),
                },
                'dateRange': {"startDate": str(START_DATE),
                              "endDate": str(END_DATE)
                              }
                             ,
            }
        ).execute()
        print(data)

Is there a better way of extracting this historical data without running into the API quota limit for each profile and pinging multiple users at the same time? I've also tried to create more than one profile in GCP to by-pass the quota limit, but with no success.

In the front-end of GA3, there's no way to extract this raw data so any help would be much appreciated.

Many thanks!

Solution

  • There are three types of quotas for the Google analytics apis.

    • User based quotas
    • View based quotas
    • Project based quotas.

    View Based Quotas

    View based Quotas are based upon each view that you are accessing. They can not be extended.

    See: limits-quotas

    enter image description here

    user based quotas

    User based quotas are based upon the user who is accessing the data, either by the authorization of the user whos account you are using to access the data, or by the ip address of the system running the code. These quotas are often used for flood protection to ensure that your system is not running to fast.

    user based quotas can not be extended.

    Project based quotas.

    Project based quotas are the quota that define your project as a whole. The default project based quota is 50k and this can be extended.

    enter image description here

    So to answer your question. The 10,000 requests per view (profile) per day quota is a view based quota that can not be extended. Your best option would be to make fewer requests and only request the data you need. you can also do it over a number of days.

    If you are requesting it for more then 5 views and there for hitting the project based quota you can apply for an extension.

    There is also an option of exporting your data to BigQuery which removes the limits but its not free from what I remember see: Set up BigQuery Export It is also not a python solution.