Search code examples
pandaskaggle

Your notebook tried to allocate more memory than is available. It has restarted

I was getting started with TalkingData AdTracking, my first entry to Kaggle competitions. The first line was pd.read_csv() and I got this error

Your notebook tried to allocate more memory than is available. It has restarted

I thought my code was run in the cloud and that I don't have to worry about the memory requirements. Can someone help me with that?

Solution

  • Yes, kaggle has a memory limit: Its 8GB per kernel, or 20 Min running. It takes a lot of server juice to host such a thing.

    There are various solutions for this problem, notably loading and processing the dataset in chunks.

    There is this as well. But I have no experience with it.

    And of course you can use another cloud platform such as AWS, GCP, Oracle, etc..