Search code examples
pythonmemorystructure

Can I use pagefile or swap area for massive data structure in python?

I have question about use massive data structure (list, dict, tuple, etc.)

If my computer have memory size 2GB, Can I use 3GB or more data structure in memory?

This question occured when use massive matrix data.

For solve this question, I think use pagefile or swap area during processing.

but I can not know 'how', and I want find more other methods.

I waiting for reply. Thank you for read this question article.

Solution

  • Python uses all the resources provided by your OS (which can be quite problematic for other services). If you want to use a swap file in order to increase the memory for your application, you have to do this on the OS layer and not in your software. There are some ways to get hold of this sort of problem:

    • Write your code more efficient, for example using sparse-matrices or tile your matrix in pieces, using only the pieces necessary for the moment
    • Enable additional hardware resources, such as swap or RAM.
    • Compress the information with suitable algorithms

    To give a more precise answer, you have to provide more detailed information about your problem.