Kedro - Memory management
I am working on a Kedro 0.17.2 project that is running on out-of-memory issues and I'm trying to reduce the memory footprint.
I'm doing the profiling by using mprof from the memory-profiler library and I noticed that there is always a child process and memory seems to duplicate in the main process after the first computation in the node that is running. Is it possible that Kedro is duplicating the dataframes in memory? And, if so, is there a way to avoid this?
Notes:
- I'm using the
SequentialRunner - I'm not using the
is_asynccli option - I'm not using either multithreading or multiprocessing in the node execution
It turns out this issue is caused by a possible bug in the memory-profiler library that is used in the kedro.extras.decorators.memory_profiler.mem_profile decorator.
The kedro decorator makes use of the memory_usage function in the memory-profiler module. It is used to sample the total memory being used by the running function from within the python process.
There is an open issue about this problem but with no solution yet. https://github.com/pythonprofilers/memory_profiler/issues/332
For the moment I have just removed the decorator.
- Is it possible to call setter as a method?
- Accessing a Python traceback from the C API
- Scraping hierarchical website in a specific category
- How to offset a marker in matplotlib?
- How to access current user in Django class based view
- Is there a way to define list(obj) method on a user defined class in python?
- Deserializing Json list into class objects not able to extract attrributes
- What is the pythonic way to detect the last element in a 'for' loop?
- How to get integer values from a string in Python?
- Optimize this Python code that involves matrix inversion
- how to read a file with dictionary?
- How to extract only audio from downloading video? Python yt-dlp
- How to solve Invalid element(s) received for the 'color' property?
- Plotly Value error - Invalid property for colour
- no module named distutils....but distutils installed?
- how can i change the venv of my current shell from within a python script
- What is the difference between "cast(...)" and "... # type: ..."?
- getting the file size with pycurl
- export conda environment without prefix variable which shows local path to executable
- How can I fix datetime error when trying to use the now syntax?
- Python 3 asyncio with aioboto3 seems sequential
- How can I set OpenCV to use origin as bottom-left corner of image?
- Can I use sleep in the main thread without blocking other threads?
- Getting number of elements in an iterator in Python
- Merging tuples if they have one common element
- How to specify the response_model in FastAPI on a non-default return?
- PyCharm doesn’t see files in project
- How to tune PID controller on non-linear model (here: diving of a body)
- Create list of factorials using list comprehension
- Rewriting URL request function to satisfy GitHub CodeQL server side request forgery (SSRF) warning