I have to run data analysis on about 2 million lines of data and each line about 250 bytes long. So total about 500 megabytes of data. I am running latest Rakudo on Virtualbox Linux with 4G memory.
After about 8 hours, I got MoarVM panic due to running out of memory. How do I give more memory to MoarVM? Unfortunately I cannot break up the 2 millions into chunks and write to a files first because part of the data analysis requires the whole 2-m lines.
I suggest you tackle your problem in several steps:
Prepare two small sample files if you haven't already. Keep them very small. I suggest a 2,000 lines long file and a 20,000 line long one. If you already have some sample files of around that length then those will do. Run your program for each file, noting how long each took and how much memory was used.
Update your question with your notes about duration and RAM use; plus links to your source code if that's possible and the sample files if that's possible.
Run the two sample files again but using the profiler as explained here. See what there is to see and update your question.
If you don't know how to do any of these things, ask in the comments.
If all the above is fairly easy, repeat for a 100,000 line file.
Then we should have enough data to give you better guidance.