Search code examples
javahpc

Is there a way to write a part from memory to a file


I am trying to develop some code that will probably run multiple weeks on a HPC environment. The problem is that chances are rather small that I'll be able to monopolise a cluster to run my program as a whole and therefore I am thinking about ways to store my progress into some file system.

As I was thinking, I wondered whether there wouldn't be a way to take some snapshot from the memory my application is using and store that in some file. I have this feeling this should be the simplest way to just stop processing and start again whenever you want, because you could just load that file into memory and continue working as if nothing happened.

I started searching around on the internet trying to find something about this, but as I don't really know the term (I suppose there is a term) for this and I didn't get any further than billions of ways to write objects to files as efficient as possible. I don't believe though that those writers and streams perform something as I described above... (please correct me if I made a mistake!)

My question thus could be formulated as follows:
Is there any way to do something like this in java? If not, does there exist anything that approaches the idea or performance of what I tried to describe?


update 2

As update 1 became irrelevant, I would like to update my example of what I am trying to do as I am now using a state-object that should be stored:

abstract class State {

    private final int order;
    
    private SortedMap<Key, Long> map1 = null;
    private SortedMap<Code, Long> map2 = null;
    private SortedMap<Key, Long> derivedFromMap1 = null;
    private BigInteger sum = null;

    State(int order) {
        this.order = order;
    }

    abstract fillMap1(Algorithm algo);
    abstract fillMap2(Algorithm algo);
    abstract fillMap3(Algorithm algo);
    abstract sum(Algorithm algo);

}

What I would like to happen now is that the State-objects are persisted every now and then without losing too much of CPU-time wasted on IO (I hope I am correct in the assumption that the OS can start writing a part from memory to the file system without needing to bother the CPU for every line).

There is also a chance that these State-objects get rather large and maybe even too large to reside completely in memory (I actually have no idea, but I might want to take it into account). So instead of having loads of virtual memory, it might be useful to just have it all written to the right file at once...


Solution

  • Is there any way to do something like this in java?

    Yes, you serialize all the data you need to preserve and on startup deserialize that data and continue where you were up to. You might like to check point your application periodically in case it dies/crashes so you can continue from a recent point.

    If not, does there exist anything that approaches the idea or performance of what I tried to describe?

    Yes, you would need to provide some details get a more specific answer.