Difference between NotesDocument objects from database and in-memory?

Question

I have noticed in one of my Java agents that when storing a lot of Document objects in a Hashmap I get OutofMemoryException. What I'm trying to achieve is this:

I want to populate my Domino database with information from an Oracle database. The logic is to get all from Oracle and then get all documents from Domino database. All new from Oracle is to be saved and all not in Oracle but in database shall be removed from Domino. A sort of replication of you will…

I start with looping all documents in the Domino database and store them in a HashMap with UNID as key and Document object as value:

dominoHash.put(doc.getUniversalId(), doc);

Then I loop the resultset and creates in-memory Document objects:

Document doc = db.createDocument();
doc.replaceItemValue("MyField", rset.getString("MYCOLUMN");
oracleHash.put(rset.getString("UNID"), doc);

In my TEST environment I have +20000 Domino documents and the same in Oracle. In this case they are identical. The OutOfMemoryException occurs always when trying to store in oracleHash even if I have excluded all Domino document logic from the code to "save memory"…

I thought the objects I'm storing in the different Hashmaps are the same but some how they are not. I can successfully store my Domino Document objects in a Hashmap but not my Oracle document objects.

To make it more strange the document stored from Domino has more fields than the Oracle view has columns.

Why could be the cause of this?

In this specific case I use org.openntf.domino API and haven't tried native API (yet).

Answer 1

It is hardly a good idea to keep all Domino documents in memory. At some point in time you run out of the C handles in the core API.

You might want/need to change your approach. Trying to keep everything in memory will always run out of memory once you reach "the size". What you need is, aptly named by @Andre Guirard, "The Pirate algorithm":

Have a view in Domino and Oracle each, that are sorted by your comparison key (The UNID as I understand it from your code). Lets presume N is Notes and O is Oracle. Your code (here pseudo code) would roughly look like this:

         OEntry = O.getFirst
         for all NEntry in N
             if NEntry.key > OEntry.key
                InsertFromOracle until NEntry.key = OEntry.key
             else if Nentry.key < OEntry.key
                DeleteFromNotes until NEntry.key = OEntry.key
             else
                SyncValuesFromOracleToNotes
             end if
          end for
         if O.hasMoreEntries
             InsertFromOracle until not O.hasMoreEntries
         end if

The InsertFromOracle of course need to check if there are elements left in Oracle to do that.

A LotusScript example had been written by Andre (Look for the Pirates Agent, named after the swagger walking of a pirate)

You don't need to keep entire tables or document collections in memory. Since loading and comparing run in a single loop it should also perform quite well.

Let us know how it goes!