Search code examples
javaxmloracle-databaseguavaguava-table

Processing huge data from multiple flat files and convert into xml formation based on requirement

we have an requirement to form XML tags for data avaible in multiple tables in Database.(Ex: Customer (one row), Address (1 row), Phone (3 rows) into singe XML tag.

So arrived to export it from database as flat files as billions of records are there in different tables.

With these flat files, Need to form xml tags. i tried using Guava Table for loading these data and fetching based on row identifier. But when loading huge data in Guava table, getting following error. Any pointers on using proper approach?

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.lang.StringBuilder.toString(Unknown Source)
    at TableExample.substringeliminateDemo(TableExample.java:93)
Solution

  • Hope you are running with the default Xmx arguments. Try using a large Xmx value. Run with java -Xmx4g to get a large heap space (in this case 4GB). However, the right approach is to load data in smaller batch sizes - not the entire table data.