Concatenate a large number of HDF5 files

I have about 500 HDF5 files each of about 1.5 GB.

Each of the files has the same exact structure, which is 7 compound (int,double,double) datasets and variable number of samples.

Now I want to concatenate all this files by concatenating each of the datasets so that at the end I have a single 750 GB file with my 7 datasets.

Currently I am running a h5py script which:

creates a HDF5 file with the right datasets of unlimited max
open in sequence all the files
check what is the number of samples (as it is variable)
resize the global file
append the data

this obviously takes many hours, would you have a suggestion about improving this?

I am working on a cluster, so I could use HDF5 in parallel, but I am not good enough in C programming to implement something myself, I would need a tool already written.

Solution

I found that most of the time was spent in resizing the file, as I was resizing at each step, so I am now first going trough all my files and get their length (it is variable).

Then I create the global h5file setting the total length to the sum of all the files.

Only after this phase I fill the h5file with the data from all the small files.

now it takes about 10 seconds for each file, so it should take less than 2 hours, while before it was taking much more.