Search code examples
javadatasetnetcdfscientific-computingncml

How to create a netCDF-4 file with a variable length variable from ncml?

I'm trying to write a netCDF-4 file with a variable length variable.

Ideally I'd have used the ncgen utility, but it does not support ncml input. Therefore, I'm using the netCDF-java lib 4.6.6. The following ncml without a variable length variable produces a workable netCDF-4 file:

java -Xmx1g -classpath ~/dump/netcdfAll-4.6.6.jar ucar.nc2.dataset.NetcdfDataset -in test.ncml -out test.nc4 -netcdf4

This is the corresponding ncml/xml:

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://what.no">
  <group name="data">
    <dimension name="number_packets" length="6"/>
    <variable name="packet_time_utc" shape="number_packets" type="double">
    </variable>
  </group>
</netcdf>

If I'm adding another unlimited dimension to the ncml, failure occurs

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://what.no">
  <group name="data">
    <dimension name="number_packets" length="6" isVariableLength="true" isShared="false"/>
    <variable name="packet_time_utc" shape="number_packets" type="double">
    </variable>
  </group>
</netcdf>

According to the ncml schema, the definition of the dimension is correct. Error msg NetCDF: Variable not found:

NetcdfDatataset read from test-vlen.ncml write to test-Vlen.nc4 java.io.IOException: -49: NetCDF: Variable not found
        at ucar.nc2.jni.netcdf.Nc4Iosp.writeData(Nc4Iosp.java:2835)
        at ucar.nc2.jni.netcdf.Nc4Iosp.writeData(Nc4Iosp.java:2789)
        at ucar.nc2.NetcdfFileWriter.write(NetcdfFileWriter.java:954)
        at ucar.nc2.NetcdfFileWriter.write(NetcdfFileWriter.java:938)
        at ucar.nc2.FileWriter2.copyAll(FileWriter2.java:431)
        at ucar.nc2.FileWriter2.copyVarData(FileWriter2.java:384)
        at ucar.nc2.FileWriter2.write(FileWriter2.java:199)
        at ucar.nc2.dataset.NetcdfDataset.main(NetcdfDataset.java:1888)
Exception in thread "main" java.io.IOException: -101: NetCDF: HDF error
        at ucar.nc2.jni.netcdf.Nc4Iosp.close(Nc4Iosp.java:289)
        at ucar.nc2.NetcdfFileWriter.abort(NetcdfFileWriter.java:1032)
        at ucar.nc2.FileWriter2.write(FileWriter2.java:207)
        at ucar.nc2.dataset.NetcdfDataset.main(NetcdfDataset.java:1888)

Specifying length="*", as recommended here, results in a different NumberFormatException error.

I'm using the latest versions available:

  • HDF 1.8.17
  • netcdf C 4.4.0
  • netCDF-java lib 4.6.6

The error is about the failure to find the variable (FileWrite wants to copy all variables). However, I'm stuck trying to find the origin of the error.

Solution

  • I reproduced this with the netCDF-java lib branch 5.0.0; commit 0cc266d, but the error is a little clearer:

    With your second example: 

    java.lang.IllegalArgumentException: Dimensions added to a group must be shared.

    Upon removing isShared="false"

    java.lang.IllegalArgumentException: variable length dimension cannot be shared or unlimited

    It looks to me that there is no support just yet.