Search code examples
pythonpandaspython-xarraynetcdf

Xarray failing to save NetCDF due to illegal characters

I have a DataFrame in python. Using "df.to_excel('temp/df.xlsx')" i have exported it, here is a screenshot of the resulting file in Excel: table This exists in python as a Pandas DataFrame. I have pushed a few of these together into a 3d array using xarray. I can not save this to netCDF due to: RuntimeError: NetCDF: Name contains illegal characters, and yet it does not appear to contain any illegal characters. Is there a hidden part to this DataFrame that I can not see that is causing problems?

import pandas ad pd
import xarray as xr

dflist.append(df)
xarray = xr.concat([df.to_xarray() for df in dflist], dim=f'{["stationIDlist"]}')
xarray.to_netcdf('temp/xarray.nc')

The error is as follows:

xarray.to_netcdf('temp/xarray.nc', format='NETCDF4')
Traceback (most recent call last):
  File "~/python3.11/site-packages/IPython/core/interactiveshell.py", line 3577, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-19-65321829c66f>", line 1, in <module>
    xarray.to_netcdf('temp/xarray.nc', format='NETCDF4')
  File "~/python3.11/site-packages/xarray/core/dataset.py", line 2327, in to_netcdf
    return to_netcdf(  # type: ignore  # mypy cannot resolve the overloads:(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/python3.11/site-packages/xarray/backends/api.py", line 1337, in to_netcdf
    dump_to_store(
  File "~/python3.11/site-packages/xarray/backends/api.py", line 1384, in dump_to_store
    store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
  File "~/python3.11/site-packages/xarray/backends/common.py", line 366, in store
    self.set_dimensions(variables, unlimited_dims=unlimited_dims)
  File "~/python3.11/site-packages/xarray/backends/common.py", line 443, in set_dimensions
    self.set_dimension(dim, length, is_unlimited)
  File "~/python3.11/site-packages/xarray/backends/netCDF4_.py", line 481, in set_dimension
    self.ds.createDimension(name, size=dim_length)
  File "src/netCDF4/_netCDF4.pyx", line 2710, in netCDF4._netCDF4.Dataset.createDimension
  File "src/netCDF4/_netCDF4.pyx", line 3711, in netCDF4._netCDF4.Dimension.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2028, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Name contains illegal characters
Solution

  • Try this: xarray = xr.concat([df.to_xarray() for df in dflist], dim="stationIDlist"), this fixed an issue on my end.

    I had a similar issue and it seems that netcdf doesn't like any special characters (you included square brackets [] as part of your dimension name).