I am converting a text file to netCDF format using xarray. When I am using netCDF4 format and Python3, it is storing string variables as strings but when I use Python2 it stores them as n-dimensional character arrays. I have tried to set dtype='str' in encoding and that didn't make any difference. Is there a way to make these variables to have string data-type using Python2? Any thoughts would be appreciated.
Here is my code:
import pandas as pd
import xarray as xr
column_names = ['timestamp', 'air_temp', 'vtempdiff', 'rh', 'pressure', 'wind_dir', 'wind_spd']
df = pd.read_csv(args.input_file, skiprows = 1, header=None, names = column_names)
ds = xr.Dataset.from_dataframe(df)
encoding = {'timestamp': {'dtype': 'str'},
'air_temp': {'_FillValue': 9.96921e+36, 'dtype': 'f4'}
}
ds.to_netcdf(op_file.nc, format = 'NETCDF4', unlimited_dims={'time':True}, encoding = encoding)
When I do ncdump of the op_file.nc using Python3.6, I get:
netcdf op_file {
dimensions:
time = UNLIMITED ; // (24 currently)
variables:
string timestamp(time) ;
float air_temp(time) ;
.
.
.
And when I use Python2.7, I get:
netcdf op_file {
dimensions:
time = UNLIMITED ; // (24 currently)
string20 = 20 ;
variables:
char timestamp(time, string20) ;
timestamp:_Encoding = "utf-8" ;
float air_temp(time) ;
.
.
.
The sample input file looks like this:
# Fields: stamp,AGO-4.air_temp,AGO-4.vtempdiff,AGO-4.rh,AGO-4.pressure,AGO-4.wind_dir,AGO-4.wind_spd
2016-11-30T00:00:00Z,-36.50,,56.00,624.60,269.00,5.80
2016-11-30T01:00:00Z,-35.70,,55.80,624.70,265.00,5.90
Xarray maps Python 2's str/bytes type to NetCDF's NC_CHAR type. Both these types represent single byte character data (generally ASCII) so this makes a certain amount of sense.
To get a netCDF string NC_STRING, you need to pass pass unicode data (str on Python 3). You can get this by explicitly coercing your timestamp column to unicode, either with .astype(unicode) or by passing {'dtype': unicode} in encoding.