Adding a new column to dask dataframe throws ValueError: Length of values does not match length of index

Question

i understand that this traceback

ValueError: Length of values does not match length of index

arises from the fact that one dataframe is longer or shorter than the other dataframe during ddf.assign(new_col=ts_col or the same operation in ddf['ts_col'] = ts_col.

The problem is, I do not see how the length are differ - Explain in code:

from dask import dataframe as dd

# Read data
ddf = dd.read_csv(csv_path)
ddf.persist()


# Convert to unixtimestamp to pandas timestamp
ts_col = pd.to_datetime(ddf.ts_unixtime_sec_prec, unit='s', errors='coerce')
ts_col.fillna()

# Check data
> ts_col[0:2]
< DatetimeIndex(['2019-05-23 09:09:56', '2019-05-23 09:09:56'], dtype='datetime64[ns]', freq=None)

# Checking length
> len(ddf.index) 
< 11227296

> len(ts_col)
< 11227296

# Try to assign it to dataframe.
> ddf['ts_col'] = ts_col
< ValueError: Length of values does not match length of index <<< Error

Answer 1

Got it working with a lambda / map function:

df['ts'] = df['ts'].map(lambda x: pd.to_datetime(x, errors='coerce'))

Source