Search code examples
csvgnu-parallel

gnu-parallel doesn't work without trailing commas in CSV

I have the following parallel command that I run

parallel --header : --colsep , \
  python -m imitation.scripts.train \
  with \
  "$@" \
  gail \
  {env_config} \
  log_root="${OUTPUT_DIR}" \
  init_trainer_kwargs.rollout_glob=expert_models/rollouts/{rollout_glob} \
  init_trainer_kwargs.n_expert_demos={n_demonstrations} \
  :::: experiments/gail_benchmark_config.csv

which pulls out {rollout_glob}, {n_demonstrations}, and {env_config} from the following csv file:

$ cat -vet experiments/gail_benchmark_config.csv | head
env_config,rollout_glob,n_gen_steps_per_epoch,n_epochs,n_demonstrations^M$
cartpole,CartPole-v1.pkl,5000,300,1^M$
cartpole,CartPole-v1.pkl,5000,300,4^M$
cartpole,CartPole-v1.pkl,5000,300,7^M$
cartpole,CartPole-v1.pkl,5000,300,10^M$
mountain_car,MountainCar-v0.pkl,5000,300,1^M$
mountain_car,MountainCar-v0.pkl,5000,300,4^M$
mountain_car,MountainCar-v0.pkl,5000,300,7^M$
mountain_car,MountainCar-v0.pkl,5000,300,10^M$
acrobot,Acrobot-v1.pkl,5000,300,1^M$

It seems that parallel can't read n_demonstrations properly, and so it leaves {n_demonstrations} unfilled. I need to append a comma to the end of each CSV line to get my parallel script to work.

Example incorrectly generated command:

python -m imitation.scripts.train with gail cartpole [...] init_trainer_kwargs.n_expert_demos={n_demonstrations} seed=0

In other questions people seem to be reading CSVs fine without using these trailing commas.

Anyone know what I can do differently so that I don't have to add trailing commas to my CSV? (This is annoying because it is equivalent to an empty column, and Google Sheets ignores empty columns when I'm trying to export my CSV.)

Solution

  • I cannot produce your error. I have tested every version of GNU Parallel since 20120722, and all work fine if you remove the last ,.

    So something else is going on.

    It can be in your environment. Can you find a different machine (e.g. a VM from http://www.osboxes.org/virtualbox-images/) and run it there?

    It can be in the input file. Maybe there is characters that are invisible? Can you upload the actual file (preferably zipped/gzipped so we know nothing in it has changed)?