Search code examples
speech-recognitionqsubkaldi

Kaldi: Output of qsub was: qsub: illegal -c value "" when trying to run the Common Voice recipe


I am trying to run Kaldi's Common Voice recipe (kaldi/egs/commonvoice/s5/run.sh) on my computer (i.e., not on a cluster). It crashes with the error message Output of qsub was: qsub: illegal -c value "". What could be the issue?


Specifically, here is the entire error stack:

[...]
Succeeded in formatting LM: 'data/local/lm.gz'
steps/make_mfcc.sh --cmd queue.pl --mem 2G --nj 20 data/valid_train exp/make_mfcc/valid_train mfcc
utils/validate_data_dir.sh: Successfully validated data-directory data/valid_train
steps/make_mfcc.sh: [info]: no segments file exists: assuming wav.scp indexed by utterance.
queue.pl: Error submitting jobs to queue (return status was 512)
queue log file is exp/make_mfcc/valid_train/q/make_mfcc_valid_train.log, command was qsub -v PATH -cwd -S /bin/bash -j y -l arch=*64* -o exp/make_mfcc/valid_train/q/make_mfcc_valid_train.log  -l mem_free=2G,ram_free=2G  -t 1:20 /home/ubuntu/kaldi/egs/commonvoice/s5/exp/make_mfcc/valid_train/q/make_mfcc_valid_train.sh >>exp/make_mfcc/valid_train/q/make_mfcc_valid_train.log 2>&1
Output of qsub was: qsub: illegal -c value ""
usage: qsub [-a date_time] [-A account_string] [-b secs]
      [-c [ none | { enabled | periodic | shutdown |
      depth=<int> | dir=<path> | interval=<minutes>}... ]
      [-C directive_prefix] [-d path] [-D path]
      [-e path] [-h] [-I] [-j oe] [-k {oe}] [-l resource_list] [-m n|{abe}]
      [-M user_list] [-N jobname] [-o path] [-p priority] [-P proxy_user] [-q queue]
      [-r y|n] [-S path] [-t number_to_submit] [-T type]  [-u user_list] [-w] path
      [-W otherattributes=value...] [-v variable_list] [-V ] [-x] [-X] [-z] [script]

I ran the Common Voice recipe as follows:

## Retrieving Kaldi source code 
git clone https://github.com/kaldi-asr/kaldi.git
cd kaldi
export KALDI_GIT_ROOT=`pwd`
echo $KALDI_GIT_ROOT

## Compile Kaldi tools
cd $KALDI_GIT_ROOT/tools
sudo apt-get install -y  zlib1g-dev automake autoconf libtool libatlas3-base subversion 
make -j 24

## Compile Kaldi
cd $KALDI_GIT_ROOT/src
./configure
make -j 24

## Common Voice
cd $KALDI_GIT_ROOT/tools
sudo ./install_srilm.sh
# required by install_sequitur.sh
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
sudo python get-pip.py
sudo apt-get install -y swig python-setuptools python-dev
sudo pip install numpy
sudo ./extras/install_sequitur.sh
# For qsub
sudo apt-get install -y torque-server torque-client torque-mom torque-pam
# Start Kaldi training on Common Voice
sudo apt install sox libsox-fmt-mp3 # run.sh will convert Common Voice MP3s to WAV using sox
cd $KALDI_GIT_ROOT/egs/commonvoice/s5
source $KALDI_GIT_ROOT/tools/env.sh
./run.sh

I use Ubuntu 16.04.4 LTS.


Solution

  • The issue is that qsub was the wrong one: one should use the included in Sun Grid Engine, and not the one used in the Torque program.

    sudo apt-get remove torque-server torque-client torque-mom torque-pam
    sudo apt-get install gridengine-master gridengine-client gridengine-exec
    

    Note that if Sun Grid Engine can be tricky to configure and it's not much advantageous to go through it if one uses single computer.

    To train Kaldi on Common Voice without Sun Grid Engine, one can replace queue.pl to run.pl in /kaldi/egs/commonvoice/s5/cmd.sh:

    export train_cmd="run.pl --mem 2G"
    export decode_cmd="run.pl --mem 4G"
    export mkgraph_cmd="run.pl --mem 8G"