Bash: text processing command

Question

I have been able to do what I want with one command one line, but I do know there must be some more elegant way to do what I am doing. Please tell me what your methods are... I would like to learn more sophisticated way of processing text files...

Original file is a vcf file looks like this

##fileformat=VCFv4.2
##FILTER=<ID=PASS,Description="All filters passed">
##fileDate=20180307
##source=PLINKv1.90
##contig=<ID=1,length=249214117>
##contig=<ID=2,length=242842533>
##contig=<ID=3,length=197896741>
...
...
...
#CHROM  POS ID  REF ALT QUAL    FILTER  INFO    FORMAT
22  16258171    22:16258171:D:3 A   .   .   .   .   GT
22  16258174    22:16258174:T:C T   .   .   .   .   GT
22  16258183    22:16258183:A:T A   .   .   .   .   GT
22  16258189    22:16258189:G:T G   .   .   .   .   GT

My goal is to generate a file looks like this:

22  16258171  16258171  D  3
22  16258174  16258174  T  C
22  16258183  16258183  A  T
22  16258189  16258189  G  T
22  16258211  16258211  A  G
22  16258211  16258211  A  T
22  16258220  16258220  T  G
22  16258221  16258221  C  T
22  16258224  16258224  C  T
22  16258227  16258227  G  A

I did the following steps to achieve the final goal but it's so cumbersome and so ugly too...

#remove comments
sed '/^[[:blank:]]*#/d;s/#.*//' chr22.vcf > no_comment_chr22.vcf

#take out the third columns for splitting
cut -d $'\t' -f 3 no_comment_chr22.vcf > no_comment_chr22.col3_to_split.txt

#Split string by delimiter and get N-th element, use as col4
cut -d':' -f3 no_comment_chr22.col3_to_split.txt > chr22_as_col4.txt

#Split string by delimiter and get N-th element, use as col5
cut -d':' -f4 no_comment_chr22.col3_to_split.txt > chr22_as_col5.txt

#get first 2 columns
cut -d $'\t' -f 1-2 no_comment_chr22.vcf > no_comment_chr22.col1to2.txt

#get the second column as col3 
cut -d $'\t' -f 2 no_comment_chr22.vcf > no_comment_chr22.ascol3.txt

#Combine files column-wise
paste no_comment_chr22.col1to2.txt no_comment_chr22.ascol3.txt chr22_as_col4.txt chr22_as_col5.txt | column -s $'\t' -t  > chr22_input_5cols.txt

I was able to get what I need but .. gahhh, this is so ugly. Please tell me what people do to advance their text processing skills and how to improve things like this.. thank you!!

Answer 1

Using awk:

awk -F'(:| +)' '/^#/ {next} {print $1,$2,$4,$5,$6}' sample.vcf


22 16258171 16258171 D 3
22 16258174 16258174 T C
22 16258183 16258183 A T
22 16258189 16258189 G T

This is specifying a regular expression as the filed delimiter (-F) and then ignoring the comment lines (^#) or printing the corresponding fields (1,2,4,5,6).