Search code examples
htmlbashcurlcalcexpr

Arithmetic in web scraping in a shell


so, I have the example code here:

#!/bin/bash
clear

curl -s  https://www.cnbcindonesia.com/market-data/currencies/IDR=/USD-IDR |
html2text |
sed -n '/USD\/IDR/,$p' |
sed -n '/Last updated/q;p' |
tail -n-1 |
head -c+6 && printf "\n"

exit 0

this should print out some number range 14000~15000

lets start from the very basic one, what I have to do in order to print result + 1 ? so if the printout is 14000 and increment it to 1 become 14001. I suppose the result of the html2text is not calculatable since it should be something like string output not integer.

the more advance thing i want to know is how to calculate the result of 2 curl results?


Solution

  • What I would do, + :

    $ num=$(xidel -se '//div[@class="mark_val"]/span[1]/text()' 'https://url')
    $ num=$((${num//,/}+1)) # num was 14050
    $ echo $num 
    

    Output

    14051
    

     Explanations

    $((...))
    

    is an arithmetic substitution. After doing the arithmetic, the whole thing is replaced by the value of the expression. See http://mywiki.wooledge.org/ArithmeticExpression

    Command Substitution: "$(cmd "foo bar")" causes the command 'cmd' to be executed with the argument 'foo bar' and "$(..)" will be replaced by the output. See http://mywiki.wooledge.org/BashFAQ/002 and http://mywiki.wooledge.org/CommandSubstitution

    Bonus

    You can compute directly in , thanks Reino using syntax :

    $ xidel -s <url> e 'replace(//div[@class="mark_val"]/span[1],",","") + 1' 
    

    And to do addition arithmetic of 2 values :

    $ xidel -s <url> -e '
        let $num:=replace(//div[@class="mark_val"]/span[1],",","")
        return $num + $num
    '