Search code examples
htmlbashawkweb-scrapingxmlstarlet

How to scrape a HTML table and print it on terminal using Bash?

I've been trying to scrape (using BASH) the table from this website: http://www.tesouro.fazenda.gov.br/tesouro-direto-precos-e-taxas-dos-titulos , which are the current prices for Brazil Government Bonds. I want to develop a script that gives me the same table, formatted to terminal output so I can assign it to an alias and get the prices quickly (and publish it on github so others can use it as well, or even use it on other code).

What I've done so far:

#!/bin/bash
url="http://www.tesouro.fazenda.gov.br/tesouro-direto-precos-e-taxas-dos-titulos"

lynx -source "$url" |
grep -o '<tbody>.*</tbody>' |
#gets the entire line
sed 's/\(<tr>\|<\/tr>\)//g' |  
sed 's/<[^>]*>//g'
#cleans the html tags

Running "./script.sh > file.txt" gives me the output:

Investir Mercado Aberto 9h30min às 18h Preços e taxas dos títulos públicos disponíveis para investir Título Vencimento Indexador Taxa de Rendimento (% a.a.) Valor Mínimo Preço Unitário Indexados ao IPCA Tesouro IPCA+ 2024 (NTNB Princ) 15/08/2024 5,05 R$43,62 R$2.181,47 Tesouro IPCA+ 2035 (NTNB Princ) 15/05/2035 5,48 R$35,96 R$1.198,82 Tesouro IPCA+ 2045 (NTNB Princ) 15/05/2045 5,48 R$35,21 R$704,20 Tesouro IPCA+ com Juros Semestrais 2026 (NTNB) 15/08/2026 5,10 R$32,72 R$3.272,36 Tesouro IPCA+ com Juros Semestrais 2035 (NTNB) 15/05/2035 5,35 R$32,63 R$3.263,54 Tesouro IPCA+ com Juros Semestrais 2050 (NTNB) 15/08/2050 5,42 R$33,55 R$3.355,64 Prefixados Tesouro Prefixado 2020 (LTN) 01/01/2020 8,40 R$33,84 R$846,13 Tesouro Prefixado 2023 (LTN) 01/01/2023 10,20 R$30,58 R$611,76 Tesouro Prefixado com Juros Semestrais 2027 (NTNF) 01/01/2027 10,36 R$30,69 R$1.023,16 Indexados à Taxa Selic Tesouro Selic 2023 (LFT) 01/03/2023 0,00 R$92,38 R$9.238,83 Resgatar Mercado Aberto 9h30min às 18h Preços e taxas dos títulos públicos disponíveis para resgatar Título Vencimento Indexador Taxa de Rendimento (% a.a.) Preço Unitário Indexados ao IPCA Tesouro IPCA+ 2019 (NTNB Princ) 15/05/2019 2,93 R$2.907,76 Tesouro IPCA+ 2024 (NTNB Princ) 15/08/2024 5,17 R$2.164,92 Tesouro IPCA+ 2035 (NTNB Princ) 15/05/2035 5,60 R$1.175,37 Tesouro IPCA+ 2045 (NTNB Princ) 15/05/2045 5,60 R$682,63 Tesouro IPCA+ com Juros Semestrais 2020 (NTNB) 15/08/2020 4,17 R$3.221,84 Tesouro IPCA+ com Juros Semestrais 2024 (NTNB) 15/08/2024 5,08 R$3.240,43 Tesouro IPCA+ com Juros Semestrais 2026 (NTNB) 15/08/2026 5,22 R$3.246,88 Tesouro IPCA+ com Juros Semestrais 2035 (NTNB) 15/05/2035 5,47 R$3.221,99 Tesouro IPCA+ com Juros Semestrais 2045 (NTNB) 15/05/2045 5,56 R$3.231,12 Tesouro IPCA+ com Juros Semestrais 2050 (NTNB) 15/08/2050 5,54 R$3.298,33 Prefixados Tesouro Prefixado 2018 (LTN) 01/01/2018 7,06 R$994,86 Tesouro Prefixado 2019 (LTN) 01/01/2019 7,13 R$929,11 Tesouro Prefixado 2020 (LTN) 01/01/2020 8,52 R$844,19 Tesouro Prefixado 2021 (LTN) 01/01/2021 9,46 R$757,85 Tesouro Prefixado 2023 (LTN) 01/01/2023 10,32 R$608,40 Tesouro Prefixado com Juros Semestrais 2021 (NTNF) 01/01/2021 9,28 R$1.060,17 Tesouro Prefixado com Juros Semestrais 2023 (NTNF) 01/01/2023 10,08 R$1.039,70 Tesouro Prefixado com Juros Semestrais 2025 (NTNF) 01/01/2025 10,37 R$1.025,50 Tesouro Prefixado com Juros Semestrais 2027 (NTNF) 01/01/2027 10,48 R$1.016,60 Indexados à Taxa Selic Tesouro Selic 2021 (LFT) 01/03/2021 0,04 R$9.226,94 Tesouro Selic 2023 (LFT) 01/03/2023 0,04 R$9.219,57 Indexados ao IGP-M Tesouro IGPM+ com Juros Semestrais 2021 (NTNC) 01/04/2021 4,42 R$3.756,86 Tesouro IGPM+ com Juros Semestrais 2031 (NTNC) 01/01/2031 5,33 R$5.863,43

which is in portuguese, so I'll try my best to help getting helped:

  1. Everything, unfortunately, is in one single line, I believe it's because the HTML file is this way and grep gets it.
  2. The important data is right after the last word "investir" and the first "Resgatar". These are the bonds available to buy from the government.
  3. Between the last word "resgatar" and the end of table, are the bonds which are no longer available for buying, but you can sell back to the government anytime until due date.
  4. The table field titles are "Titulo - Vencimento - Taxa de Rendimento (% a.a.) - Preço Unitario", which translates to "Title - Due date - Interst Rate (% year) - Unit Price"

If you guys could help me at least removing these unwanted 'description words' between the table titles (which is, to say, from beginning of line until first ocurrence of "Título" and then from the first "Resgatar" until the next ocurrence of "Título"), I believe this tool could help others as well.

From there, I can try to AWK my way out of the rest of the file and format the way I want. I just need to remove these, which I haven't been able to do, despite searching a couple of questions here in stack. Also, if someone has a more smart way of doing what I intend to do, please feel free to discard everything dumb I've done and suggest a more intelligent approach in BASH!

Thank you very much in advance.

Solution

  • An Example:

    wget -q -O - 'http://www.tesouro.fazenda.gov.br/tesouro-direto-precos-e-taxas-dos-titulos' |\
  xmlstarlet format --recover --html 2>/dev/null |\
  xmlstarlet select --html --template --value-of "/html/body/div/div/div/div/div/div/div/div/div/div/table/tbody/tr/td[@class='listing0' or @class='listing' or @class='listing ']" |\
  paste -d ";" - - - - - |\
  column -s ";" -t

    Output:

    Tesouro IPCA+ 2024 (NTNB Princ)                     15/08/2024  5,06   R$43,60  R$2.180,09
Tesouro IPCA+ 2035 (NTNB Princ)                     15/05/2035  5,48   R$35,96  R$1.198,82
Tesouro IPCA+ 2045 (NTNB Princ)                     15/05/2045  5,48   R$35,21  R$704,20
Tesouro IPCA+ com Juros Semestrais 2026 (NTNB)      15/08/2026  5,10   R$32,72  R$3.272,36
Tesouro IPCA+ com Juros Semestrais 2035 (NTNB)      15/05/2035  5,35   R$32,63  R$3.263,54
Tesouro IPCA+ com Juros Semestrais 2050 (NTNB)      15/08/2050  5,41   R$33,60  R$3.360,49
Tesouro Prefixado 2020 (LTN)                        01/01/2020  8,38   R$33,85  R$846,45
Tesouro Prefixado 2023 (LTN)                        01/01/2023  10,20  R$30,58  R$611,76
Tesouro Prefixado com Juros Semestrais 2027 (NTNF)  01/01/2027  10,37  R$30,67  R$1.022,61
Tesouro Selic 2023 (LFT)                            01/03/2023  0,00   R$92,38  R$9.238,83

    I inserted xmlstarlet format --recover --html 2>/dev/null because HTML from this URL is not valid.

    See: xmlstarlet select --help, man paste and man column