Search code examples
bashcurluserscripts

How to extract links behind a text tag of web page (using either curl,wget or userscript)


I'm trying to extract href links under tag. Refer to the attachment. I want to save all link under the tag "PDF". http://tinypic.com/r/2n9erdj/8 Sorry I'm not allowed update pictures as yet.

Specifically the href details appear as arnumber=60940cc as shown in red circle.

Can someone suggest how to implement this. I'm intending to use either a userscript or bash commands.

html elements details relavant to a single pdf is shown below.

<a aria-label="Download or View the PDF:  IEEE Transactions on Power Electronics publication information" href="/stampPDF/getPDF.jsp?tp=&amp;arnumber=6094072"><img class="button" src="http://staticieeexplore.ieee.org/assets/img/iconPdf.png" alt="PDF file icon" title="Download or View the PDF">PDF</a>

The web page I'm testing is

http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6088512

The objective is to filter the content named as "pdf" and its urls.


Solution

  • Try this: Tweak sed part if you don't want "http://ieeexplorer.ieee.org/" just before stamp word or etc

    wget http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6088512 -O file.html
    grep -o "href.*stamp.*\"><" file.html |sed 's#"#"http://ieeexplorer.ieee.org#;s#><##'
    
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=6094070"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=6094072"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=6094110"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=6088513"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5680978"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5985544"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5723758"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5716681"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5936741"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5934597"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5734858"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5756244"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5759746"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5958614"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5999721"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=6021380"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5961632"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5951783"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5983448"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5934423"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5957306"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5898425"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5959991"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5776690"
    href="http://ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5953525"
    

    OR

    $ grep -o "href.*stamp.*\"><" file.html |sed 's#href="#ieeexplorer.ieee.org#;s#"><##'
    
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=6094070
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=6094072
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=6094110
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=6088513
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5680978
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5985544
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5723758
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5716681
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5936741
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5934597
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5734858
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5756244
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5759746
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5958614
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5999721
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=6021380
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5961632
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5951783
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5983448
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5934423
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5957306
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5898425
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5959991
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5776690
    ieeexplorer.ieee.org/stamp/stamp.jsp?tp=&arnumber=5953525