Search code examples
javascriptpythonselenium-webdriverweb-scraping

Python_crawling <li style="display: none;">


I am trying to scrape below HTML using Selenium on Python, but I cannot scrape the text


<div class="tt">
    <ul style="list-style-type:none" id="abs2">
        <li>
        Funeral rites have considerably been transformed as family structure has shifted to nuclear family and urbanization has been expedited. People came to consign more and more the rites commitment to funeral service providers and the ritual process itsel...
        </li>
    <li style="display:none">
        Funeral rites have considerably been transformed as family structure has shifted to nuclear family and urbanization has been expedited. People came to consign more and more the rites commitment to funeral service providers and the ritual process itself became a lot modernized and commercialized at the same time. As a result, the cost spent for family rites each year has increaseda lot.<br>Today, the number of single family is on the increase. If this trend lasts continuously for so long, the cost for funeral rites will become a heavy burden to the bereaved in near future. 
    </li>
    </ul>
</div>

I am not getting success in crawling by clicking next page. Please help me. Here is my code:

a = driver.find_element_by_css_selector('#abs1 > li:nth-child(2)').get_attribute('style')

print(a)

Output of "print(a)" as below;

display: none;


Solution

  • You are printing the style attribute.. Have you tried text?

    a = driver.find_element_by_css_selector('#abs2 > li:nth-child(2)').text
    
    print(a)
    

    If that does not work, try to get innerText, like this:

    a = driver.find_element_by_css_selector('#abs2 > li:nth-child(2)').get_attribute('innerText')
    
    print(a)