Search code examples
pythonpandasxpathweb-crawler

X Path Python Error- 'list' object has no attribute 'xpath'

I am trying to do a Web-Crawling project, and I am keep on getting an error code called "'list' object has no attribute 'xpath'"/

This is the code so far:

import requests 
from lxml import html
import pandas 

url = "http://www.cnbc.com/"
response = requests.get(url)
doc = html.fromstring(response.text)

headlineNode = doc.xpath('//div[@class="headline"]')
len(headlineNode)

headlines = headlineNode.xpath('/a')
len(headlines)

I want to extraxt the headlines from CNBC Home, but I am keep on getting an error saying

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-15-d9f54582133c> in <module>()
----> 1 headlines = headlineNode.xpath('/a')
      2 len(headlines)

AttributeError: 'list' object has no attribute 'xpath'

Is there anyway to fix this?

Solution

  • The reason you are getting this error is because xpath always results a list of elements and xpath can be called on elements.

    So to get expected results, what you can do is iterate over the headlineNode list and then call the xpath to the element nodes such as node.xpath('/a')

    The following code will give you the list of all the url you needed as a list. You can do something like this :

    import requests 
from lxml import html
import pandas 

url = "http://www.cnbc.com/"
response = requests.get(url)
doc = html.fromstring(response.text)

headlineNode = doc.xpath('//div[@class="headline"]')
len(headlineNode)

url_list  = []
for node in headlineNode : 
    url_node = node.xpath('./a/@href')
    if url_node : 
        url_list.append(url + url_node[0].strip())
print(url_list)
len(url_list)

    This will result in :

    ['http://www.cnbc.com/2017/03/28/asia-markets-to-watch-brexit-news-dollar-movements-oil-prices.html', 'http://www.cnbc.com/2017/03/28/us-markets.html', 'http://www.cnbc.com/2017/03/28/european-markets-investors-shrug-off-trump-concerns.html', 'http://www.cnbc.com/2017/03/27/gold-steady-as-dollar-edges-up-focus-on-trump-agenda.html', 'http://www.cnbc.com/2017/03/27/weaker-dollar-lifts-oil-futures-but-soaring-us-output-weighs.html', 'http://www.cnbc.com/2017/03/28/us-treasurys-edge-lower-as-investors-eye-data-auctions.html', 'http://www.cnbc.com/2017/03/27/dollar-limps-away-from-multi-month-lows-still-seen-vulnerable.html', 'http://www.cnbc.com/2017/03/27/dollar-limps-away-from-multi-month-lows-still-seen-vulnerable.html', 'http://www.cnbc.com/2017/03/28/markets-could-see-heartbreak-if-brexit-is-hard-to-do.html', 'http://www.cnbc.com/2017/03/28/former-google-exec-heres-how-to-know-its-time-to-fire-an-employee.html', 'http://www.cnbc.com/2017/03/28/chinas-secret-plan-to-crush-spacex-and-the-us-space-program.html', 'http://www.cnbc.com/2017/03/28/le-pen-victory-could-be-five-times-as-dangerous-as-greeces-financial-meltdown-ubs.html', 'http://www.cnbc.com/2017/03/28/nuclear-nonproliferation-lessons-from-myanmar.html', 'http://www.cnbc.com/2017/03/28/following-tuesdays-rally-stocks-will-grind-higher-into-2018-analyst-says.html', 'http://www.cnbc.com/2017/03/28/asia-markets-to-watch-brexit-news-dollar-movements-oil-prices.html', 'http://www.cnbc.com/2017/03/28/made-in-america-impossible-tech-startups.html', 'http://www.cnbc.com/2017/03/28/toshibas-westinghouse-to-file-for-us-bankruptcy-sources.html', 'http://www.cnbc.com/2017/03/28/early-reports-of-damage-in-wake-of-australias-cyclone-debbie.html', 'http://www.cnbc.com/2017/03/28/messaging-app-parent-tencent-takes-stake-in-tesla.html', 'http://www.cnbc.com/2017/03/28/rodrigo-duterte-turns-72-a-look-at-his-controversial-presidency-so-far-commentary.html', 'http://www.cnbc.com/2017/03/28/north-korea-could-be-in-final-stages-of-nuclear-test-preparations-report.html', 'http://www.cnbc.com/2017/03/28/congress-clears-way-for-isps-to-sell-browsing-history.html', 'http://www.cnbc.com/2017/03/28/british-pm-may-to-fire-starting-gun-on-brexit.html', 'http://www.cnbc.com/2017/03/28/brexit-is-like-arthritis-it-doesnt-kill-you-but-it-hits-you-says-former-boe-member.html', 'http://www.cnbc.com/2017/03/27/florida-preparesfor-xi-jinpingsvisit.html', 'http://www.cnbc.com/2017/03/28/us-heading-for-recession-after-2-years-of-unsustainable-growth-economist-says.html', 'http://www.cnbc.com/2017/03/28/this-is-what-immigration-means-to-the-us-economy-in-two-charts.html', 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ceo-calls-to-step-down.html', 'http://www.cnbc.com/2017/03/28/next-iphone-is-last-hurrah-for-smartphones-apple.html', 'http://www.cnbc.com/2017/03/28/south-africa-speculates-over-possible-firing-of-finance-minister.html', 'http://www.cnbc.com/2017/03/28/snap-was-a-drunk-at-the-wheel-but-now-looks-like-a-very-real-business-fund-manager-says.html', 'http://www.cnbc.com/2017/03/28/this-ad-agency-is-rebranding-with-the-name-of-its-jewish-founders.html', 'http://www.cnbc.com/2017/03/28/siemens-medical-solutions-wins-41-billion-us-defense-contract.html', 'http://www.cnbc.com/2017/03/28/google-home-ai-speaker-is-coming-to-the-uk-to-challenge-amazon-echo.html', 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ai-impact-jobs-mnuchin-government.html', 'http://www.cnbc.com/2017/03/28/uk-introduces-super-secure-new-one-pound-coin.html', 'http://www.cnbc.com/2017/03/28/major-solar-project-gets-the-green-light-in-northern-ireland.html', 'http://www.cnbc.com/2017/03/28/trump-to-meet-egyptian-president-at-white-house-on-april-3.html', 'http://www.cnbc.com/2017/03/28/tesla-is-rejecting-a-racial-harassment-lawsuit-filed-by-an-assembly-line-worker.html', 'http://www.cnbc.com/2017/03/28/jpmorgan-says-markets-are-underestimating-the-upside-of-asian-banks.html', 'http://www.cnbc.com/2017/03/28/square-uk-launch-ceo-jack-dorsey-ai-acquisitions.html', 'http://www.cnbc.com/https://www.facebook.com/cnbcinternational/', 'http://www.cnbc.com/https://twitter.com/cnbcworld', 'http://www.cnbc.com/http://www.linkedin.com/company/2966126', 'http://www.cnbc.com/http://www.youtube.com/user/CNBCInternational', 'http://www.cnbc.com/http://www.cnbc.com/id/28295763', 'http://www.cnbc.com/pre-markets/', 'http://www.cnbc.com/http://www.cnbc.com/world-markets-heat-map/', 'http://www.cnbc.com/dow-components/', 'http://www.cnbc.com/http://data.cnbc.com/quotes/aapl%2Caxp%2Cba%2Ccat%2Ccsco%2Ccvx%2Cdd%2Cdis%2Cge%2Cgs%2Chd%2Cibm%2Cintc%2Cjnj%2Cjpm%2Cko%2Cmcd%2Cmmm%2Cmrk%2Cmsft%2Cnke%2Cpfe%2Cpg%2Ctrv%2Cunh%2Cutx%2Cv%2Cvz%2Cwmt%2Cxom', 'http://www.cnbc.com/sectors/', 'http://www.cnbc.com/2017/03/27/dollar-limps-away-from-multi-month-lows-still-seen-vulnerable.html', 'http://www.cnbc.com/2017/03/28/le-pen-victory-could-be-five-times-as-dangerous-as-greeces-financial-meltdown-ubs.html', 'http://www.cnbc.com/2017/03/28/iran-says-russia-can-use-its-military-bases-on-case-by-case-basis.html', 'http://www.cnbc.com/2017/03/28/uk-introduces-super-secure-new-one-pound-coin.html', 'http://www.cnbc.com/2017/03/28/google-home-ai-speaker-is-coming-to-the-uk-to-challenge-amazon-echo.html', 'http://www.cnbc.com/2017/03/28/nuclear-nonproliferation-lessons-from-myanmar.html', 'http://www.cnbc.com/2017/03/28/north-korea-could-be-in-final-stages-of-nuclear-test-preparations-report.html', 'http://www.cnbc.com/2017/03/28/hootsuite-ceo-talks-social-media-and-trump.html', 'http://www.cnbc.com/2017/03/28/messaging-app-parent-tencent-takes-stake-in-tesla.html', 'http://www.cnbc.com/2017/03/28/dow-trump-dollar-kushner-stoxx-wallstreet.html', 'http://www.cnbc.com/2017/03/27/trump-brexit-healthcare-russia-dow-stoxx.html', 'http://www.cnbc.com/2017/03/24/trump-healthcare-markets-us-dow-stocks.html', 'http://www.cnbc.com/2017/03/23/trump-london-terrorism-markets-stocks-dow-ftse-may-uk-us.html', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000603723', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000602217', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000600643', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000598102', 'http://www.cnbc.com/2017/03/26/trader-poll-which-will-garner-the-most-market-attention.html', 'http://www.cnbc.com/2017/03/20/trader-poll-what-are-chinas-greatest-challenges.html', 'http://www.cnbc.com/2017/03/12/trader-poll-is-the-fed-definitely-going-to-hike-rates-at-this-meeting.html', 'http://www.cnbc.com/2017/03/06/trader-poll-has-the-us-dollar-got-higher-to-go.html', 'http://www.cnbc.com/2017/03/28/this-ad-agency-is-rebranding-with-the-name-of-its-jewish-founders.html', 'http://www.cnbc.com/advertorial/2017/03/09/breathing-new-life-into-brands.html', 'http://www.cnbc.com/2017/03/23/make-sure-youre-not-replaced-by-a-machine-microsoft-us-chief-marketer.html', 'http://www.cnbc.com/2017/03/23/online-ads-need-to-stop-being-intrusive-eurosport.html', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000595689', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000595687', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000595688', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000595690', 'http://www.cnbc.com/2017/03/28/major-solar-project-gets-the-green-light-in-northern-ireland.html', 'http://www.cnbc.com/2017/03/23/engineers-develop-solar-powered-skin-which-could-transform-lives.html', 'http://www.cnbc.com/2017/03/23/palm-oil-its-uses-and-ways-to-limit-its-impact.html', 'http://www.cnbc.com/advertorial/2017/03/13/access-to-energy-when-business-issues-meet-social-impacts.html', 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ceo-calls-to-step-down.html', 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ai-impact-jobs-mnuchin-government.html', 'http://www.cnbc.com/2017/03/28/square-uk-launch-ceo-jack-dorsey-ai-acquisitions.html', 'http://www.cnbc.com/2017/03/28/amazon-to-acquire-souqcom-a-large-e-commerce-site-in-the-middle-east.html', 'http://www.cnbc.com/advertorial/2016/12/13/a-new-era-of-intelligent-machinery.html', 'http://www.cnbc.com/advertorial/2016/12/05/the-reimagining-of-cities.html', 'http://www.cnbc.com/advertorial/2016/11/22/could-mining-asteroids-save-earth.html', 'http://www.cnbc.com/advertorial/2016/11/02/smarter-ships-safer-seas.html', 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ai-impact-jobs-mnuchin-government.html', 'http://www.cnbc.com/2017/03/28/square-uk-launch-ceo-jack-dorsey-ai-acquisitions.html', 'http://www.cnbc.com/2017/03/28/risk-of-correction-quite-high-but-wont-kill-bull-market-says-goldman-sachs-strategist.html', 'http://www.cnbc.com/2017/03/28/samsung-galaxy-s8-apple-iphone-8-launch.html', 'http://www.cnbc.com/2017/03/28/made-in-america-impossible-tech-startups.html', 'http://www.cnbc.com/2017/03/28/facebook-stole-these-features-from-snapchat.html', 'http://www.cnbc.com/2017/03/28/cisco-separating-some-key-networking-software-from-hardware--report.html', 'http://www.cnbc.com/2017/03/28/uber-releases-diversity-report-and-repudiates-its-hard-charging-attitude.html', 'http://www.cnbc.com/2017/03/28/cramers-charts-explain-why-general-mills-looks-ripe-for-a-run.html', 'http://www.cnbc.com/2017/03/28/cramer-why-it-might-be-time-for-you-to-unfollow-trumps-twitter-feed.html', 'http://www.cnbc.com/2017/03/28/blackrock-plans-major-changes-to-its-active-stockpicking-business.html', 'http://www.cnbc.com/2017/03/28/darden-shares-pop-4-percent-on-strong-earnings-beat-cheddars-acquisition.html', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000605169', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000605171', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000605046', 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000605164', 'http://www.cnbc.com/2017/03/28/nunes-seems-like-the-wrong-person-to-lead-trump-russia-probe-commentary.html', 'http://www.cnbc.com/2017/03/28/donald-trump-is-making-his-russia-scandal-about-clintons-commentary.html', 'http://www.cnbc.com/2017/03/28/self-made-millionaire-shares-3-tricks-for-raising-successful-kids.html', 'http://www.cnbc.com/2017/03/28/president-trump-is-directing-his-anger-at-the-wrong-people-commentary.html']

    To add title also to the result_list, you can modify it something like this :

    import requests 
from lxml import html
import pandas 

url = "http://www.cnbc.com/"
response = requests.get(url)
doc = html.fromstring(response.text)

headlineNode = doc.xpath('//div[@class="headline"]')
len(headlineNode)

result_list  = []
for node in headlineNode : 
    url_node = node.xpath('./a/@href')
    title = node.xpath('./a/text()')
    if (url_node and title) : 
        result_list.append({'url' : url + url_node[0].strip(),
            'title' : title[0].strip()})
print(result_list)
len(result_list)

    This will result in :

    [{'url': 'http://www.cnbc.com/2017/03/28/asia-markets-to-watch-brexit-news-dollar-movements-oil-prices.html', 'title': 'Asia up higher after Dow breaks 8-day losing streak, Brexit ahead'}, {'url': 'http://www.cnbc.com/2017/03/28/us-markets.html', 'title': 'Dow snaps 8-day losing streak as consumer confidence surges; financials lead'}, {'url': 'http://www.cnbc.com/2017/03/28/european-markets-investors-shrug-off-trump-concerns.html', 'title': 'European stocks close higher as investors shrug off Trump concerns; Wolseley up 5%'}, {'url': 'http://www.cnbc.com/2017/03/27/gold-steady-as-dollar-edges-up-focus-on-trump-agenda.html', 'title': 'Gold rises as investors focus on Trump agenda'}, {'url': 'http://www.cnbc.com/2017/03/27/weaker-dollar-lifts-oil-futures-but-soaring-us-output-weighs.html', 'title': 'US crude settles at $48.37, rising 64 cents with Libyan output disruption'}, {'url': 'http://www.cnbc.com/2017/03/28/us-treasurys-edge-lower-as-investors-eye-data-auctions.html', 'title': 'US Treasurys mostly higher as investors digest data, 5-year note auction'}, {'url': 'http://www.cnbc.com/2017/03/28/dollar-bounces-on-fed-talk-sterling-wobbles-on-brexit-anxiety.html', 'title': 'Dollar bounces on Fed talk, sterling wobbles on Brexit anxiety'}, {'url': 'http://www.cnbc.com/2017/03/28/dollar-bounces-on-fed-talk-sterling-wobbles-on-brexit-anxiety.html', 'title': 'Dollar bounces on Fed talk, sterling wobbles on Brexit anxiety'}, {'url': 'http://www.cnbc.com/2017/03/28/markets-could-see-heartbreak-if-brexit-is-hard-to-do.html', 'title': 'Markets could see heartbreak if Brexit is hard to do'}, {'url': 'http://www.cnbc.com/2017/03/28/former-google-exec-heres-how-to-know-its-time-to-fire-an-employee.html', 'title': "Former Google exec: Here's how to know it's time to fire an employee"}, {'url': 'http://www.cnbc.com/2017/03/28/chinas-secret-plan-to-crush-spacex-and-the-us-space-program.html', 'title': "China's secret plan to crush SpaceX and the US space program"}, {'url': 'http://www.cnbc.com/2017/03/28/le-pen-victory-could-be-five-times-as-dangerous-as-greeces-financial-meltdown-ubs.html', 'title': "Le Pen victory could be five times as dangerous as Greece's financial meltdown: UBS"}, {'url': 'http://www.cnbc.com/2017/03/28/nuclear-nonproliferation-lessons-from-myanmar.html', 'title': 'How this tiny country went from nuclear problem child to role model'}, {'url': 'http://www.cnbc.com/2017/03/28/following-tuesdays-rally-stocks-will-grind-higher-into-2018-analyst-says.html', 'title': "Following Tuesday's rally, stocks will 'grind higher' into 2018, analyst says"}, {'url': 'http://www.cnbc.com/2017/03/28/asia-markets-to-watch-brexit-news-dollar-movements-oil-prices.html', 'title': 'Asia up higher after Dow breaks 8-day losing streak, Brexit ahead'}, {'url': 'http://www.cnbc.com/2017/03/28/made-in-america-impossible-tech-startups.html', 'title': "These three start-ups show why it's so hard to build products in America"}, {'url': 'http://www.cnbc.com/2017/03/28/japans-tepid-retail-sales-raise-concerns-about-consumption-growth.html', 'title': "Japan's tepid retail sales raise concerns about consumption, growth"}, {'url': 'http://www.cnbc.com/2017/03/28/toshibas-westinghouse-to-file-for-us-bankruptcy-sources.html', 'title': "Toshiba's Westinghouse to file for US bankruptcy: Sources"}, {'url': 'http://www.cnbc.com/2017/03/28/early-reports-of-damage-in-wake-of-australias-cyclone-debbie.html', 'title': "Early reports of damage in wake of Australia's Cyclone Debbie"}, {'url': 'http://www.cnbc.com/2017/03/28/messaging-app-parent-tencent-takes-stake-in-tesla.html', 'title': 'Tesla deal boosts Chinese presence in US auto tech'}, {'url': 'http://www.cnbc.com/2017/03/28/rodrigo-duterte-turns-72-a-look-at-his-controversial-presidency-so-far-commentary.html', 'title': 'As Rodrigo Duterte turns 72, a look at his controversial presidency so far'}, {'url': 'http://www.cnbc.com/2017/03/28/north-korea-could-be-in-final-stages-of-nuclear-test-preparations-report.html', 'title': 'North Korea could be in final stages of nuclear test preparations: Report'}, {'url': 'http://www.cnbc.com/2017/03/28/congress-clears-way-for-isps-to-sell-browsing-history.html', 'title': 'Congress just cleared the way for internet providers to sell your web browsing history'}, {'url': 'http://www.cnbc.com/2017/03/28/british-pm-may-to-fire-starting-gun-on-brexit.html', 'title': 'British PM May to fire starting gun on Brexit'}, {'url': 'http://www.cnbc.com/2017/03/28/brexit-is-like-arthritis-it-doesnt-kill-you-but-it-hits-you-says-former-boe-member.html', 'title': u'Brexit is like arthritis, it doesn\u2019t kill you but it hits you, says former BOE member'}, {'url': 'http://www.cnbc.com/2017/03/27/florida-preparesfor-xi-jinpingsvisit.html', 'title': u"Florida authorities prep\xa0for Xi Jinping's\xa0visit"}, {'url': 'http://www.cnbc.com/2017/03/28/us-heading-for-recession-after-2-years-of-unsustainable-growth-economist-says.html', 'title': 'US heading for recession after 2 years of unsustainable growth, economist says'}, {'url': 'http://www.cnbc.com/2017/03/28/this-is-what-immigration-means-to-the-us-economy-in-two-charts.html', 'title': 'This is what immigration means to the U.S. economy in two charts'}, {'url': 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ceo-calls-to-step-down.html', 'title': u'Jack Dorsey brushes off calls to quit as CEO, says will do \u2018whatever it takes\u2019 for Twitter, Square to succeed'}, {'url': 'http://www.cnbc.com/2017/03/28/next-iphone-is-last-hurrah-for-smartphones-apple.html', 'title': 'The next iPhone is the last hurrah for smartphones: Fund manager'}, {'url': 'http://www.cnbc.com/2017/03/28/south-africa-speculates-over-possible-firing-of-finance-minister.html', 'title': 'South Africa faces uncertain wait amid speculation finance minister is to be fired'}, {'url': 'http://www.cnbc.com/2017/03/28/snap-was-a-drunk-at-the-wheel-but-now-looks-like-a-very-real-business-fund-manager-says.html', 'title': "Snap was a 'drunk at the wheel' but now looks like a very real business, fund manager says"}, {'url': 'http://www.cnbc.com/2017/03/28/this-ad-agency-is-rebranding-with-the-name-of-its-jewish-founders.html', 'title': 'This ad agency is standing against xenophobia and rebranding with the name of its Jewish founders'}, {'url': 'http://www.cnbc.com/2017/03/28/siemens-medical-solutions-wins-41-billion-us-defense-contract.html', 'title': 'Siemens Medical Solutions wins $4.1 billion US defense contract'}, {'url': 'http://www.cnbc.com/2017/03/28/google-home-ai-speaker-is-coming-to-the-uk-to-challenge-amazon-echo.html', 'title': 'Google Home A.I. speaker is coming to the UK to challenge Amazon Echo'}, {'url': 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ai-impact-jobs-mnuchin-government.html', 'title': u'Is the US government ready for the impact of AI? \u2018I don\u2019t know,\u2019 Twitter CEO Jack Dorsey says'}, {'url': 'http://www.cnbc.com/2017/03/28/uk-introduces-super-secure-new-one-pound-coin.html', 'title': u'UK introduces super secure new \xa31 coin'}, {'url': 'http://www.cnbc.com/2017/03/28/major-solar-project-gets-the-green-light-in-northern-ireland.html', 'title': "'Major' solar project gets the green light in Northern Ireland"}, {'url': 'http://www.cnbc.com/2017/03/28/trump-to-meet-egyptian-president-at-white-house-on-april-3.html', 'title': 'Trump to meet Egyptian president at White House on April 3'}, {'url': 'http://www.cnbc.com/2017/03/28/tesla-is-rejecting-a-racial-harassment-lawsuit-filed-by-an-assembly-line-worker.html', 'title': 'Tesla rejects racial harassment lawsuit filed by assembly line worker'}, {'url': 'http://www.cnbc.com/2017/03/28/jpmorgan-says-markets-are-underestimating-the-upside-of-asian-banks.html', 'title': 'JPMorgan says markets are underestimating the upside of Asian banks'}, {'url': 'http://www.cnbc.com/https://www.facebook.com/cnbcinternational/', 'title': 'Facebook'}, {'url': 'http://www.cnbc.com/https://twitter.com/cnbcworld', 'title': 'Twitter'}, {'url': 'http://www.cnbc.com/http://www.linkedin.com/company/2966126', 'title': 'LinkedIn'}, {'url': 'http://www.cnbc.com/http://www.youtube.com/user/CNBCInternational', 'title': 'Youtube'}, {'url': 'http://www.cnbc.com/http://www.cnbc.com/id/28295763', 'title': 'RSS'}, {'url': 'http://www.cnbc.com/pre-markets/', 'title': 'Pre-markets'}, {'url': 'http://www.cnbc.com/http://www.cnbc.com/world-markets-heat-map/', 'title': 'World stock markets heat map'}, {'url': 'http://www.cnbc.com/dow-components/', 'title': 'Dow 30: Real-time quotes'}, {'url': 'http://www.cnbc.com/http://data.cnbc.com/quotes/aapl%2Caxp%2Cba%2Ccat%2Ccsco%2Ccvx%2Cdd%2Cdis%2Cge%2Cgs%2Chd%2Cibm%2Cintc%2Cjnj%2Cjpm%2Cko%2Cmcd%2Cmmm%2Cmrk%2Cmsft%2Cnke%2Cpfe%2Cpg%2Ctrv%2Cunh%2Cutx%2Cv%2Cvz%2Cwmt%2Cxom', 'title': 'Dow 30: After-hours quotes'}, {'url': 'http://www.cnbc.com/sectors/', 'title': 'Sector watch'}, {'url': 'http://www.cnbc.com/2017/03/28/dollar-bounces-on-fed-talk-sterling-wobbles-on-brexit-anxiety.html', 'title': 'Dollar bounces on Fed talk, sterling wobbles on Brexit anxiety'}, {'url': 'http://www.cnbc.com/2017/03/28/le-pen-victory-could-be-five-times-as-dangerous-as-greeces-financial-meltdown-ubs.html', 'title': "Le Pen victory could be five times as perilous as Greece's financial crisis: UBS"}, {'url': 'http://www.cnbc.com/2017/03/28/iran-says-russia-can-use-its-military-bases-on-case-by-case-basis.html', 'title': "Iran says Russia can use its military bases 'on case by case basis'"}, {'url': 'http://www.cnbc.com/2017/03/28/uk-introduces-super-secure-new-one-pound-coin.html', 'title': u'UK introduces super secure new \xa31 coin'}, {'url': 'http://www.cnbc.com/2017/03/28/google-home-ai-speaker-is-coming-to-the-uk-to-challenge-amazon-echo.html', 'title': 'Google Home A.I. speaker is coming to the UK to challenge Amazon Echo'}, {'url': 'http://www.cnbc.com/2017/03/28/nuclear-nonproliferation-lessons-from-myanmar.html', 'title': 'Nuclear nonproliferation: Lessons from Myanmar'}, {'url': 'http://www.cnbc.com/2017/03/28/north-korea-could-be-in-final-stages-of-nuclear-test-preparations-report.html', 'title': 'North Korea could be in final stages of nuclear test preparations: Report'}, {'url': 'http://www.cnbc.com/2017/03/28/hootsuite-ceo-talks-social-media-and-trump.html', 'title': 'Companies need top-down approach to social media: Hootsuite CEO'}, {'url': 'http://www.cnbc.com/2017/03/28/messaging-app-parent-tencent-takes-stake-in-tesla.html', 'title': 'Tesla deal boosts Chinese presence in US auto tech'}, {'url': 'http://www.cnbc.com/2017/03/28/dow-trump-dollar-kushner-stoxx-wallstreet.html', 'title': 'Reaction: Global markets stepped past Trump wobble on Obamacare'}, {'url': 'http://www.cnbc.com/2017/03/27/trump-brexit-healthcare-russia-dow-stoxx.html', 'title': 'Reaction: Risk-off hits markets after Trump fails to pass Obamacare hurdle'}, {'url': 'http://www.cnbc.com/2017/03/24/trump-healthcare-markets-us-dow-stocks.html', 'title': 'Reaction: Wall Street stocks move higher as investors countdown to health-care vote'}, {'url': 'http://www.cnbc.com/2017/03/23/trump-london-terrorism-markets-stocks-dow-ftse-may-uk-us.html', 'title': 'Reaction: London police name attacker in Westminster terrorist incident'}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000603723', 'title': "Weekly Agenda: From EU visas to 'Fearless Girl'"}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000602217', 'title': 'Weekly Agenda: From the Canadian budget to the Boao Forum'}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000600643', 'title': "Weekly Agenda: From Merkel's US visit to the FOMC meeting"}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000598102', 'title': "Weekly Agenda: From 'mini' EU summit to US daylight saving"}, {'url': 'http://www.cnbc.com/2017/03/26/trader-poll-which-will-garner-the-most-market-attention.html', 'title': 'Trader poll: Which will garner the most market attention?'}, {'url': 'http://www.cnbc.com/2017/03/20/trader-poll-what-are-chinas-greatest-challenges.html', 'title': u'Trader poll: What are China\u2019s greatest challenges?'}, {'url': 'http://www.cnbc.com/2017/03/12/trader-poll-is-the-fed-definitely-going-to-hike-rates-at-this-meeting.html', 'title': 'Trader Poll: Is the Fed definitely going to hike rates at this meeting?'}, {'url': 'http://www.cnbc.com/2017/03/06/trader-poll-has-the-us-dollar-got-higher-to-go.html', 'title': 'Trader poll: Has the US dollar got higher to go?'}, {'url': 'http://www.cnbc.com/2017/03/28/this-ad-agency-is-rebranding-with-the-name-of-its-jewish-founders.html', 'title': 'This ad agency is standing against xenophobia; using name of its Jewish founders'}, {'url': 'http://www.cnbc.com/advertorial/2017/03/09/breathing-new-life-into-brands.html', 'title': 'Breathing new life into brands'}, {'url': 'http://www.cnbc.com/2017/03/23/make-sure-youre-not-replaced-by-a-machine-microsoft-us-chief-marketer.html', 'title': "Make sure you're not replaced by a machine: Microsoft US chief marketer"}, {'url': 'http://www.cnbc.com/2017/03/23/online-ads-need-to-stop-being-intrusive-eurosport.html', 'title': 'Online ads need to stop being intrusive, especially around 2018 Olympics: Eurosport MD'}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000595689', 'title': 'Why naming a product is so important'}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000595687', 'title': u'\u2018Building up a sofa\u2019 that\u2019s fits customer needs: Matthew\u2019s brand proposal'}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000595688', 'title': u'A men\u2019s \u2018grooming brand with an organic twist\u2019: Connor\u2019s proposal'}, {'url': 'http://www.cnbc.com/http://video.cnbc.com/gallery/?video=3000595690', 'title': 'Why passion is the true driver of a successful entrepreneur'}, {'url': 'http://www.cnbc.com/2017/03/28/major-solar-project-gets-the-green-light-in-northern-ireland.html', 'title': "'Major' solar project gets the green light in Northern Ireland"}, {'url': 'http://www.cnbc.com/2017/03/23/engineers-develop-solar-powered-skin-which-could-transform-lives.html', 'title': "Engineers develop solar powered 'skin' which could transform lives"}, {'url': 'http://www.cnbc.com/2017/03/23/palm-oil-its-uses-and-ways-to-limit-its-impact.html', 'title': 'Palm oil: Its uses and ways to limit its impact'}, {'url': 'http://www.cnbc.com/advertorial/2017/03/13/access-to-energy-when-business-issues-meet-social-impacts.html', 'title': 'Access to energy: When business issues meet social impacts'}, {'url': 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ceo-calls-to-step-down.html', 'title': u"Jack Dorsey: I'll do \u2018whatever it takes\u2019 for Twitter, Square to succeed"}, {'url': 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ai-impact-jobs-mnuchin-government.html', 'title': u'Is the US government ready for the impact of AI? \u2018I don\u2019t know,\u2019 Twitter CEO Jack Dorsey says'}, {'url': 'http://www.cnbc.com/2017/03/28/square-uk-launch-ceo-jack-dorsey-ai-acquisitions.html', 'title': 'Square launches in UK, CEO Jack Dorsey says on the lookout for AI firms to acquire'}, {'url': 'http://www.cnbc.com/2017/03/28/amazon-to-acquire-souqcom-a-large-e-commerce-site-in-the-middle-east.html', 'title': 'Amazon reaches deal to acquire Middle East e-commerce site Souq.com'}, {'url': 'http://www.cnbc.com/advertorial/2016/12/13/a-new-era-of-intelligent-machinery.html', 'title': 'A new era of intelligent machinery'}, {'url': 'http://www.cnbc.com/advertorial/2016/12/05/the-reimagining-of-cities.html', 'title': 'The reimagining of cities'}, {'url': 'http://www.cnbc.com/advertorial/2016/11/22/could-mining-asteroids-save-earth.html', 'title': 'Could mining asteroids save Earth?'}, {'url': 'http://www.cnbc.com/advertorial/2016/11/02/smarter-ships-safer-seas.html', 'title': 'At sea, every hour is rush hour. Are autonomous ships the solution?'}, {'url': 'http://www.cnbc.com/2017/03/28/jack-dorsey-twitter-ai-impact-jobs-mnuchin-government.html', 'title': u'Is the US government ready for the impact of AI? \u2018I don\u2019t know,\u2019 Twitter CEO says'}]

    You can refer to documentation of scraping and lxml for more understanding.