Google news crawler to return results with url,title and briefing

Question

I am new to crawler and I am using Python 3.X. Currently I am practicing to crawl google news for fresh start but I have encounter some problem with my code(the code runs but did not return anything). I want the code to crawl google news for query and return results with url, title and briefing appear in results.

Many thanks for your time. my code is below:

import sys
import urllib
import requests
from bs4 import BeautifulSoup
import time

s = "Stack Overflow"
url = "http://www.google.com.sg/search?q="+s+"&tbm=nws&tbs=qdr:y"
#htmlpage = urllib2.urlopen(url).read()
time.sleep(randint(0, 2))
htmlpage = requests.get(url)
soup = BeautifulSoup(htmlpage.text,'lxml')
#print (len(soup.findAll("table", {"class": "result"})))
for result_table in soup.findAll("table", {"class": "result"}):
    a_click = result_table.find("a")
    print ("-----Title----\n" + a_click.renderContents())#Title
    print ("----URL----\n" + str(a_click.get("href")))#URL
    print ("----Brief----\n" + result_table.find("div", {"class": "c-abstract"}).renderContents())#Brief
    print ("Done")

Answer 1

This is how i got results, hope it helps:

>>> for result_table in soup.findAll("div", {"class": "g"}):
...     a_click = result_table.find("a")
...     print ("-----Title----\n" + str(a_click.renderContents()))#Title
...     print ("----URL----\n" + str(a_click.get("href")))#URL
...     print ("----Brief----\n" + str(result_table.find("div", {"class": "st"}).renderContents()))#Brief
...     print ("Done")
... 
-----Title----
b"<b>Stack Overflow</b>: Like sleep? Don't code in C"
----URL----
/url?q=http://www.infoworld.com/article/3190701/application-development/stack-overflow-like-sleep-dont-code-in-c.html&sa=U&ved=0ahUKEwjc34W_3NLTAhVIMY8KHVu_BoUQqQIIFigAMAA&usg=AFQjCNE7xDqkg-kyFR65krfMIJqIchHFwg
----Brief----
b'In analysis of programming traffic on the <b>Stack Overflow</b> online community over for four weeks last August, <b>Stack Overflow</b> Insights data scientist David Robinson,\xc2\xa0...'
Done