Search code examples
pythonpython-newspaper

Python: Newspaper Module - Downloading from multiple URLs


I hate to start a new post but I am trying to accomplish the exact thing described in this question: Python: Newspaper Module - Any way to pool getting articles straight from URLs?

In attempting to implement the solution, though, I am getting the following error:

NameError   Traceback (most recent call last)
<ipython-input-38-2707f1416873> in <module>()
----> 1 sources = [SingleSource(articleURL=u) for u in urls]
      2 
      3 newspaper.news_pool.set(sources)
      4 newspaper.news_pool.join()
      5 

<ipython-input-38-2707f1416873> in <listcomp>(.0)
----> 1 sources = [SingleSource(articleURL=u) for u in urls]
      2 
      3 newspaper.news_pool.set(sources)
      4 newspaper.news_pool.join()
      5 

<ipython-input-37-4949a9e51da5> in __init__(self, articleURL)
      1 class SingleSource(newspaper.Source):
      2     def __init__(self, articleURL):
----> 3         super(StubSource, self).__init__("http://localhost")
      4         self.articles = [newspaper.Article(url=url)]

NameError: name 'StubSource' is not defined

Would very much appreciate a push in the right direction.


Solution

  • Looks like a typo in the linked answer as StubSource is not defined

    class SingleSource(newspaper.Source):
        def __init__(self, articleURL):
            super(StubSource, self).__init__("http://localhost")
            self.articles = [newspaper.Article(url=url)]
    

    It should probably be:

    class SingleSource(newspaper.Source):
        def __init__(self, articleURL):
            super(SingleSource, self).__init__("http://localhost")
            self.articles = [newspaper.Article(url=url)]