README

Newspaper3k: Article scraping & curation
========================================

.. image:: https://badge.fury.io/py/newspaper3k.svg
    :target: http://badge.fury.io/py/newspaper3k.svg
        :alt: Latest version

.. image:: https://travis-ci.org/codelucas/newspaper.svg
        :target: http://travis-ci.org/codelucas/newspaper/
        :alt: Build status

.. image:: https://coveralls.io/repos/github/codelucas/newspaper/badge.svg?branch=master
        :target: https://coveralls.io/github/codelucas/newspaper
        :alt: Coverage status

Inspired by requests_ for its simplicity and powered by lxml_ for its speed:

    "Newspaper is an amazing python library for extracting & curating articles."
    -- tweeted by_ Kenneth Reitz, Author of requests_

    "Newspaper delivers Instapaper style article extraction." -- The Changelog_

.. _tweeted by: https://twitter.com/kennethreitz/status/419520678862548992
.. _The Changelog: http://thechangelog.com/newspaper-delivers-instapaper-style-article-extraction/

Newspaper is a Python3 library! Or, view our deprecated and buggy Python2 branch_

.. _Python2 branch: https://github.com/codelucas/newspaper/tree/python-2-head

A Glance:
---------

.. code-block:: pycon

     from newspaper import Article

     url = 'http://fox13now.com/2013/12/30/new-year-new-laws-obamacare-pot-guns-and-drones/'
     article = Article(url)

.. code-block:: pycon

     article.download()

     article.html
    ' article.parse()

     article.authors
    ['Leigh Ann Caldwell', 'John Honway']

     article.publish_date
    datetime.datetime(2013, 12, 30, 0, 0)

     article.text
    'Washington (CNN) -- Not everyone subscribes to a New Year's resolution...'

     article.top_image
    'http://someCDN.com/blah/blah/blah/file.png'

     article.movies
    ['http://youtube.com/path/to/link.com', ...]

.. code-block:: pycon

     article.nlp()

     article.keywords
    ['New Years', 'resolution', ...]

     article.summary
    'The study shows that 93% of people ...'

.. code-block:: pycon

     import newspaper

     cnn_paper = newspaper.build('http://cnn.com')

     for article in cnn_paper.articles:
         print(article.url)
    http://www.cnn.com/2013/11/27/justice/tucson-arizona-captive-girls/
    http://www.cnn.com/2013/12/11/us/texas-teen-dwi-wreck/index.html
    ...

     for category in cnn_paper.category_urls():
         print(category)

    http://lifestyle.cnn.com
    http://cnn.com/world
    http://tech.cnn.com
    ...

     cnn_article = cnn_paper.articles[0]
     cnn_article.download()
     cnn_article.parse()
     cnn_article.nlp()
    ...

.. code-block:: pycon

     from newspaper import fulltext

     html = requests.get(...).text
     text = fulltext(html)


Newspaper can extract and detect languages seamlessly.
If no language is specified, Newspaper will attempt to auto detect a language.

.. code-block:: pycon

     from newspaper import Article
     url = 'http://www.bbc.co.uk/zhongwen/simp/chinese_news/2012/