README

Requests-HTML: HTML Parsing for Humans™
=======================================

.. image:: https://farm5.staticflickr.com/4695/39152770914_a3ab8af40d_k_d.jpg

.. image:: https://travis-ci.com/psf/requests-html.svg?branch=master
    :target: https://travis-ci.com/psf/requests-html

This library intends to make parsing HTML (e.g. scraping the web) as
simple and intuitive as possible.

When using this library you automatically get:

- Full JavaScript support! (Using Chromium, thanks to pyppeteer)
- CSS Selectors (a.k.a jQuery-style, thanks to PyQuery).
- XPath Selectors, for the faint of heart.
- Mocked user-agent (like a real web browser).
- Automatic following of redirects.
- Connection–pooling and cookie persistence.
- The Requests experience you know and love, with magical parsing abilities.
- Async Support

.. Other nice features include:

    - Markdown export of pages and elements.


Tutorial & Usage
================

Make a GET request to 'python.org', using Requests:

.. code-block:: pycon

     from requests_html import HTMLSession
     session = HTMLSession()
     r = session.get('https://python.org/')

Try async and get some sites at the same time:

.. code-block:: pycon

     from requests_html import AsyncHTMLSession
     asession = AsyncHTMLSession()
     async def get_pythonorg():
    ...     r = await asession.get('https://python.org/')
    ...     return r
    ...
     async def get_reddit():
    ...    r = await asession.get('https://reddit.com/')
    ...    return r
    ...
     async def get_google():
    ...    r = await asession.get('https://google.com/')
    ...    return r
    ...
     results = asession.run(get_pythonorg, get_reddit, get_google)
     results  check the requests all returned a 200 (success) code
    [, , ]
      Each item in the results list is a response object and can be interacted with as such
     for result in results: 
    ...     print(result.html.url)
    ... 
    https://www.python.org/
    https://www.google.com/
    https://www.reddit.com/

Note that the order of the objects in the results list represents the order they were returned in, not the order that the coroutines are passed to the run method, which is shown in the example by the order being different. 

Grab a list of all links on the page, as–is (anchors excluded):

.. code-block:: pycon

     r.html.links
    {'//docs.python.org/3/tutorial/', '/about/apps/', 'https://github.com/python/pythondotorg/issues', '/accounts/login/', '/dev/peps/', '/about/legal/', '//docs.python.org/3/tutorial/introduction.htmllists', '/download/alternatives', 'http://feedproxy.google.com/~r/PythonInsider/~3/kihd2DW98YY/python-370a4-is-available-for-testing.html', '/download/other/', '/downloads/windows/', 'https://mail.python.org/mailman/listinfo/python-dev', '/doc/av', 'https://devguide.python.org/', '/about/success/engineering', 'https://wiki.python.org/moin/PythonEventsCalendarSubmitting_an_Event', 'https://www.openstack.org', '/about/gettingstarted/', 'htt