65,305
stars
+18
12,113
forks
1,743
watchers
5,032
prs
3,323
issues
55
releases
25
discussions
README
|logo|
.. |logo| image:: https://raw.githubusercontent.com/scrapy/scrapy/master/docs/_static/logo.svg
:target: https://scrapy.org
:alt: Scrapy
:width: 480px
|version| |python_version| |tests| |coverage| |conda| |deepwiki|
.. |version| image:: https://img.shields.io/pypi/v/Scrapy.svg
:target: https://pypi.org/pypi/Scrapy
:alt: PyPI Version
.. |python_version| image:: https://img.shields.io/pypi/pyversions/Scrapy.svg
:target: https://pypi.org/pypi/Scrapy
:alt: Supported Python Versions
.. |tests| image:: https://img.shields.io/github/check-runs/scrapy/scrapy/master?label=tests
:target: https://github.com/scrapy/scrapy/actions?query=branch%3Amaster
:alt: Tests
.. |coverage| image:: https://img.shields.io/codecov/c/github/scrapy/scrapy/master.svg
:target: https://codecov.io/github/scrapy/scrapy?branch=master
:alt: Coverage report
.. |conda| image:: https://anaconda.org/conda-forge/scrapy/badges/version.svg
:target: https://anaconda.org/conda-forge/scrapy
:alt: Conda Version
.. |deepwiki| image:: https://deepwiki.com/badge.svg
:target: https://deepwiki.com/scrapy/scrapy
:alt: Ask DeepWiki
Scrapy_ is a web scraping framework to extract structured data from websites.
It is cross-platform, and requires Python 3.11+. It is maintained by Zyte_
(formerly Scrapinghub) and many other contributors_.
.. _many other contributors: https://github.com/scrapy/scrapy/graphs/contributors
.. _Scrapy: https://scrapy.org/
.. _Zyte: https://www.zyte.com/
Install with:
.. code:: bash
pip install scrapy
And follow the documentation_ to learn how to use it.
.. _documentation: https://docs.scrapy.org/en/latest/
If you wish to contribute, see Contributing_.
.. _Contributing: https://docs.scrapy.org/en/master/contributing.htmlTop issues
#1306Speedup & fix URL parsing39 comments · 2015-06-15#6597Scrapy needs to have sane defaults that do no harm32 comments · 2025-01-01#4240Wrong type(response) for binary responses28 comments · 2019-12-17#1144Awaiting on Requests in callbacks directly25 comments · 2015-04-10#356Support list type arguments from command line.24 comments · 2013-07-24#1988"Content-Encoding" header gets stripped from response headers20 comments · 2016-05-12#1225dupefilter skips a request when a page is redirected to itself 20 comments · 2015-05-12#713Bring docstrings back17 comments · 2014-05-07
Discussions
#8368PR automatically flagged as spam, looking for maintainer help1 comments · 2026-10-17#7582Choosing Issue1 comments · 2026-06-08#7362Newbie question about scraping3 comments · 2026-03-25#7343How to Automatically Retrieve Web Resources That Require On-Site Search to Locate, Instead of Just Recursively Crawling URLs?2 comments · 2026-03-21#7326Scrapy doesn't seem to understand what 403 and 444 mean.3 comments · 2026-03-11#7325Providing Spider specific settings to custom DownloadHandlers1 comments · 2026-03-11#7322License and Attribution of the Scrapy Logo?2 comments · 2026-03-09#7320Localization of The Scrapy Documentation0 comments · 2026-03-09
Latest release
2.20.0 2.20.0 2026-11-06- Web-browser-like detection of response encodings, base URLs and meta refresh redirects, powered by frostwork - Requests can be awaited to get their response - Non-asyncio Twisted reactors are deprecated Full changelog