{"full_name":"D4Vinci/Scrapling","owner":"D4Vinci","name":"Scrapling","url":"https://github.com/D4Vinci/Scrapling","description":"🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev","homepage":"https://scrapling.readthedocs.io/en/latest/","topics":[],"language":"Python","license":"BSD-3-Clause","avatar":"https://avatars.githubusercontent.com/u/20604835?u=09b7c5077ccfe08e9d863f07bfa60fa0168d0061&v=4","created_at":"2024-10-13T20:29:53Z","pushed_at":"2026-11-09T10:36:37Z","metrics":{"stars":100817,"forks":10527,"watchers":348,"issues":183,"prs":316,"releases":55,"issues_open":null,"issues_closed":null,"prs_open":null,"prs_closed":null,"prs_merged":null,"discussions":52},"growth":{"day":{"stars":655,"forks":61,"base_day":"2026-11-09"},"week":{"stars":8212,"forks":770,"base_day":"2026-11-03"},"month":{"stars":14146,"forks":1648,"base_day":"2026-10-11"}},"history":{"dates":["2026-09-06","2026-09-07","2026-09-08","2026-09-09","2026-09-10","2026-09-11","2026-09-12","2026-09-13","2026-09-14","2026-09-15","2026-09-16","2026-09-17","2026-09-18","2026-09-19","2026-09-20","2026-09-21","2026-09-22","2026-09-23","2026-09-24","2026-09-25","2026-09-26","2026-09-27","2026-09-28","2026-09-29","2026-09-30","2026-10-01","2026-10-02","2026-10-03","2026-10-04","2026-10-05","2026-10-06","2026-10-07","2026-10-08","2026-10-09","2026-10-10","2026-10-11","2026-10-12","2026-10-13","2026-10-14","2026-10-15","2026-10-16","2026-10-17","2026-10-18","2026-10-19","2026-10-20","2026-10-21","2026-10-22","2026-10-23","2026-10-24","2026-10-25","2026-10-26","2026-10-27","2026-10-28","2026-10-29","2026-10-30","2026-10-31","2026-11-01","2026-11-02","2026-11-03","2026-11-04","2026-11-09","2026-11-10"],"stars":[78544,78704,79083,79382,79651,79968,80277,80495,80712,80928,81144,81423,81792,82119,82385,82605,82808,82963,83161,83410,83671,83864,84039,84300,84511,84677,84982,85197,85435,85656,85824,85966,86146,86354,86507,86671,86826,86984,87306,87558,87868,88088,88274,88457,88651,88851,89022,89179,89332,89459,89681,89863,90039,90192,90309,90461,90810,91692,92605,94393,100162,100817],"forks":[7906,7932,7960,7979,8013,8057,8103,8125,8146,8170,8202,8253,8307,8358,8394,8416,8433,8444,8483,8522,8550,8565,8596,8619,8641,8657,8699,8727,8751,8772,8792,8803,8828,8854,8865,8879,8890,8910,8951,8994,9028,9065,9089,9123,9146,9168,9182,9195,9215,9230,9265,9289,9309,9323,9349,9370,9458,9652,9757,9900,10466,10527],"watchers":[286,287,286,289,289,290,294,295,297,296,297,297,299,299,299,301,301,301,302,307,308,310,309,311,312,313,315,315,316,317,319,318,319,319,320,320,321,322,319,319,318,320,321,322,323,324,324,324,324,320,321,321,322,322,320,321,320,325,328,329,348,348],"issues_open":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],"prs_open":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null]},"boards":{"daily":52,"weekly":9,"monthly":38,"weekly_rate":44,"monthly_rate":105,"total":141},"same_owner":[],"issues":[{"number":563,"title":"<base href> ignored when resolving relative URLs","comments":1,"created_at":"2026-11-05T15:10:10Z"},{"number":558,"title":"Response.follow() inherits credentials when following links to a different origin","comments":0,"created_at":"2026-11-05T02:25:07Z"},{"number":557,"title":"cannot access online sites like coupang.","comments":0,"created_at":"2026-11-04T23:10:53Z"}],"discussions":[{"number":566,"title":"When the model card does not say: Scrapling + quote-checked field extraction","comments":0,"category":"Show and tell","created_at":"2026-11-06T02:05:16Z"},{"number":549,"title":"docker pull ghcr.io/d4vinci/scrapling:latest","comments":1,"category":"General","created_at":"2026-11-03T13:08:09Z"},{"number":544,"title":"ai cloud gpt Facebook TikTok telegram","comments":1,"category":"Ideas","created_at":"2026-11-02T17:56:47Z"},{"number":532,"title":"Add link11 waf bypass","comments":0,"category":"Ideas","created_at":"2026-10-23T20:35:53Z"},{"number":512,"title":"Showcase: ATS Zero-Browser Gateway — Protocol-level scraping for Workday & ATS boards","comments":0,"category":"Show and tell","created_at":"2026-10-15T20:42:32Z"},{"number":479,"title":"Free, open-source Chromium with native fingerprint spoofing, usable via executable_path (from the AI agent that runs on it)","comments":1,"category":"Show and tell","created_at":"2026-10-05T19:11:18Z"},{"number":449,"title":"similarity-based element relocation","comments":1,"category":"General","created_at":"2026-09-14T18:37:23Z"},{"number":369,"title":"Freshness at the fact level — verify an extracted value is still current before an agent acts","comments":0,"category":"Ideas","created_at":"2026-07-06T23:59:33Z"}],"latest_release":{"name":"Release v0.5.1","tag":"v0.5.1","published_at":"2026-11-04T20:29:16Z","body":"**Small version bump, big browser upgrade! This version lets AI agents follow motion and control drag and drop, adds dedicated remote browser sessions, and brings fixes across parsing, spiders, proxies, and the shell 🚀**\r\n\r\n**With this update, we extend the limits of current AI models** 🔥 \r\n\r\n> [!NOTE]\r\n> **[Follow us on X for daily tips and tricks](https://x.com/Scrapling_dev)**\r\n\r\n> [!WARNING]\r\n> **This release introduces breaking changes to remote browser connections. Connecting to remote browsers now requires a new browser session class. Check the [remote-session migration guide](https://scrapling.readthedocs.io/en/latest/migrating-to-v0.5.html#move-cdp-connections-to-remote-sessions) before updating.**\r\n\r\n## 🚀 New Stuff and quality of life changes\r\n\r\n- **AI agents can now follow motion and take finer control of the mouse** (Check the [docs](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html)):\r\n\r\n    - **Watch the page in motion**: record a short video and share it as a sequence of frames with timestamps to help the agent understand animations, moving controls, and changes that a single screenshot can miss. Pair this with the existing ability to chain actions!\r\n    - **Drag, inspect, adjust, and drop**: hold a mouse button, move an item or slider, inspect the page while still holding it, then adjust and release. Supports left, right, and middle buttons.\r\n    - **Reuse a logged-in remote browser** or start with a separate browser context, so the agent can choose the right session for the job.\r\n    - **Spend fewer tokens on tool descriptions** with shorter text and less repeated guidance.\r\n\r\n- **Dedicated remote browser sessions with control over login reuse**. `RemoteSession` and `AsyncRemoteSession` connect to an existing Chromium browser through an HTTP or WebSocket CDP endpoint. They support the usual stealth session features, including Cloudflare solving. Both classes are also available in the interactive shell. (Check the [docs](https://scrapling.readthedocs.io/en/latest/fetching/stealthy.html#3-cdp-connection))\r\n\r\n    ```python\r\n    from scrapling.fetchers import RemoteSession\r\n\r\n    with RemoteSession(cdp_url=\"http://localhost:9222\", new_context=False) as session:\r\n        page = session.fetch(\"https://example.com\")\r\n    ```\r\n\r\n    - `new_context=False` reuses the browser's existing default context and login state. Its context settings stay in use, and its existing tabs remain open when Scrapling disconnects.\r\n    - The default, `new_context=True`, creates a separate context and closes it when the session ends. Scrapling uses its own tabs in either mode.\r\n\r\n- **Choose your spider's log file encoding** with `log_file_encoding=\"utf-8\"` or another encoding. The default, `None`, keeps Python's default encoding. (Check the [docs](https://scrapling.readthedocs.io/en/latest/spiders/advanced.html#logging))\r\n\r\n## 🐛 Bug Fixes\r\n\r\n- **XPath expressions now return their values correctly**, including `string()`, `count()`, and `boolean()`. Empty strings, zero, and false remain available through the usual selector API, by @jayhemnani9910 in [#552](https://github.com/D4Vinci/Scrapling/pull/552).\r\n- **Fixed parsing of HTML with no root element**, including whitespace-only, comment-only, and doctype-only input, by @rushilnimbalker250 in [#536](https://github.com/D4Vinci/Scrapling/pull/536).\r\n- **Fixed `find_similar(similarity_threshold=0)` when the reference has no attributes to compare**, by @Master-Norna in [#539](https://github.com/D4Vinci/Scrapling/pull/539).\r\n\r\n- **Fixed duplicate robots.txt fetches when concurrent requests reach the same domain**, by @RayenMlayeh in [#538](https://github.com/D4Vinci/Scrapling/pull/538).\r\n- **Fixed temporary-file collisions during concurrent development-cache writes**, by @xixihaha-ha in [#546](https://github.com/D4Vinci/Scrapling/pull/546).\r\n- **Malformed links no longer stop extraction of valid links on the same page**, by @dajiaohuang in [#548](https://github.com/D4Vinci/Scrapling/"},"readme":true,"readme_zh":false,"news":[{"title":"Scrapling: Fast, Adaptive Web Scraping for Python","url":"https://github.com/D4Vinci/Scrapling","hn":"https://news.ycombinator.com/item?id=41832425","source":"Hacker News","lang":"en","date":"2024-10-13","points":4,"comments":1},{"title":"Scrapling v0.2.99 – Easy, effortless Web Scraping With Python as it should be","url":"https://github.com/D4Vinci/Scrapling","hn":"https://news.ycombinator.com/item?id=43852100","source":"Hacker News","lang":"en","date":"2025-04-30","points":1,"comments":0},{"title":"爬虫改版就废？8.2 万星的 Scrapling 自己把元素找回来了","url":"https://www.bilibili.com/video/BV1Pyhj6KEPk/","source":"bilibili","lang":"zh","date":"2026-09-21"},{"title":"Scrapling 保姆级教程来了，零基础入门！狂揽 29.9 万 Star 的爬虫神器","url":"https://blog.csdn.net/chenchuang0128/article/details/162428493","source":"blog.csdn","lang":"zh","date":"2026-06-29"},{"title":"GitHub 开源项目解析：D4Vinci/Scrapling —— Python 网页抓取与自动化处理工具","url":"https://blog.csdn.net/Rthan/article/details/161626274","source":"blog.csdn","lang":"zh","date":"2026-06-02"},{"title":"Scrapling成OpenClaw“神助攻”：轻松绕拦截，网页抓取稳又省!","url":"https://www.msn.com/zh-cn/news/other/scrapling%E6%88%90openclaw-%E7%A5%9E%E5%8A%A9%E6%94%BB-%E8%BD%BB%E6%9D%BE%E7%BB%95%E6%8B%A6%E6%88%AA-%E7%BD%91%E9%A1%B5%E6%8A%93%E5%8F%96%E7%A8%B3%E5%8F%88%E7%9C%81/ar-AA1XL0fU","source":"MSN","lang":"zh","date":"2026-03-12"},{"title":"Clandestine on X: \"GitHub - D4Vinci/Scrapling: ️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! · GitHub https ...","url":"https://x.com/akaclandestine/status/2029484921912213891","source":"x","lang":"en","date":"2026-03-05"},{"title":"「Scrapling」でボット対策を回避？ OpenClawユーザーの間で拡散するツール","url":"https://wired.jp/article/openclaw-users-bypass-anti-bot-systems-cloudflare-scrapling/","source":"WIRED","lang":"zh","date":"2026-03-01"},{"title":"OpenClaw Users Are Allegedly Bypassing Anti-Bot Systems","url":"https://www.wired.com/story/openclaw-users-bypass-anti-bot-systems-cloudflare-scrapling/","source":"Wired","lang":"en","date":"2026-02-25"},{"title":"10 GitHub Repos for Scraping the Internet in 2026 (Real Stars)","url":"https://news.google.com/rss/articles/CBMiiAFBVV95cUxPMWZEd05SYXh2UUhNa2I4S2JBZjRfVktYUnhTV1FBdW5JMFgxRGVqWXdybGUxUkZyQTZGTGpYTlZNdXhoWTF3Ymd5T1dnTWJORExBdjJNTXpoQmQ3UnNQaVREMVRHMnZZWjdtTDZFcVllb3hISjVuVkN0VGY4NUV0RE50cWdDR0NR?oc=5","source":"Pasquale Pillitteri","lang":"en","date":"2026-06-30"},{"title":"2026 年抓取互联网的 10 个 GitHub 仓库 (附真实星标)","url":"https://news.google.com/rss/articles/CBMihAFBVV95cUxORGZEaDBndmMzUDdFYUdOTnRmeG9QQUw0RHRIZUFCTlNTMncyTjJIUWRWOVpLejNMa011VU5XeXNaZzQ1R0dEWk1IMWQweUFNZ2hyQjJMdlE3cy0xSDYxYTB0QllyOFh0QU1nNWxXVjRTazVzdDd1MTJ5R2VnbXdVRFpVenA?oc=5","source":"Pasquale Pillitteri","lang":"zh","date":"2026-06-30"},{"title":"OpenClaw最强外挂出现：小龙虾抓不到数据有救了！_手机新浪网","url":"https://news.google.com/rss/articles/CBMihwFBVV95cUxPMWl6VDIwWUVfdWZPeHFlZzk4VTFQaGxJZTZ1RUw5WWkwUU1YS01VcFAtbVRWbGdka0liTlcxeXg3b3BNbFFCcE5xUEZudDRLeWxkQ2lYUVRnWWVIaEk4bFpnX2lsNEZOdXNlX2YtcGszdFJYR3VTVzQxU3Y0YmpGdC1lNlFrdm8?oc=5","source":"新浪财经","lang":"zh","date":"2026-06-02"},{"title":"Scrapling - The Python scraper that fixes itself","url":"https://news.google.com/rss/articles/CBMic0FVX3lxTE1pYVAxSGdlUXl5ZjBiOVdHZ3p5QUhBWEJaTGQ1alRRSDhIRmR4ZUgzaGozZlJ2TGd2cGx6dmlxazFXSXN5c09MWWZRSVhCX1d4a0FJdmZnY0dhX01FX2xWRVJCQ0g2VmJ5eEtTbWdFdjhJWFE?oc=5","source":"Korben","lang":"en","date":"2026-04-28"}],"stale":false,"snapshot_at":"2026-11-10T09:35:25Z","history_start":"2026-09-06"}