{"full_name":"pathwaycom/pathway","owner":"pathwaycom","name":"pathway","url":"https://github.com/pathwaycom/pathway","description":"Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.","homepage":"https://pathway.com","topics":[],"language":"Python","license":"NOASSERTION","avatar":"https://avatars.githubusercontent.com/u/25750857?v=4","created_at":"2022-11-27T13:01:14Z","pushed_at":"2026-11-10T06:39:23Z","metrics":{"stars":62012,"forks":1707,"watchers":115,"issues":174,"prs":81,"releases":89,"issues_open":null,"issues_closed":null,"prs_open":null,"prs_closed":null,"prs_merged":null,"discussions":0},"growth":{"day":{"stars":0,"forks":-1,"base_day":"2026-11-09"},"week":{"stars":-13,"forks":-2,"base_day":"2026-11-03"},"month":{"stars":-131,"forks":27,"base_day":"2026-10-11"}},"history":{"dates":["2026-09-06","2026-09-07","2026-09-08","2026-09-09","2026-09-10","2026-09-11","2026-09-12","2026-09-13","2026-09-14","2026-09-15","2026-09-16","2026-09-17","2026-09-18","2026-09-19","2026-09-20","2026-09-21","2026-09-22","2026-09-23","2026-09-24","2026-09-25","2026-09-26","2026-09-27","2026-09-28","2026-09-29","2026-09-30","2026-10-01","2026-10-02","2026-10-03","2026-10-04","2026-10-05","2026-10-06","2026-10-07","2026-10-08","2026-10-09","2026-10-10","2026-10-11","2026-10-12","2026-10-13","2026-10-14","2026-10-15","2026-10-16","2026-10-17","2026-10-18","2026-10-19","2026-10-20","2026-10-21","2026-10-22","2026-10-23","2026-10-24","2026-10-25","2026-10-26","2026-10-27","2026-10-28","2026-10-29","2026-10-30","2026-10-31","2026-11-01","2026-11-02","2026-11-03","2026-11-04","2026-11-09","2026-11-10"],"stars":[62339,62336,62330,62320,62313,62307,62298,62295,62290,62286,62284,62280,62274,62268,62261,62253,62254,62245,62245,62242,62237,62232,62222,62215,62214,62208,62204,62195,62186,62182,62176,62167,62160,62151,62144,62143,62134,62130,62128,62122,62116,62110,62108,62103,62096,62088,62088,62083,62073,62070,62063,62060,62051,62047,62042,62043,62038,62027,62025,62021,62012,62012],"forks":[1686,1686,1686,1686,1685,1685,1684,1684,1685,1685,1685,1686,1687,1687,1685,1685,1686,1685,1685,1682,1681,1682,1682,1682,1682,1683,1683,1683,1684,1684,1684,1681,1681,1681,1681,1680,1681,1680,1681,1683,1683,1683,1680,1678,1677,1679,1682,1680,1680,1680,1682,1684,1690,1691,1696,1701,1701,1705,1709,1714,1708,1707],"watchers":[120,121,121,121,121,121,121,121,121,121,121,121,121,121,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,121,121,121,121,121,121,121,121,121,121,121,121,121,121,121,121,116,116,116,116,116,115,115,115,115,115,115,115,115],"issues_open":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],"prs_open":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null]},"boards":{},"same_owner":[{"full_name":"pathwaycom/serviette","name":"serviette","boards":{}}],"issues":[{"number":67,"title":"Allow setting query transformers in the BaseRAGQA","comments":15,"created_at":"2024-07-22T07:41:13Z"},{"number":21,"title":"[Bug]: Simple processing of 1M rows takes more than 10 minutes","comments":12,"created_at":"2024-03-18T02:27:38Z"},{"number":17,"title":"[QUESTION] How to create accumulated batch for processing?","comments":8,"created_at":"2024-03-16T02:50:28Z"},{"number":194,"title":"[Proposal] Example: streaming RAG diagnostics with the WFGY 16-problem map in Pathway","comments":5,"created_at":"2026-02-20T13:08:41Z"},{"number":227,"title":"[Bug]: TypeError: Cannot instantiate typing.Any","comments":4,"created_at":"2026-05-01T08:04:16Z"},{"number":19,"title":"Add syntax to be able to reduce over multiple windows","comments":3,"created_at":"2024-03-17T20:32:26Z"},{"number":230,"title":"Entra Authentication Support (credential handler and/or password callback)","comments":2,"created_at":"2026-05-09T14:02:35Z"},{"number":75,"title":"Enable Link-Time Optimization (LTO) for Release builds","comments":2,"created_at":"2024-11-30T10:55:15Z"}],"discussions":[],"latest_release":{"name":"v0.32.1","tag":"v0.32.1","published_at":"2026-08-01T09:17:35Z","body":"### Added\r\n- `pw.io.chroma.write` writes a Pathway table to a [Chroma](https://www.trychroma.com/) collection, keeping the collection in sync with the table as rows are added, changed, and removed. The columns are mapped onto Chroma's record fields explicitly: the optional `primary_key` becomes the record id (when omitted, the row's internal Pathway key is used instead), `embedding` the vector, the optional `document` column the stored text, and `metadata_columns` the record metadata. The collection must already exist. The server is addressed with `host`/`port` (plus `ssl`, `headers`, `tenant`, and `database` for authenticated deployments such as Chroma Cloud).\r\n- `pw.io.qdrant.write` writes a Pathway table to a [Qdrant](https://qdrant.tech/) collection. Each row addition is upserted as a point and each deletion removes the corresponding point, so an update replaces a point rather than duplicating it. The pre-created collection's schema drives the writing: every named vector slot of the collection is bound to the table column with the same name — dense slots to `list[float]` (or 1-D `numpy.ndarray`) columns, sparse slots to `list[tuple[int, float]]` columns of `(index, weight)` pairs — and all vectors of a point are written atomically in one upsert, enabling native hybrid (dense + BM25) search. The remaining columns are stored in the point payload. The connector fails fast on a missing collection or any slot/column mismatch instead of degrading silently. The `batch_size` parameter bounds how many points are sent per request, and an optional `api_key` authenticates against Qdrant Cloud or a secured instance.\r\n- `pw.io.duckdb.write` writes a Pathway table into a DuckDB database file through a native, in-process connector, in either `\"stream_of_changes\"` or `\"snapshot\"` mode. Embeddings stored as `numpy` arrays or `list[float]` columns land in native `DOUBLE[]` list columns, so the result is directly searchable with DuckDB's vector-distance functions for RAG retrieval. The `detach_between_batches` option makes the writer close the database after every minibatch commit and reopen it for the next one, releasing the file lock in between — so a separate process (e.g. a query server) can read committed data with short-lived read-only connections while the pipeline keeps running; the writer retries the reopen with a backoff when a reader momentarily holds the lock.\r\n- `pw.io.weaviate.write` writes a Pathway table to a [Weaviate](https://weaviate.io/) collection, keeping it in sync with the table: additions and updates upsert objects (keyed by a UUID derived from the required `primary_key`), and deletions remove them. An optional `vector` column is stored as the object's embedding; the remaining columns become object properties. The target collection must already exist. The connector writes in parallel across workers (`pathway spawn -n`), with `batch_size` and `concurrency` to tune throughput, and supports `api_key`/`headers` authentication for self-hosted and Weaviate Cloud deployments.\r\n- `pw.io.pinecone.write` writes a Pathway table to a [Pinecone](https://www.pinecone.io/) index for use as a vector store in RAG pipelines. It keeps the index in sync with the current state of the table: a row is upserted under its record id and a row removed from the table is deleted from the index. By default the id is the table's internal row key (unique, and written in parallel across workers); pass a `primary_key` column to use your own ids instead, in which case the values must uniquely identify rows (a collision raises an error) and writing runs on a single worker. The `vector` column holds either a dense embedding (`list[float]` or a 1-D `numpy.ndarray`), written to a dense index, or a sparse vector (`list[tuple[int, float]]` of `(index, weight)` pairs), written to a sparse index — so a hybrid (dense + BM25) setup is two indexes fed by two `write()` calls off the same table, with the same record ids landing in both for client-side fusion. The r"},"readme":true,"readme_zh":false,"news":[],"stale":false,"snapshot_at":"2026-11-10T09:35:25Z","history_start":"2026-09-06"}