README

The Data Engineering Handbook


This repo has all the resources you need to become an amazing data engineer!

 Join the free Databricks AI boot camp on August 3rd here and signup for Databricks Free Edition here

 Getting started

If you are new to data engineering, start by following this 2024 breaking into data engineering roadmap

If you are here for the 4-week free beginner boot camp you can check out:
- introduction
- software needed

If you are here for the 6-week free intermediate boot camp you can check out
- introduction
- software needed


For more applied learning:
- Check out the projects section for more hands-on examples!
- Check out the interviews section for more advice on how to pass data engineering interviews!
- Check out the books section for a list of high quality data engineering books
- Check out the communities section for a list of high quality data engineering communities to join
- Check out the newsletter section to learn via email 


 Resources

 Great list of over 25 books

Top 3 must read books are:
- Fundamentals of Data Engineering
- Designing Data-Intensive Applications
- Designing Machine Learning Systems

 Great list of over 10 communities to join:

Top must-join communities for DE:
- DataExpert.io Community Discord
- Data Talks Club Slack
- Data Engineer Things Community

Top must-join communities for ML:
- AdalFlow Discord
- Chip Huyen MLOps Discord

 Companies:

- Orchestration  
  - Mage
  - Astronomer
  - Prefect
  - Dagster
  - Airflow
  - Kestra 
  - Shipyard
  - Hamilton
- Data Lake / Cloud
  - Tabular
  - Microsoft
  - Databricks
  - Onehouse
  - Delta Lake
  - Ilum
  - DuckLake
  - Apache Iceberg
  - Apache Polaris
  - Lakekeeper
- Data Warehouse
  - Snowflake
  - Firebolt
  - Databend
- Data Quality
  - dbt
  - Metaplane
  - Gable
  - Great Expectations
  - Streamdal
  - Coalesce
  - Soda
  - DQOps
  - HEDDA.IO
  - Dingo
- Education Companies
  - DataExpert.io
  - LearnDataEngineering.com
  - AlgoExpert
  - ByteByteGo
- Analytics / Visualization
  - Preset
  - Starburst
  - Metabase
  - Looker Studio
  - Tableau
  - Power BI
  - Hex
  - Apache Superset
  - Evidence
  - Redash
  - Lightdash
- Data Integration
  - Cube
  - Fivetran
  - Airbyte
  - dlt
  - Sling
  - Meltano
  - Estuary
  - Arpe.io
- Semantic Layers
  - Cube
  - dbt Semantic Layer 
- Modern OLAP
  - Apache Druid
  - ClickHouse
  - Apache Pinot
  - Apache Kylin
  - DuckDB
  - QuestDB
  - StarRocks
- LLM application library
  - AdalFlow
  - LangChain
  - LlamaIndex
- Real-Time Data
  - Aggregations.io
  - Responsive
  - RisingWave
  - Striim
- Data Lineage
  - OpenLineage


 Data Engineering blogs of companies:

- Netflix
- Uber
- Databricks
- Airbnb
- Amazon AWS Blog
- Microsoft Data Architecture Blogs
- Microsoft Fabric Blog
- Oracle
- Meta
- Onehouse
- Estuary Blog

 Data Engineering Whitepapers:

- A Five-Layered Business Intelligence Architecture
- Lakehouse:A New Generation of Open Platforms that Unify Data Warehousing and Advanced An

Top issues

Discussions