Data Engineering Journey

Tracking my progress from SQL fundamentals toward production data systems, and later AI engineering.

42%complete

Updated

  • Overall Progress42%
  • Completed14 skills5 core + 9 SQL topics
  • In Progress2 topicsActive this month
  • Projects30 active, 3 upcoming
  • Current FocusDocker+ Data Warehousing

Data Engineering roadmap

6 stages
  1. Foundations
  2. Database
  3. Pipelines
  4. Warehousing
  5. Cloud
  6. Advanced

Currently Learning

2 active

Roadmap

  1. Stage 1
    Foundations4 of 4 skills done
    Completed100%

    Core SQL querying, Python scripting, and relational schema design.

    • Basic SQL100%
      • SELECT, WHERE, ORDER BY, LIMIT
      • GROUP BY, HAVING, aggregates
      • INNER, LEFT, RIGHT JOIN
      • CASE, CRUD
    • Python Fundamentals100%
      • Control flow, functions
      • Lists, dicts, sets
      • File handling, JSON / CSV
      • Calling APIs
    • Database Design100%
      • ER diagrams
      • Primary, foreign, surrogate keys
      • Normalization (1NF to 3NF)
      • One-to-many, many-to-many
    • Data Modeling100%
      • Conceptual, logical, physical models
      • OLTP vs OLAP
  2. Stage 2
    Core Database Skills1 of 2 skills done
    In Progress50%

    Analytical SQL, query plans, indexing, and a production-grade database.

    • Advanced SQL100%
      • Complex Joins
      • CTEs
      • Window Functions
      • Subqueries
      • Views
      • Query Optimization
      • Indexes
      • Materialized Views
      • Transactions
      • EXPLAIN / EXPLAIN ANALYZE
    • PostgreSQLPlanned
      • Schemas, constraints
      • Transactions
      • Indexing, partitioning
      • JSONB
  3. Stage 3
    Data Pipelines0 of 2 skills done
    In Progress25%

    Moving, validating, and containerizing batch data workflows.

    • ETL / ELTPlanned
      • Python ingestion scripts
      • API ingestion
      • CSV / JSON ingestion
      • Incremental loading
      • Deduplication
      • Data validation
      • Error handling
      • Logging
    • Docker50%
      • Containerized databases
      • Volumes
      • Docker Compose
  4. Stage 4
    Data Warehousing0 of 2 skills done
    In Progress20%

    Dimensional modeling, star schemas, and modular transformations.

    • Data Warehousing40%
      • Fact tables
      • Dimension tables
      • Star schema
      • Grain
      • Surrogate keys
      • Slowly Changing Dimensions
      • Partitioning
    • dbtFuture
      • Models, sources, ref()
      • Tests, documentation
      • Incremental models
  5. Stage 5
    Orchestration & Cloud0 of 2 skills done
    Future0%

    Scheduling pipelines and running them on cloud infrastructure.

    • AirflowFuture
      • DAGs
      • Scheduling
      • Dependencies
      • Retries
      • Backfills
      • Monitoring
    • CloudFuture
      • AWS / GCP / Azure fundamentals
      • Object storage
      • Managed databases
      • IAM
      • Cloud warehouses
  6. Stage 6
    Advanced Data Engineering0 of 2 skills done
    Future0%

    Distributed batch processing and real-time streaming.

    • Spark / PySparkFuture
      • Spark
      • PySpark
      • Parquet
      • Distributed processing
    • KafkaFuture
      • Kafka
      • Streaming pipelines
  7. Stage 7Later
    AI Engineering1 of 15 skills done
    In Progress7%

    A later extension once the data foundations are solid.

    • AI Search Algorithms100%
    • Math / StatisticsFuture
    • Machine LearningFuture
    • scikit-learnFuture
    • PyTorchFuture
    • Deep LearningFuture
    • TransformersFuture
    • Hugging FaceFuture
    • LLM APIsFuture
    • EmbeddingsFuture
    • Vector databasesFuture
    • RAGFuture
    • AI AgentsFuture
    • MLOpsFuture
    • Production AI systemsFuture

Current Goals

  • Docker

    Master:

  • Data Warehousing

    Master:

Learning paths

Data first, AI later
Data EngineeringPrimary
  1. SQL
  2. PostgreSQL
  3. ETL
  4. dbt
  5. Airflow
  6. Spark
AI EngineeringFuture
  1. ML
  2. PyTorch
  3. Transformers
  4. RAG
  5. Agents
  6. MLOps

Projects

All projects →

  • Olist E-Commerce Analytics & OLAP

    Completed
    • MySQL
    • Docker
    • CTEs
    • Window Functions
    • OLAP

  • API to PostgreSQL Pipeline

    Planned
    • Python
    • API
    • PostgreSQL
    • ETL

  • Modern Data Warehouse

    Future
    • PostgreSQL
    • dbt
    • Airflow