dlt Advanced Course#

In this course, you'll go far beyond the basics. You’ll build production-grade data pipelines with custom implementations, advanced patterns, and performance optimizations.

Lessons#

Lesson 1: Custom Sources – REST APIs & RESTClient #

Learn how to build flexible REST API connectors from scratch using @dlt.resource and the powerful RESTClient.

Lesson 2: Custom Sources – SQL Databases #

Connect to any SQL-compatible database, reflect table schemas, write query adapters, and selectively ingest data using sql_database.

Lesson 3: Custom Sources – Filesystems & Cloud Storage #

Build sources that read from local or remote files (S3, GCS, Azure).

Lesson 4: Custom Destinations – Reverse ETL #

Use @dlt.destination to send data back to APIs like Notion, Slack, or Airtable. Learn batching, retries, and idempotent patterns.

Lesson 5: Transforming Data Before & After Load #

Learn when and how to apply add_map, add_filter, @dlt.transformer, or even post-load transformations via SQL or Ibis. Control exactly how your data looks.

Lesson 6: Write Disposition Strategies & Advanced Tricks #

Understand how to use replace and merge, and combine them with schema hints and incremental loading.

Lesson 7: Data Contracts #

Define expectations on schema, enforce data types and behaviors, and lock down your schema evolution. Ensure reliable downstream use of your data.

Lesson 8: Logging & Tracing #

Track every step of your pipeline: from extraction to load. Use logs, traces, and metadata to debug and analyze performance.

Lesson 9: Performance Optimization #

Handle large datasets, tune buffer sizes, parallelize resource extraction, optimize memory usage, and reduce pipeline runtime.

Homework & Certification#

You’ve finished the dlt Advanced Course — well done! Test your skills with the Advanced Certification Homework.