Release highlights: 1.12.3 & 1.13.0 & 1.14.1#

Breaking change: Ibis dataset/relation functionality + new non-Ibis relation features#

What changed

Accessing Ibis relations using dataset["table_name"] or dataset.table_name no longer works.

Migration required

To get an Ibis table relation, explicitly set the table type to get the table expression first, then convert it to a table relation:

# Previously: dataset["customers"] # now returns a regular relation
customers_expression = dataset.table("customers", table_type="ibis")
customers_relation = dataset(customers_expression)

df = customers_relation.df() # fetch as DataFrame

Check out the updated docs and migration guide for full details and examples.

New: non-Ibis relation enhancements

You can now use a more SQL-like interface on regular relations:

customers = dataset.table("customers")

# Sort
customers.order_by("created_at").fetchall()

# Filter
customers.filter("id = 1").fetchall()

# Aggregation
customers.select("id").max().scalar()

REST API: New paginator for header-based cursors#

PR: #2798

Overview

The new header_cursor paginator enables pagination for APIs that return the next page cursor in the response headers, such as those using custom headers like NextPageToken.

Use case

Some REST APIs respond like this:

# response header
Content-Type: application/json
NextPageToken: 123456

# response JSON
[
  {"id": 1, "name": "item1"},
  {"id": 2, "name": "item2"}
]

To fetch the next page:

https://api.example.com/items?page=123456

Example configuration

{
  "path": "items",
  "paginator": {
    "type": "header_cursor",
    "cursor_key": "NextPageToken", # name of the response header
    "cursor_param": "page" # query parameter used to pass the token
  }
}

You can now extract cursors from headers just like from JSON paths in the response body.

Autocompletion in Notebooks: Dataset and relation#

PR: #2891

Overview

Working in notebooks just got smoother. We’ve added tab autocompletion for dlt.Dataset and dlt.Relation objects.

Now you can:

• Use dataset.<TAB> to see available tables
• Use relation.<TAB> to explore available columns

How it works

The feature overrides _ipython_key_completions_ to inspect dataset/relation structure and return available names.

Example

dataset # <TAB> will suggest all table names
dataset.customers # <TAB> will suggest column names like id, email, created_at

DuckLake setup guide for MotherDuck#

PR: #2842

Overview

We’ve added a dedicated section in the docs for setting up DuckLake, a powerful way to persist MotherDuck databases on external storage like S3.

The guide includes:

1. Creating a DuckLake-managed database on S3
2. Storing S3 credentials securely in MotherDuck
3. Connecting dlt to DuckLake via secrets.toml

Why use DuckLake?

• Offload MotherDuck data to S3
• Control over storage location and retention
• Load data via dlt like with any DuckDB destination

DuckLake setup in docs →

Athena + Iceberg + Lake Formation tags and table properties#

PR: #2808

Overview

You can now apply Lake Formation tags to the Glue database used by the Athena destination. All tables created by dlt will inherit these tags.

Example configuration

[destination.athena.lakeformation_config]
enabled = true

[destination.athena.lakeformation_config.tags]
Environment = "prod"
Team = "analytics"

Add post-create SQL statements#

PR: #2791

You can now run custom SQL statements after tables are created or altered during schema migration. Useful for foreign keys, constraints, etc.

Example usage

Override _get_table_post_update_sql in your SqlJobClientBase:

def _get_table_post_update_sql(self, partial_table):
    if partial_table["name"] == "orders":
        return [
            "ALTER TABLE orders ADD CONSTRAINT fk_user FOREIGN KEY (user_id) REFERENCES users(id)"
        ]
    return []

These run after all tables are updated, allowing dependency-safe logic.

CLI: Friendly drop command 🐕‍🦺#

PR: #2720

Overview

The dlt drop command now includes smarter validation and warnings to avoid accidental misconfigurations.

Warning types

1. Run directory validation

   ⧉

   You should run this from the same directory as the pipeline script...
   

2. Misaligned config context detection

   ⧉

   WARNING: Active pipeline `my_pipeline` used `/home/user/project/.dlt`...
   

3. Pipeline script vs. run dir mismatch

   ⧉

   WARNING: Your run dir (/home/user/tmp) is different from the pipeline script...
   

4. Credential visibility notice

   ⧉

   WARNING: When accessing data from the command line, dlt does not execute your pipeline code...
   

5. Table drop preview
   Lists tables that will be deleted before performing the drop.

Shout-out to new contributors#

Big thanks to our newest contributors:

• @AyushPatel101 (#2803)
• @nicob3y (#2755)
• @kaliole (#2849)
• @franloza (#2869)

Full release notes

View the complete list of changes →