Ibis#

Ibis is a powerful portable Python dataframe library. Learn more about what it is and how to use it in the official documentation.

dlt provides an easy way to hand over your loaded dataset to an Ibis backend connection.

Prerequisites#

To use the Ibis backend, you will need to have the ibis-framework package with the correct Ibis extra installed. The following example will install the DuckDB backend:

pip install ibis-framework[duckdb]

Get an Ibis connection from your dataset#

dlt datasets have a helper method to return an Ibis connection to the destination they live on. The returned object is a native Ibis connection to the destination, which you can use to read and even transform data. Please consult the Ibis documentation to learn more about what you can do with Ibis.

:::caution Breaking change in dlt 1.25.0
dataset.ibis() now passes all schemas from the dataset to the Ibis backend. On filesystem destinations, this means Ibis will see tables from every schema in the dataset and not just the default one. If two schemas define the same table name, the Ibis table will contain rows from both schemas combined. To get the previous single-schema behavior, create the dataset with an explicit schema: pipeline.dataset(schema="my_schema").ibis().
:::

# get the dataset from the pipeline
dataset = pipeline.dataset()
dataset_name = pipeline.dataset_name

# get the native ibis connection from the dataset
ibis_connection = dataset.ibis()

# list all tables in the dataset
# NOTE: You need to provide the dataset name to ibis, in ibis datasets are named databases
print(ibis_connection.list_tables(database=dataset_name))

# get the items table
table = ibis_connection.table("items", database=dataset_name)

# print the first 10 rows
print(table.limit(10).execute())

# Visit the ibis docs to learn more about the available methods