Command Line Interface Reference#

This page contains all commands available in the dlt CLI and is generated
automatically from the fully populated python argparse object of dlt.

dlt#

Creates, adds, inspects and deploys dlt pipelines. Further help is available at https://dlthub.com/docs/reference/command-line-interface.

Usage

dlt [-h] [-v] [--non-interactive] [-y] [--debug] [--version]
    [--disable-telemetry] [--enable-telemetry] [--no-pwd]
    {telemetry,schema,pipeline,init,deploy,dashboard,ai} ...

dlt telemetry#

Shows telemetry status.

Usage

dlt telemetry [-h]

Description

Shows the current status of dlt telemetry. Learn more about telemetry and what we send in our telemetry docs.

dlt schema#

Shows, converts and upgrades schemas.

Usage

dlt schema [-h] [--format {json,yaml,dbml,dot,mermaid}] [--remove-defaults] file

Description

Loads, validates and prints out a dlt schema from a yaml or json file.

dlt pipeline#

Inspects pipeline state, trace, load packages, provides basic maintenance.

Usage

dlt pipeline [-h] [--list-pipelines] [--pipelines-dir PIPELINES_DIR]
    [pipeline_name]
    {info,show,failed-jobs,drop-pending-packages,sync,trace,schema,drop,load-package,mcp}
    ...

Description

Provides tools to inspect the pipeline working directory, tables, and data in the destination, and to check for problems encountered during data loading.

dlt pipeline info#

Displays state of the pipeline, use -v or -vv for more info.

Usage

dlt pipeline [pipeline_name] info [-h]

Description

Displays the content of the working directory of the pipeline: dataset name, destination, list of
schemas, resources in schemas, list of completed and normalized load packages, and optionally a
pipeline state set by the resources during the extraction process.

dlt pipeline show#

Generates and launches workspace dashboard with the loading status and dataset explorer.

Usage

dlt pipeline [pipeline_name] show [-h] [--edit]

Description

Launches the workspace dashboard with a comprehensive interface to inspect the pipeline state, schemas, and data in the destination.

This dashboard should be executed from the same folder from which you ran the pipeline script to be able access destination credentials.

If the --edit flag is used, will launch the editable version of the dashboard if it exists in the current directory, or create this version and launch it in edit mode.

Requires marimo to be installed in the current environment: pip install marimo.

dlt pipeline failed-jobs#

Displays information on all the failed loads in all completed packages, failed jobs and associated error messages.

Usage

dlt pipeline [pipeline_name] failed-jobs [-h]

Description

This command scans all the load packages looking for failed jobs and then displays information on
files that got loaded and the failure message from the destination.

dlt pipeline drop-pending-packages#

Deletes all extracted and normalized packages including those that are partially loaded.

Usage

dlt pipeline [pipeline_name] drop-pending-packages [-h]

Description

Removes all extracted and normalized packages in the pipeline's working dir.
dlt keeps extracted and normalized load packages in the pipeline working directory. When the run method is called, it will attempt to normalize and load
pending packages first. This command removes such packages. Note that pipeline state is not reverted to the state at which the deleted packages
were created. Using the sync sub-command is recommended if your destination supports state sync.

dlt pipeline sync#

Drops the local state of the pipeline and resets all the schemas and restores it from destination. The destination state, data and schemas are left intact.

Usage

dlt pipeline [pipeline_name] sync [-h] [--destination DESTINATION]
    [--dataset-name DATASET_NAME]

Description

This command will remove the pipeline working directory with all pending packages, not synchronized
state changes, and schemas and retrieve the last synchronized data from the destination. If you drop
the dataset the pipeline is loading to, this command results in a complete reset of the pipeline state.

In case of a pipeline without a working directory, this command may be used to create one from the
destination. In order to do that, you need to pass the dataset name and destination name to the CLI
and provide the credentials to connect to the destination (i.e., in .dlt/secrets.toml) placed in the
folder where you run it.

dlt pipeline trace#

Displays last run trace, use -v or -vv for more info.

Usage

dlt pipeline [pipeline_name] trace [-h]

Description

Displays the trace of the last pipeline run containing the start date of the run, elapsed time, and the
same information for all the steps (extract, normalize, and load). If any of the steps failed,
you'll see the message of the exceptions that caused that problem. Successful load and run steps
will display the load info instead.

dlt pipeline schema#

Displays default schema.

Usage

dlt pipeline [pipeline_name] schema [-h] [--format {json,yaml,dbml,dot,mermaid}]
    [--remove-defaults]

Description

Displays the default schema for the selected pipeline.

dlt pipeline drop#

Selectively drop tables and reset state.

Usage

dlt pipeline [pipeline_name] drop [-h] [--destination DESTINATION]
    [--dataset-name DATASET_NAME] [--drop-all] [--state-paths [STATE_PATHS ...]]
    [--schema SCHEMA_NAME] [--state-only] [resources ...]

Description

Selectively drop tables and reset state.

dlt pipeline <pipeline name> drop [resource_1] [resource_2]

Drops tables generated by selected resources and resets the state associated with them. Mainly used
to force a full refresh on selected tables. In the example below, we drop all tables generated by
the repo_events resource in the GitHub pipeline:

dlt pipeline github_events drop repo_events

dlt will inform you of the names of dropped tables and the resource state slots that will be
reset:

About to drop the following data in dataset airflow_events_1 in destination dlt.destinations.duckdb:
Selected schema:: github_repo_events
Selected resource(s):: ['repo_events']
Table(s) to drop:: ['issues_event', 'fork_event', 'pull_request_event', 'pull_request_review_event', 'pull_request_review_comment_event', 'watch_event', 'issue_comment_event', 'push_event__payload__commits', 'push_event']
Resource(s) state to reset:: ['repo_events']
Source state path(s) to reset:: []
Do you want to apply these changes? [y/N]

As a result of the command above the following will happen:

1. All the indicated tables will be dropped in the destination. Note that dlt drops the nested
   tables as well.
2. All the indicated tables will be removed from the indicated schema.
3. The state for the resource repo_events was found and will be reset.
4. New schema and state will be stored in the destination.

The drop command accepts several advanced settings:

1. You can use regexes to select resources. Prepend the re: string to indicate a regex pattern. The example
   below will select all resources starting with repo:

dlt pipeline github_events drop "re:^repo"

2. You can drop all tables in the indicated schema:

dlt pipeline chess drop --drop-all

3. You can indicate additional state slots to reset by passing JsonPath to the source state. In the example
   below, we reset the archives slot in the source state:

dlt pipeline chess_pipeline drop --state-paths archives

This will select the archives key in the chess source.

{
  "sources":{
    "chess": {
      "archives": [
        "https://api.chess.com/pub/player/magnuscarlsen/games/2022/05"
      ]
    }
  }
}

This command is still experimental and the interface will most probably change.

dlt pipeline load-package#

Displays information on load package, use -v or -vv for more info.

Usage

dlt pipeline [pipeline_name] load-package [-h] [load-id]

Description

Shows information on a load package with a given load_id. The load_id parameter defaults to the
most recent package. Package information includes its state (COMPLETED/PROCESSED) and list of all
jobs in a package with their statuses, file sizes, types, and in case of failed jobs—the error
messages from the destination. With the verbose flag set (-v), you can also see the
list of all tables and columns created at the destination during the loading of that package.

dlt pipeline mcp#

Launch MCP server attached to this pipeline.

Usage

dlt pipeline [pipeline_name] mcp [-h] [--stdio] [--sse] [--port PORT]
    [--features [FEATURES ...]]

Description

This MCP facilitates schema and data exploration for the dataset created with this pipeline.

dlt init#

Creates a pipeline in the current folder by adding existing verified source or creating a new one from template.

Usage

dlt init [-h] [--list-sources] [--list-destinations] [--location LOCATION]
    [--branch BRANCH] [--eject] [source] [destination]

Description

This command creates a new dlt pipeline script that loads data from source to destination. When you run the command, several things happen:

1. Creates a basic project structure if the current folder is empty by adding .dlt/config.toml, .dlt/secrets.toml, and .gitignore files.
2. Checks if the source argument matches one of our verified sources and, if so, adds it to your project.
3. If the source is unknown, uses a generic template to get you started.
4. Rewrites the pipeline scripts to use your destination.
5. Creates sample config and credentials in secrets.toml and config.toml for the specified source and destination.
6. Creates requirements.txt with dependencies required by the source and destination. If one exists, prints instructions on what to add to it.

This command can be used several times in the same folder to add more sources, destinations, and pipelines. It will also update the verified source code to the newest
version if run again with an existing source name. You will be warned if files will be overwritten or if the dlt version needs an upgrade to run a particular pipeline.

dlt deploy#

Creates a deployment package for a selected pipeline script.

Usage

dlt deploy [-h] pipeline-script-path {github-action,airflow-composer} ...

Description

Prepares your pipeline for deployment and gives you step-by-step instructions on how to accomplish it. To enable this functionality, please first execute pip install "dlt[cli]" which adds additional packages to the current environment.

dlt deploy github-action#

Deploys the pipeline to Github Actions.

Usage

dlt deploy pipeline-script-path github-action [-h] [--location LOCATION]
    [--branch BRANCH] --schedule SCHEDULE [--run-manually] [--run-on-push]

Description

Deploys the pipeline to GitHub Actions.

GitHub Actions (https://github.com/features/actions) is a CI/CD runner with a large free tier which you can use to run your pipelines.

You must specify when the GitHub Action should run using a cron schedule expression. The command also takes additional flags:
--run-on-push (default is False) and --run-manually (default is True). Remember to put the cron
schedule expression in quotation marks.

For the chess.com API example from our docs, you can deploy it with dlt deploy chess.py github-action --schedule "*/30 * * * *".

Follow the guide on how to deploy a pipeline with GitHub Actions in our documentation for more information.

dlt deploy airflow-composer#

Deploys the pipeline to Airflow.

Usage

dlt deploy pipeline-script-path airflow-composer [-h] [--location LOCATION]
    [--branch BRANCH] [--secrets-format {env,toml}]

Description

Google Composer (https://cloud.google.com/composer?hl=en) is a managed Airflow environment provided by Google. Follow the guide in our docs on how to deploy a pipeline with Airflow to learn more. This command will:

• create an Airflow DAG for your pipeline script that you can customize. The DAG uses
  the dlt Airflow wrapper (https://github.com/dlt-hub/dlt/blob/devel/dlt/helpers/airflow_helper.py#L37) to make this process trivial.

• provide you with the environment variables and secrets that you must add to Airflow.

• provide you with a cloudbuild file to sync your GitHub repository with the dag folder of your Airflow Composer instance.

dlt dashboard#

Shows the dlthub workspace dashboard.

Usage

dlt dashboard [-h] [--pipelines-dir PIPELINES_DIR] [--edit]

Description

This command shows the dlt workspace dashboard. You can use the dashboard:

• to list and inspect local pipelines
• browse the full pipeline schema and all hints
• browse the data in the destination
• inspect the pipeline state.

dlt ai#

Moved to dlthub ai (run pip install dlt[hub]).

Usage

dlt ai [-h]

Description

ai command moved to dlthub, pip install dlt[hub] and dlthub ai to use.