A profile is a set of configurations and secrets defined for a specific use case. Profiles provide a way to manage different configurations for different environments.

They are defined in the dlt.yml under the profiles section.

profiles:
  # profiles allow you to configure different settings for different environments
  dev:
    sources:
      my_arrow_source:
        row_count: 100
    runtime:
      log_level: DEBUG
  prod:
    sources:
      my_arrow_source:
        row_count: 200
    runtime:
      log_level: INFO
    destinations:
      my_duckdb_destination:
        credentials: my_data_prod.duckdb

Every project includes two implicit profiles by default: dev and tests. If no profile is specified, the dev profile is loaded by default.
All CLI commands that run on a project support the --profile option, allowing you to specify the desired profile. For example,

dlt project --profile dev my_pipeline run
dlt dataset --profile prod my_duckdb_destination_dataset row-counts

Using config files with profiles#

All the configuration and secrets for profiles can also be placed in TOML files, as described in dlt OSS documentation.
Each profile can have its own secrets.toml file, which is only loaded when that profile is active.

For example, if you have two secrets files under .dlt:

.
├── .dlt/ # your dlt settings including profile settings
│ ├── config.toml
│ ├── dev.secrets.toml
│ └── tests.secrets.toml

You can run a pipeline with different profiles as follows:

dlt pipeline --profile dev my_pipeline run
dlt pipeline --profile tests my_pipeline run

Pinning profiles#

You can pin a profile locally, making the given profile name the default one. This is useful, for example, when deploying your project in a production or staging environment.

dlt profile prod pin

will pin the prod profile and from now on all Python scripts and cli commands will see it as the default and switch to it automatically.
The profile pin is kept in the .dlt/profile-name file. Remove this file to unpin. Note that our default .gitignore prevents this file from being added.

Switching profile with an environment variable#

You can override the active profile for a single command or script by setting the WORKSPACE__PROFILE environment variable. This takes precedence over the pinned profile but yields to a profile set explicitly in code (e.g. --profile flag).

Run a CLI command with a different profile:

WORKSPACE__PROFILE=prod dlt pipeline my_pipeline info

Run a Python script with a different profile:

WORKSPACE__PROFILE=tests python my_pipeline.py

This is useful in CI/CD pipelines or when you want to quickly test against a different profile without changing the pin:

# run tests against the prod profile, then go back to normal
WORKSPACE__PROFILE=prod pytest tests/

Settings in the dlt.yml file vs TOML files#

For dltHub Projects, it's best practice to keep all non-secret settings in dlt.yml and store secrets only in .dlt/secrets.toml. This ensures that sensitive data is only available in the necessary profiles or environments.

In the example above, some non-secret values were moved to .dlt/secrets.toml for demonstration purposes only - this is not the recommended approach.