REST API Source with dltHub AI Workbench#

Overview#

Build a custom REST API connector for any of the 8k+ available sources — often in a single session. Instead of generating ad-hoc code, the AI assistant follows a defined sequence of steps from start to finish to help you build production-grade pipelines following the dltHub best practices.

The rest-api-pipeline toolkit is part of the dltHub AI Workbench and gives your coding assistant a structured, guided workflow — skills, rules, and an MCP server — tied together by a workflow that tells the assistant which skill to run at each step. It is designed to support an iterative data pipeline development flow and helps you validate results step by step.

The dltHub AI workbench works with Claude Code, Cursor, and Codex.

Setup#

Python environment#

Install uv (instructions) and create a virtual environment:

uv venv && source .venv/bin/activate

We recommend using uv. If you use uv, prefix all following commands with uv run.

Install dlt#

Install the dlt workspace:

pip install "dlt[hub]"

or upgrade to the latest version:

pip install --upgrade "dlt[hub]"

Initialize the AI assistant#

First create the dlthub workspace and install dependencies (see the installation guide):

uvx dlthub-start@latest

<Tabs values={[{"label": "Claude Code", "value": "claude"}, {"label": "Cursor", "value": "cursor"}, {"label": "Codex", "value": "codex"}]} groupId="ai-agent" defaultValue="claude">

dlthub ai init --agent claude

dlthub ai init --agent cursor

dlthub ai init --agent codex

dlthub ai init detects your coding assistant from environment variables and config files (unless you have multiple coding assistants set up), then installs skills, rules, and the MCP server in the correct locations for that tool.

Install toolkit#

dlthub ai toolkit install rest-api-pipeline

This installs the rest-api-pipeline toolkit: a workflow that orchestrates the assistant, an entry point skill, and step-by-step skills for each phase.

:::info Claude Code marketplace plugin
The workbench is also available as a Claude Code marketplace plugin. You don't need uv, Python, or dlt installed to get started — the bootstrap plugin handles everything from scratch.

The Claude Code plugin may not yet provide the best linking experience between different toolkits. We recommend using the dlthub ai CLI above for the best experience.

In Claude Code, run:

/plugin marketplace add dlt-hub/dlthub-ai-workbench
/plugin install bootstrap@dlthub-ai-workbench --scope project

In your terminal, exit the Claude session and restart (claude) — plugins take effect only after restarting Claude Code. Then run:

/init-workspace

The bootstrap skill checks for uv and Python, installs what's missing, creates a virtual environment, dlthub workspace and initializes AI assistant support — so you go from an empty directory to a fully configured workspace in one step.
:::

Build your first pipeline#

The toolkit provides a skill-driven workflow. Each skill listed below is triggered by the identified user intent or explicitly by naming it in prompt — the workflow guides the assistant through the right sequence.

/find-source — discover your data source#

Start a conversation with the /find-source skill. You can be explicit or let your coding assistant identify your intent:

/find-source I need to ingest pull requests and issues from the GitHub REST API

/find-source I want to track customer orders from our internal fulfillment API

Less explicit prompts let the assistant research the API on its own, which often produces better results for well-documented public APIs.

/create-rest-api-pipeline — scaffold the pipeline#

This step makes the coding assistant run dlt init, read the API documentation, and presents endpoint options. It produces a pipeline file with a structure like this:

import dlt
from dlt.sources.rest_api import rest_api_resources
from dlt.sources.rest_api.typing import RESTAPIConfig

@dlt.source
def github_source(
    access_token: str = dlt.secrets.value,
):
    config: RESTAPIConfig = {
        "client": {
            "base_url": "https://api.github.com/",
            "auth": {"type": "bearer", "token": access_token},
        },
        "resources": [
            "issues",
        ],
    }
    return rest_api_resources(config)

The assistant starts with a single endpoint, sets dev_mode=True, and adds .add_limit(1) so the first run finishes fast and validates the structure with a sample.

setup-secrets — configure credentials#

The assistant uses MCP tools to manage secrets — it is instructed not to read secrets.toml directly.
It will tell you what credentials are needed and where to obtain them. The coding assistant can see the shape of the secrets without actual values, so it is still able to detect misconfigurations and other dlt-related problems.

/debug-pipeline — run and fix#

The first run validates the pipeline structure. A credential error is expected if secrets are not yet configured:

dlt.common.configuration.exceptions.ConfigFieldMissingException:
  Missing 1 field(s) in configuration: `access_token`

The assistant sets INFO logging, inspects traces, and iterates until the pipeline completes:

Pipeline github_source load step completed in 0.26 seconds
1 load package(s) were loaded to destination duckdb and into dataset github_source_data
The duckdb destination used duckdb:/github_source.duckdb location to store data
Load package 1749667187.541553 is LOADED and contains no failed jobs

/validate-data — check your data#

A pipeline that runs without errors is not necessarily correct. Before moving on, the assistant helps you validate results using the dashboard, MCP queries, and schema inspection.

You can open Workspace Dashboard to apply your own judgement:

dlthub local pipeline show github_pipeline

See the full checklist for detailed steps.

Extend and harden#

/adjust-endpoint — remove dev limits and configure loading#

This step removes dev_mode and .add_limit(), verifies that pagination works end-to-end, and adds incremental loading so subsequent runs only fetch new data.

/new-endpoint — add more resources#

With this skill, you can add additional API endpoints to your source. The assistant is instructed to test them individually with with_resources() before running the full pipeline.

/view-data — explore your dataset#

Open the dlt Dashboard for visual inspection, or use the Python dataset API for programmatic exploration:

import dlt

github_pipeline = dlt.pipeline("github_pipeline")
github_dataset = github_pipeline.dataset()
github_dataset.tables
github_dataset.table("pull_requests").columns
github_dataset.table("pull_requests").df()

This works well in interactive environments like marimo and Jupyter.

Anatomy of the REST API pipeline toolkit#

A toolkit contains skills, rules, and an MCP server — tied together by a workflow that tells the assistant which skill to run at each step and how to leverage the MCP.

How the components interact#

The workflow is loaded into the assistant's context automatically as a rule. When you invoke the entry skill, it tells the assistant which skill to run next based on the current state.
Skills are prompts with embedded instructions — they teach the assistant how to use dlt APIs, REST API configuration, and MCP tools.
The MCP server exposes tools like list_pipelines, get_table_schema, and execute_sql_query so the assistant can inspect results without you copying output manually.

Handover to other toolkits#

Once your pipeline is validated, you can continue to the next phase of the data engineering lifecycle by installing additional toolkits:

• data-exploration — connects to your loaded pipeline data and builds an interactive marimo dashboard with Altair charts as the final output.
  The assistant adapts to how explicitly the user specifies their intent:

  • High-intent (user has a specific question to be answered by the data, e.g. "show revenue by month"): scans the schema, plans the chart, and renders it directly to the dashboard.
  • Low-intent (e.g. "explore my github data"): profiles the data, suggests 5–10 business questions, and builds a dashboard from your selection.

  Two skills drive the workflow: /explore-data (entry point skill) connects to your data, profiles it by mapping the intent to the available data and identifying potential gaps, while /build-notebook assembles the charts into a marimo dashboard and launches it in your browser.
  The toolkit is designed for iterative development: create one chart and launch the notebook, then iterate — add or refine charts and re-launch.

• dlthub-runtime — deploy, schedule, and monitor your pipeline in production

uv run dlthub ai toolkit install data-exploration
uv run dlthub ai toolkit install dlthub-runtime

Results#

By the end of this guide, you should have:

• A workspace with coding assistant rules and MCP tools configured
• A working REST API source with validated endpoints
• A local dataset you have inspected and verified

Next steps:

• Explore the dataset and build a data product
• Replace the local destination with your data warehouse