import Header from './_source-info-header.md';

Zendesk#

Zendesk is a cloud-based customer service and support platform. It offers a range of features,
including ticket management, self-service options, knowledge base management, live chat, customer
analytics, and talks.

This Zendesk dlt verified source and
pipeline example
loads data using the “Zendesk Support API”, "Zendesk Chat API", and "Zendesk Talk API" to the destination
of your choice.

Endpoints that can be loaded using this verified source are:

To get the complete list of endpoints, please refer to
"zendesk/settings.py".

Setup guide#

Grab credentials#

You can load data from three types of Zendesk services, which are:

• Zendesk Support
• Zendesk Chat
• Zendesk Talk

Zendesk support#

Zendesk support can be authenticated using one of the following three methods:

• Method 1 (subdomain + email address + password)
• Method 2 (subdomain + email address + API token)
• Method 3 (subdomain + OAuth token)

The simplest way to authenticate is via subdomain + email address + password, since these details
are already available and you don't have to generate any tokens. Alternatively, you can also use API
tokens or OAuth tokens.

Grab subdomain#

1. Log into Zendesk to find your subdomain in the URL. E.g., for https://www.dlthub.zendesk.com, the
   subdomain is "dlthub".

Grab Zendesk support API token#

1. In Zendesk (top right), select Admin Center.
2. Choose "Apps and Integrations".
3. Navigate to APIs and select Zendesk API. Activate “Password access” & “Token access”.
4. Click “Add API token”, add a description, and note down the API token.
5. The token displays just once; note it safely.

Grab Zendesk support OAuth token#

To get the OAuth token, see the
documentation.
Here's a summarized version:

1. Obtain client ID via Zendesk API: send
   this curl
   request and note the response's client ID.

2. Alternatively, fetch client ID via OAuth using
   this.

3. To get the full token using the client ID obtained above, you can follow the instructions
   here.

   ⧉

    curl https://{subdomain}.zendesk.com/api/v2/oauth/tokens.json \
   -X POST \
   -v -u {email_address}:{password} \
   -H "Content-Type: application/json" \
   -d '{
     "token": {
       "client_id": 223443,
       "scopes": [
         "read"
       ]
     }
   }'
   

   We've set the scope as 'read', but you can customize the scope as needed.

4. In response to the above request, you'll get a full token which can be used to configure Zendesk
   support.

Zendesk chat#

Zendesk chat can be authenticated using this method:

• Method 1 (subdomain + Chat OAuth token)

  Note: OAuth tokens for Zendesk chat and support differ, requiring separate generation
  procedures.

Subdomain#

Log into Zendesk to find your subdomain in the URL. E.g., for https://www.dlthub.zendesk.com, the
subdomain is "dlthub".

Grab Zendesk chat OAuth token#

To generate a Zendesk chat OAuth token, please refer to this
documentation
. Below is a summary of the steps:

1. Access Zendesk Chat directly or through the top right "Chat" option in Zendesk product.
2. Navigate to "Settings" > "Account" > "API" > "Add API client".
3. Fill in the client name, company, and redirect URLs (default: http://localhost:8080).
4. Record the "CLIENT_ID" and "SUBDOMAIN".
5. Format the below URL with your own CLIENT_ID and SUBDOMAIN, paste it into a new browser tab, and
   press Enter.
   ⧉

   https://www.zopim.com/oauth2/authorizations/new?response_type=token&client_id=CLIENT_ID&scope=read%20write&subdomain=SUBDOMAIN
   

6. The call will be made, possibly asking you to log in and select 'Allow' to generate the token.
7. If the call succeeds, your browser's address field will contain your new OAuth token (returned as
   the access_token value).
8. Despite the seeming error message displayed in the browser's main window, if 'access_token' is
   returned in the browser's URL field then it worked!
   
9. Safely store the OAuth token to authenticate Zendesk Chat for retrieving data.
10. There are several other methods to obtain a Zendesk chat token as given in the full
    documentation here.

Zendesk talk#

Zendesk Talk fetches the data using the Zendesk Talk API.

1. Obtaining credentials for Zendesk Talk mirrors the process for
   Zendesk support.
2. Use existing Zendesk support credentials or create new ones.

Note: The Zendesk UI, which is described here, might change.
The full guide is available at this link.

Initialize the verified source#

To get started with your data pipeline, follow these steps:

1. Enter the following command:

   ⧉

   dlt init zendesk duckdb
   

   This command will initialize
   the pipeline example
   with Zendesk as the source and duckdb
   as the destination.

2. If you'd like to use a different destination, simply replace duckdb with the name of your
   preferred destination.

3. After running this command, a new directory will be created with the necessary files and
   configuration settings to get started.

Add credentials#

1. In the .dlt folder, there's a file called secrets.toml. It's where you store sensitive
   information securely, like access tokens. Keep this file safe. Here's its format for service
   account authentication:

   ⧉

   #Zendesk support credentials
   [sources.zendesk.credentials]
   subdomain = "subdomain" # Zendesk subdomain
   email = "set me up" # Email used to login to Zendesk
   password = "set me up" # Password for Zendesk account
   token = "set me up" # For API token auth
   oauth_token = "set me up" # Use Zendesk support OAuth token or Zendesk chat OAuth token
   

   For data retrieval from Zendesk Support or Talk, choose one of the following verification
   methods:

   • Method 1 (subdomain + email address + password)
   • Method 2 (subdomain + email address + API token)
   • Method 3 (subdomain + OAuth token)

   To load data from Zendesk Chat, use the following method for authentication:

   • Method 1 (subdomain + OAuth token)

   Note: Use the Zendesk Support OAuth token for configuring Zendesk Support, and for
   Chat, utilize the OAuth token specific to Zendesk Chat.

2. Finally, enter credentials for your chosen destination as per the docs.

For more information, read the General Usage: Credentials.

Run the pipeline#

1. Before running the pipeline, ensure that you have installed all the necessary dependencies by
   running the command:

   ⧉

   pip install -r requirements.txt
   

2. You're now ready to run the pipeline! To get started, run the following command:

   ⧉

   python zendesk_pipeline.py
   

3. Once the pipeline has finished running, you can verify that everything loaded correctly by using
   the following command:

   ⧉

   dlt pipeline <pipeline_name> show
   

   For example, the pipeline_name for the above pipeline example is dlt_zendesk_pipeline, you
   may also use any custom name instead.

For more information, read the guide on how to run a pipeline.

Sources and resources#

dlt works on the principle of sources and
resources.

Source zendesk_talk#

This function retrieves data from Zendesk Talk for phone calls and voicemails.

@dlt.source(max_table_nesting=2)
def zendesk_talk(
    credentials: TZendeskCredentials = dlt.secrets.value,
    start_date: Optional[TAnyDateTime] = START_DATE,
    end_date: Optional[TAnyDateTime] = None,
) -> Iterable[DltResource]:
   ...

credentials: Authentication credentials.

start_date: Start time for data range, defaults to January 1st, 2000.

end_date: End time for data range. If not provided, only new data is retrieved after the initial
run.

Resource talk_resource#

This function loads data from the Zendesk Talk endpoint.

def talk_resource(
    zendesk_client: ZendeskAPIClient,
    talk_endpoint_name: str,
    talk_endpoint: str,
    pagination_type: PaginationType,
) -> Iterator[TDataItem]:
   ...

zendesk_client: An instance of ZendeskAPIClient for making API calls to Zendesk Talk.

talk_endpoint_name: The name of the talk endpoint.

talk_endpoint: The actual URL ending of the endpoint.

pagination_type: Type of pagination used by the endpoint.

Other functions similar to the source zendesk_talk and resources similar to talk_endpoint are:

Customization#

Create your own pipeline#

If you wish to create your own pipelines, you can leverage source and resource methods from this
verified source.

1. Configure the pipeline by specifying the pipeline name, destination, and dataset as follows:

   ⧉

   pipeline = dlt.pipeline(
       pipeline_name="dlt_zendesk_pipeline", # Use a custom name if desired
       destination="duckdb", # Choose the appropriate destination (e.g., duckdb, redshift, post)
       dataset_name="sample_zendesk_data" # Use a custom name if desired
   )
   

2. To load data related to support, talk, and chat:

   ⧉

    # Zendesk support source function
    data_support = zendesk_support(load_all=True)
    # Zendesk chat source function
    data_chat = zendesk_chat()
    # Zendesk talk source function
    data_talk = zendesk_talk()
    # Run pipeline with all 3 sources
    info = pipeline.run([data_support, data_chat, data_talk])
    print(info)
   

3. To load data related to support, chat, and talk in incremental mode:

   ⧉

   pipeline = dlt.pipeline(
        pipeline_name="dlt_zendesk_pipeline", # Use a custom name if desired
        destination="duckdb", # Choose the appropriate destination (e.g., duckdb, redshift, post)
        dev_mode=False,
        dataset_name="sample_zendesk_data" # Use a custom name if desired
   )
   data = zendesk_support(load_all=True, start_date=START_DATE)
   data_chat = zendesk_chat(start_date=START_DATE)
   data_talk = zendesk_talk(start_date=START_DATE)
   info = pipeline.run(data=[data, data_chat, data_talk])
   print(info)
   

   Supports incremental loading for Support, Chat, and Talk endpoints. By default, it fetches data
   from the last load time in the dlt state or from 1st Jan 2000 if no prior load. This approach
   ensures data retrieval since the specified date, while still updating the last load time.

4. To load historical data in weekly ranges from Jan 1st, 2023, then switch to incremental loading
   for new tickets.

   ⧉

    # Load ranges of dates between January 1st, 2023, and today
    min_start_date = pendulum.DateTime(year=2023, month=1, day=1).in_timezone("UTC")
    max_end_date = pendulum.today()
    # Generate tuples of date ranges, each with 1 week in between.
    ranges = make_date_ranges(min_start_date, max_end_date, datetime.timedelta(weeks=1))
   
    # Run the pipeline in a loop for each 1-week range
    for start, end in ranges:
        print(f"Loading tickets between {start} and {end}")
        data = zendesk_support(start_date=start, end_date=end).with_resources("tickets")
        info = pipeline.run(data=data)
        print(info)
   
    # Backloading is done, now we continue loading with incremental state, starting where the backloading left off
    print(f"Loading with incremental state, starting at {end}")
    data = zendesk_support(start_date=end).with_resources("tickets")
    info = pipeline.run(data)
    print(info)
   

   This can be useful to reduce the potential failure window when loading large amounts of historic
   data. This approach can be used with all incremental Zendesk sources.