import Header from './_source-info-header.md';

GitHub#

This verified source can be used to load data on issues or pull requests from any GitHub repository onto a destination of your choice using the GitHub API.

Resources that can be loaded using this verified source are:

Name	Description
github_reactions	Retrieves all issues, pull requests, comments, and reactions associated with them
github_repo_events	Gets all the repo events associated with the repository

Setup guide#

Grab credentials#

To get the API token, sign in to your GitHub account and follow these steps:

Click on your profile picture in the top right corner.
Choose "Settings".
Select "Developer settings" on the left panel.
Under "Personal access tokens", click on "Generate a personal access token (preferably under Tokens(classic))".

Grant at least the following scopes to the token by checking them.

Scope	Description
public_repo	Limits access to public repositories
read	Grants read and ping access to hooks in public or private repositories
read	Read-only access to organization membership, organization projects, and team membership
read	Grants access to read a user's profile data
read	Grants read-only access to user and organization projects
read	Allows read access for team discussions

Finally, click "Generate token".
Copy the token and save it. This is to be added later in the dlt configuration.

You can optionally add API access tokens to avoid making requests as an unauthorized user.
If you wish to load data using the github_reaction source, the access token is mandatory.

For more information, see the
GitHub authentication
and
GitHub API token scopes
documentation.

Initialize the verified source#

To get started with your data pipeline, follow these steps:

Enter the following command:
```
dlt init github duckdb
```
This command will initialize
the pipeline example
with GitHub as the source and duckdb
as the destination.
If you'd like to use a different destination, simply replace duckdb with the name of your
preferred destination.
After running this command, a new directory will be created with the necessary files and
configuration settings to get started.

Add credentials#

In .dlt/secrets.toml, you can securely store your access tokens and other sensitive information. It's important to handle this file with care and keep it safe. Here's what the file looks like:
```
# Put your secret values and credentials here
# GitHub access token (must be classic for reactions source)
[sources.github]
access_token="please set me up!" # use GitHub access token here
```
Replace the API token value with the previously copied one to ensure secure access to your GitHub resources.
Next, follow the destination documentation instructions to add credentials for your chosen destination, ensuring proper routing of your data to the final destination.

For more information, read the General Usage: Credentials.

Run the pipeline#

Before running the pipeline, ensure that you have installed all the necessary dependencies by running the command:
```
pip install -r requirements.txt
```
You're now ready to run the pipeline! To get started, run the following command:
```
python github_pipeline.py
```
Once the pipeline has finished running, you can verify that everything loaded correctly by using the following command:
```
dlt pipeline <pipeline_name> show
```
For example, the pipeline_name for the above pipeline example is github_reactions; you may also use any custom name instead.

For more information, read the guide on how to run a pipeline.

Sources and resources#

dlt works on the principle of sources and resources.

Source `github_reactions`#

This dlt.source function uses GraphQL to fetch DltResource objects: issues and pull requests along with associated reactions, comments, and reactions to comments.

@dlt.source
def github_reactions(
    owner: str,
    name: str,
    access_token: str = dlt.secrets.value,
    items_per_page: int = 100,
    max_items: int = None,
    max_item_age_seconds: float = None,
) -> Sequence[DltResource]:

   return dlt.resource(
      _get_reactions_data(
         "issues",
         owner,
         name,
         access_token,
         items_per_page,
         max_items,
         max_item_age_seconds,
      ),
      name="issues",
      write_disposition="replace",
   )

owner: Refers to the owner of the repository.

name: Refers to the name of the repository.

access_token: A classic access token should be utilized and is stored in the .dlt/secrets.toml file.

items_per_page: The number of issues/pull requests to retrieve in a single page. Defaults to 100.

max_items: The maximum number of issues/pull requests to retrieve in total. If set to None, it means all items will be retrieved. Defaults to None.

max_item_age_seconds: The feature to restrict retrieval of items older than a specific duration is yet to be implemented. Defaults to None.

Resource `_get_reactions_data` ("issues")#

The dlt.resource function employs the _get_reactions_data method to retrieve data about issues, their associated comments, and subsequent reactions.

Source `github_repo_events`#

This dlt.source fetches repository events incrementally, dispatching them to separate tables based on event type. It loads new events only and appends them to tables.

Note: GitHub allows retrieving up to 300 events for public repositories, so frequent updates are recommended for active repos.

@dlt.source(max_table_nesting=2)
def github_repo_events(
    owner: str, name: str, access_token: str = None
) -> DltResource:
   ...

owner: Refers to the owner of the repository.

name: Denotes the name of the repository.

access_token: Optional classic or fine-grained access token. If not provided, calls are made anonymously.

max_table_nesting=2 sets the maximum nesting level to 2.

Resource `repo_events`#

This dlt.resource function serves as the resource for the github_repo_events source. It yields repository events as data items.

dlt.resource(primary_key="id", table_name=lambda i: i["type"]) # type: ignore
def repo_events(
    last_created_at: dlt.sources.incremental[str] = dlt.sources.incremental(
        "created_at", initial_value="1970-01-01T00:00:00Z", last_value_func=max
    )
) -> Iterator[TDataItems]:
   ...

primary_key: Serves as the primary key, instrumental in preventing data duplication.

table_name: Routes data to appropriate tables based on the data type.

last_created_at: This parameter determines the initial value for "last_created_at" in dlt.sources.incremental. If no value is given, the default "initial_value" is used. The function "last_value_func" determines the most recent 'created_at' value.

Customization#

Create your own pipeline#

If you wish to create your own pipelines, you can leverage source and resource methods from this verified source.

Configure the pipeline by specifying the pipeline name, destination, and dataset as follows:

pipeline = dlt.pipeline(
    pipeline_name="github_pipeline", # Use a custom name if desired
    destination="duckdb", # Choose the appropriate destination (e.g., duckdb, redshift, post)
    dataset_name="github_reaction_data" # Use a custom name if desired
)

To read more about pipeline configuration, please refer to our documentation.

To load all the data from the repo on issues, pull requests, their comments, and reactions, you can do the following:
```
load_data = github_reactions("duckdb", "duckdb")
load_info = pipeline.run(load_data)
print(load_info)
```
Here, "duckdb" is the owner of the repository and the name of the repository.

To load only the first 100 issues, you can do the following:

load_data = github_reactions("duckdb", "duckdb", max_items=100)
load_info = pipeline.run(load_data.with_resources("issues"))
print(load_info)

You can fetch and process repo events data incrementally. It loads all data during the first run and incrementally in subsequent runs.
```
load_data = github_repo_events(
    "duckdb", "duckdb", access_token=os.getenv("ACCESS_TOKEN_ENV_VAR")
)
load_info = pipeline.run(load_data)
print(load_info)
```
It is optional to use access_token or make anonymous API calls.