import Header from './_source-info-header.md';

Chess.com#

Chess.com is an online platform that offers services for chess
enthusiasts. It includes online chess games, tournaments, lessons, and more.

Resources that can be loaded using this verified source are:

Name	Description
players_profiles	retrieves player profiles for a list of player usernames
players_archives	retrieves URL to game archives for specified players
players_games	retrieves players' games that happened between start_month and end_month

Setup guide#

Grab credentials#

Chess.com API is a public API that does not
require authentication or including secrets in secrets.toml.

Initialize the verified source#

To get started with your data pipeline, follow these steps:

Enter the following command:
```
dlt init chess duckdb
```
This command will initialize
the pipeline example
with Chess.com as the source and
duckdb as the destination.
If you'd like to use a different destination, simply replace duckdb with the name of your
preferred destination.
After running this command, a new directory will be created with the necessary files and
configuration settings to get started.

Add credentials#

To add credentials to your destination, follow the instructions in the
destination documentation. This will ensure that your data is
properly routed to its final destination.

For more information, read the General Usage: Credentials.

Run the pipeline#

Before running the pipeline, ensure that you have installed all the necessary dependencies by
running the command:
```
pip install -r requirements.txt
```
You're now ready to run the pipeline! To get started, run the following command:
```
python chess_pipeline.py
```
Once the pipeline has finished running, you can verify that everything loaded correctly by using
the following command:
```
dlt pipeline <pipeline_name> show
```
For example, the pipeline_name for the above pipeline example is chess_pipeline, you may also
use any custom name instead.

For more information, read the guide on how to run a pipeline.

Sources and resources#

dlt works on the principle of sources and
resources.

Source `source`#

This is a dlt.source function for the Chess.com API named "chess", which returns a sequence of
DltResource objects. We'll discuss these in subsequent sections as resources.

dlt.source(name="chess")
def source(
    players: List[str], start_month: str = None, end_month: str = None
) -> Sequence[DltResource]:
    return (
        players_profiles(players),
        players_archives(players),
        players_games(players, start_month=start_month, end_month=end_month),
        players_online_status(players),
    )

players: This is a list of player usernames for which you want to fetch data.

start_month and end_month: These optional parameters specify the time period for which you want
to fetch game data (in "YYYY/MM" format).

Resource `players_profiles`#

This is a dlt.resource function, which returns player profiles for a list of player usernames.

@dlt.resource(write_disposition="replace")
def players_profiles(players: List[str]) -> Iterator[TDataItem]:

    @dlt.defer
    def _get_profile(username: str) -> TDataItem:
        return _get_path_with_retry(f"player/{username}")

    for username in players:
        yield _get_profile(username)

players: This is a list of player usernames for which you want to fetch profile data.

It uses the @dlt.defer decorator to enable parallel run in a thread pool.

Resource `players_archives`#

This is a dlt.resource function, which returns a URL to game archives for specified players.

@dlt.resource(write_disposition="replace", selected=False)
def players_archives(players: List[str]) -> Iterator[List[TDataItem]]:
    ...

players: This is a list of player usernames for which you want to fetch archives.

selected=False: This parameter means that this resource is not selected by default when the pipeline
runs.

Resource `players_games`#

This incremental resource takes data from players and returns games for the last month if not
specified otherwise.

@dlt.resource(write_disposition="append")
def players_games(
    players: List[str], start_month: str = None, end_month: str = None
) -> Iterator[TDataItems]:
    # gets a list of already checked (loaded) archives.
    checked_archives = dlt.current.resource_state().setdefault("archives", [])
    yield {} # return your retrieved data here

players: This is a list of player usernames for which you want to fetch games.

The list checked_archives is used to load new archives and skip the ones already loaded. It uses state
to initialize a list called "checked_archives" from the current resource
state.

Resource `players_online_status`#

The players_online_status is a dlt.resource function that checks the current online status of multiple chess players. It
retrieves their username, status, last login date, and check time.

Customization#

Create your own pipeline#

If you wish to create your own pipelines, you can leverage source and resource methods from this verified source.

To create your data loading pipeline for players and load data, follow these steps:

Configure the pipeline by specifying the pipeline name, destination, and dataset as follows:

pipeline = dlt.pipeline(
    pipeline_name="chess_pipeline", # Use a custom name if desired
    destination="duckdb", # Choose the appropriate destination (e.g., duckdb, redshift, post)
    dataset_name="chess_players_games_data", # Use a custom name if desired
)

To read more about pipeline configuration, please refer to our documentation.

To load the data from all the resources for specific players (e.g., for November), you can utilize the source method as follows:

# Loads games for Nov 2022
source_instance = chess_source(
    ["magnuscarlsen", "vincentkeymer", "dommarajugukesh", "rpragchess"],
    start_month="2022/11",
    end_month="2022/11",
)

Use the method pipeline.run() to execute the pipeline.

info = pipeline.run(source_instance)
# print the information on data that was loaded
print(info)

To load data from specific resources like "players_games" and "player_profiles", modify the above code as:

info = pipeline.run(source_instance.with_resources("players_games", "players_profiles"))
# print the information on data that was loaded
print(info)