Dosu#

📚 Getting Started#

Base requirements#

1. Install brew
2. Install make, if on Mac this requires xcode and running xcode-select --install to install utilities like make.
3. Install docker desktop, this will also install docker-compose which we use to run local development stacks.
4. Install gcloud cli
5. Install pulumi cli
6. Install tailscale

Install & Setup Supabase#

1. Install supabase: brew install supabase/tap/supabase
2. To develop locally, we need a local Supabase instance. Follow this guide to set one up.

Authenticate with gcloud#

You will need to authenticate your default application with gcloud for your CLI so that the default credentials file is generated. This file is mounted into our docker-compose containers and used to authenticate with Google Cloud SDK requests for local development.

gcloud auth login && gcloud auth application-default login

Depending on what service you are using you may need to do this up to once a day and restart your compose stack. Some services don't rely on gcloud, but those that do must refresh the tokens about once a day. I recommend just doing this at the start of your work day before starting any other docker services.

Env File Setup#

1. Authenticate with gcloud gcloud auth login

2. Login using the pulumi cli pulumi login

3. Run make get-dev-env - this will setup a .env.development.local file for you based on pulumi's environment.

   1. Migration NOTE: This command asks for an optional migration on your first run. If you previously had a .env.local file you can input it here to move your user-specific variables over and avoid the next step. Otherwise; just skip inputting a migration file and start fresh

4. Update default user-specific variables like your tailscale key - follow up with this after verifying you can run the unit tests:

Install local poetry environment#

While somewhat optional since we run things in docker mostly these days, it is nice to be able to run poetry commands and python locally. Some older code is run with this old poetry environment for local dev.

1. Install pyenv brew install xz pyenv and make sure to follow the pyenv shell configuration
2. Install python 3.12 via pyenv: pyenv install 3.12
3. Set it to the global Python version pyenv global <version installed>
4. Install pipx and poetry using pipx brew install pipx and then pipx install poetry==1.8.3 (Note the two equal signs). This is the current poetry version of the CI
5. Run cd backend to switch to the backend directory
6. Install dependencies
   ⧉

   poetry install --all-extras
   

Run the unit tests#

We should be able to run the unit tests with our out-of-the-box dev env:

make test-unit

Update defaults for user specific variables#

Setting these is not required for things like running unit tests, but are required for the docker compose local stack. Feel free to come back here once you'd like to do that.

1. Open the newly created .env.development.local file.

2. The first section contains variables we need to fill in, these will be preserved between runs of make get-dev-env, so once you set them you won't have to worry when you re-pull the environment.

   • GH_VISIBLE_AUTHORS: this will make the agent respond to you on github locally, replace the [your-github-handle] part with your handle, e.g devstein.
   • SLACK_USER_ID: you can worry about this later, it is an optional DevX improvement for running Slack, it is located in the Slack README's user ID section
   • TS_AUTHKEY: Head to the tailscale README and setup tailscale, make sure to generate the auth token for your containers and set TS_AUTHKEY in your .env.development.local file. (Reusable, Pre-approved options are required.).

Git environment setup#

1. Setup pre-commit hooks; while this is not 100% needed to run locally, it is good to setup the pre-commit hooks as part of your setup process. Follow the guide in the backend readme here.

Different poetry environments for development#

The backend/load_env.py file is responsible for loading environments in non-production environments. For the most part, when interacting with CI, the docker compose stack and make commands, you won't have to deal with this. It is useful to know how this works though if you want to setup any custom local environments. Typically when running poetry in a local shell you will want to set DEPLOY_ENV="local" if you want to use the local development environment files. E.g:

DEPLOY_ENV="local" poetry run python -m my_module.main

The EnvLoader is responsible for loading env files based on what DEPLOY_ENV is set. For example, when we run unit tests with make test-unit we pass DEPLOY_ENV="test-unit" to the pytest command. This results in us loading a slightly altered development environment for our unit tests, allowing us to do things like override settings for langsmith traces.

Running things locally#

Now that we have all of our dependencies installed we can run things locally. We primarily use docker to run our local development environments, but some older code still utilizes local poetry commands alongside fire (google CLI package for building CLI tools).

Makefile#

In the backend directory you will find a Makefile. This makefile allows us to run make commands. In order to get a list of all available commands you can run make help which is a self documenting help command. If you'd like to ever learn more about a make command just find it in the Makefile. They are all just simple scripts really. If you need to add one check out the adding make commands section (TODO).

This Makefile serves as the primary way to run and interact with our docker environment as everything is split up to use profiles. This allows us to not run the entire service stack locally saving us resources & dev time. Most services follow a scheme of make <service>-<action> where service is the service name, and action is the corresponding docker action such as build, up, down. Not all actions are supported currently, but feel free to add more as desired.

There are also helper commands in the makefile such as setting your discord OAuth redirect URI when testing locally, restarting just the slack bot container preventing your proxies from resetting by restarting the entire service stack, etc. Feel free to extend the Makefile as you see fit.

This means that while there is documentation about docker below, to just get started you don't really need to worry about it. All you need to do to run things is do:

make all-build
make all-up
# Alternatively target a specific service like the github-webhook
make github-webhook-build
make github-webhook-up

Docker#

Docker serves as the primary point of isolation for our development environment, as well as our production services. We build a very similar image for our local development stack (backend/Dockerfile.dev) as we do for production (backend/Dockerfile). This allows us to mimic a very similar environment to our production services, and easily share this environment across the team.

Docker-Compose#

Docker compose serves as an orchestration platform to run our containers and make defining them & their networking easy. This is all defined in our backend/docker-compose.yaml file. If you need to add a service check out our adding a docker service section (TODO).

Each service utilizes docker compose profiles to allow our make commands to target the services desired. You can always manually target these profiles with docker compose commands yourself, but this should only be used for specific use cases. If you find yourself doing this often it is a good sign it should probably be a make command.

Example: Running the agent for GitHub#

Let's go over an example of how to run the agent on GitHub and get it to respond locally.

Run the github-webhook service stack:

make github-webhook-build
make github-webhook-up

Wait for things to start-up, there is a dependency order. The PubSub emulator will start first, then once it is detected as healthy other services will start up. This also includes things like the pubsub setup containers which just execute singleton one-off run scripts to set things up in the environment; specifically local pubsub topics & subscriptions in this case. Once everything has started up you should see things be less noisy in your console and get some messages about all the serviecs starting successfully. Now we can move onto testing things out!

Head over to our test repo and create a new issue. This issue should be forwarded to the smee proxy container (legacy pre-tailscale reverse proxy). This will forward the request to the github-event-handler cloudfunction container which receives incoming github webhoook events. This in turn should kick off the thread pubsub stuff which will trigger the api container to run the agent for the request. Eventually the request will be answered and a response will be published. More in-depth docs on this flow would be helpful. You should be able to see this all happening in your console, and the dosu-dev bot should respond to your issue with a greeting, eventually followed by an answer if you let it run long enough (this can take anywhere from 1-10 mins).

📀 Git Patterns#

Install Git Hooks#

For the dosu.code-workspace file to be ignored by git, we use a git hook to automatically set the assume-unchanged flag. To install the hook, run the following command:

# Install the git hooks
git config core.hooksPath .githooks
# Make the scripts executable
chmod +x .githooks/*

Branch Naming#

1. Prefix your branch name with your name or username (devstein/). This allows us to know who the branch belongs to.
2. If there corresponding issue, include the issue number (devstein/111). This makes it easy to relate PRs to issues.
3. A few word description of the branch (devstein/111-fix-new-bug).

git checkout -b devstein/123-fix-new-bug

Commit Messages#

We don't enforce a commit message structure, but try your best to

1. Follow commitlint prefixes (fix:, feat:, refactor:, test:, etc)
2. Write short, useful commit messages
3. Small, frequent commits

git commit -m "feat: impl github issue label event handler"
git commit -m "chore(evals): add new wants help example"

🗺️ Layout#

/backend#

The Python monolith. This contains all backend logic from ingesting, indexing,
reacting to Github events, to the model logic.

/deploy#

The Pulumi IaC to deploy /backend to GCP. This is currently written in Go for
historical reasons.

To deploy the GCP backend, run ./scripts/deploy.sh. The script assumes you
have both pulumi and gcloud cli installed and authenticated.

gcloud auth application-default login

### `/frontend`

A placeholder for the [dosu.dev](https://dosu.dev) marketing page and the [app.dosu.dev](https://app.dosu.dev) web app

Deploy via Vercel.

### `/supabase`

Created via the Supabase CLI and used to manage database migrations and schema.

### `/scripts`

Ad-hoc scripts to deploy to GCP