Search Evals#

This package contains utilities to

• Pull data from public GitHub repositories
• Run them through the indexing pipeline locally, generating a frozen LanceDB dataset
• Generate questions using an LLM and the documents of the frozen LanceDB dataset
• Run the search worker against the LanceDB dataset with generated questions as input and compute search metrics

Generating a Dataset from a Public GitHub Repository#

To generate a LanceDB dataset and a set of generated questions:

1. Start the indexing pipeline locally with make indexing-build && make indexing-up
2. Run the dataset build script:

poetry run python -m evals.search.scripts.build_dataset --slug [SLUG] --index_request_handler_url [URL]

The output is a LanceDB dataset and a dataset of questions:

├── evals/search/data/[repo_owner]/[repo_name]/[timestamp]/
│ ├── lancedb/
│ │ ├── [data_source_id]/
│ │ ├── ...
│ ├── search/
│ │ ├── [question_id].json
│ │ ├── ...

Running an Eval#

To run the search worker on the given dataset:

poetry run python -m evals.search.scripts.run_eval --dataset [PATH]

The set of metrics will be printed at the end.

Not Supported (yet)#

• Generating data from private GitHub repositories
• Saving runs