Task#

This task simply in two solution and assesses their similarity, outputting SIMILAR, NOT_SIMILAR, and NOT_APPLICABLE if the "solutions" are not comparable.
Since we'll run this on Dosu and human messages marked resolves_thread, the NOT_APPLICABLE option is mainly to flag resolutions that do not offer an actual solution (e.g. "closed by this #XXX", "duplicate of #XXX", or "setting this aside for now").

Eval#

The eval set is built from a query that finds all resolution messages belonging to threads containing both a human resolution and a shadowed Dosu resolution.
The query is found in data/raw/raw_data_query.sql and was run against prod on August 1st, 2024.

Datasets#

• raw/raw_data.json: query results saved as JSON
• build_data.json: preprocessed query results by data/raw/create_build_data.py
• unlabeled_dataset.json: usable eval set with no labels, created by build_dataset.py
• llm_labels.json: labels generated by the LLM, created by from the 20240731154554:baseline_no_evaluation.json run
• llm_labels_with_similarities.json: same as the above, with similarity scores computed by analysis.ipynb