The eval framework#

An eval allows us to test our tasks by evaluating them on a local dataset. It is defined by 3 components:

• dataset: holds a list of records optionally labelled with their expected output.
• task: a function that takes in a record and produces an output.
• evaluator: an object whose evaluation method takes in a task output and compares it to the expected output to produce metrics.

Running an eval#

To run an eval, run the eval.py file a module:

poetry run python -m agent.tasks.auto_label.evals.eval

Example#

Please look at tasks/wants_help for an up-to-date example!

For this task, we first define a dataset by listing GitHub links and their expected outputs in data/build_data.json:

[
    {
        "link": "https://github.com/run-llama/llama_index/issues/6905",
        "expected": true
    },
    {
        "link": "https://github.com/run-llama/llama_index/issues/6822",
        "expected": false
    }
]

We then use the GitHub dataset builder to create examples from this list in build_dataset.py, saving them to data/dataset.json:

existing = Dataset.load_from_json(output_path)
updated = build_github_dataset(build_data_path, existing)
updated.save_to_json(output_path)

The task defined in eval.py simply makes a call to the main function with a custom prompt:

def wants_help_task(task_input: TaskInput) -> TaskOutput:
    output = wants_help(
        query=task_input.record.input,
        system_prompt=prompts["eval_system_prompt.j2"].render(),
    )
    return TaskOutput(output=output)

We finally define and run the eval by instantiating its class in the same file:

# We specify a runs path so we can compare against previous runs and save a new one there
dataset_path = Path(__file__).parent / "data/dataset.json"
runs_path = Path(__file__).parent / "runs"
dataset = Dataset.load_from_json(dataset_path)

# We choose the pre-made BinaryEvaluator for this task
eval = Eval(
    name="wants_help",
    dataset=dataset,
    task=wants_help_task,
    evaluator=BinaryEvaluator(),
    run_directory=runs_path,
)

# If no name is specified, the run will not be saved
eval.run("baseline")

Collecting examples#

This framework is built around the curation of quality examples to assess our tasks.
Here are a few ways you can find and add examples:

• Go through GitHub issues or PRs on repos we're active on. Copy and paste links to use in combination with the GitHub dataset builder.
• Go through LangSmith traces. See the behavior you're interested in by filtering by your task's name under the LLM calls section.
• Manually create examples. Useful for narrow-range tasks where we can easily engineer edge-cases, like robustness to adversarial attacks.

You can see how many records we have for each task by running the following script:

poetry run python scripts/print_eval_info.py