This project enables you to run Braintrust evals as part of your CI/CD workflow in Github, using Github actions. To use this action, simply include the following step in an action file:

- name: Run Evals
  uses: braintrustdata/eval-action@v1
  with:
    api_key: ${{ secrets.BRAINTRUST_API_KEY }}
    runtime: node

You can configure the following variables:

• api_key: Your Braintrust API key.
• root: The root directory containing your evals (defaults to '.'). The root directory must either have node or python configured.
• paths: Specific paths, relative to the root, containing evals you'd like to run.
• runtime: Either node or python

The action runs braintrust eval and collects experiment results, which are posted as a comment in the PR alongside a link to Braintrust. For example:

Say Hi Bot (HEAD-1714341466)

To see examples of fully configured templates, see the examples directory:

• node with npm
• node with pnpm