Skip to content

Latest commit

 

History

History

sdk_triage_bot

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
bin
bin
 
 
lib
lib
 
 
test
test
 
 
tool
tool
 
 
.gitignore
.gitignore
 
 
README.md
README.md
 
 
analysis_options.yaml
analysis_options.yaml
 
 
pubspec.yaml
pubspec.yaml
 
 

README.md

What's this?

A LLM based triage automation system for the dart-lang/sdk repo. It processes new issues filed against the repo and triages them in the same manner that a human would. This includes:

  • re-summarizing the issue for clarity
  • assigning the issues to an area- label (first line triage)

Bot trigger and entry-point

This bot is generally triggered by a GitHub workflow listening for new issues on the dart-lang/sdk repo.

See https://github.com/dart-lang/sdk/blob/main/.github/workflows/issue-triage.yml.

Overview

The general workflow of the tool is:

  • download the issue information (existing labels, title, first comment)
  • ask Gemini to summarize the issue (see prompts)
  • ask Gemini to classify the issue (see prompts)
  • create a comment on the issue (@dart-github-bot) with the summary; apply any labels produced as part of the classification

Tuning

We create a tuned Gemini model in order to improve the performance of classification. This involves:

  • downloading recent, already triaged issues (~800 issues)
  • writing them in a format suitable for tuning (either .csv or .jsonl)
  • tuning via the Gemini APIs; this gives us a new model name to use when calling Gemini (e.g., model: 'tunedModels/sdk-triage-tuned-prompt-1l96e2n')

See tool/create_tuning_data.dart.