-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
(EAI-353): Quiz question evaluation #440
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM w/ a few non-blocking comments
packages/mongodb-chatbot-evaluation/src/test/mockGenerateDataFunc.ts
Outdated
Show resolved
Hide resolved
This can be useful for evaluating how an LLM performance on the subject matter of the multiple choice questions. | ||
|
||
The prompt is based on this blog post from Hugging Face: https://huggingface.co/blog/open-llm-leaderboard-mmlu | ||
It follows the [HELM prompt format](https://huggingface.co/blog/open-llm-leaderboard-mmlu#mmlu-comes-in-all-shapes-and-sizes-looking-at-the-prompts). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it be worth splitting this out into a reusable makeHelmPrompt()
function w/ accompanying tests? Not necessary for this PR but might be useful down the line + as documentation of the format.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
HELM is just for quiz-style questions, the current makeQuizQuestionPrompt()
does what you're describing. i can refactor comments a bit to make this clearer + also export this function and test it
Jira: https://jira.mongodb.org/browse/EAI-353
Changes
In
mongodb-chatbot-evaluation
package:GeneratedData
New
model-eval
package for running evals on LLMs including:chatbot-eval-mongodb-public
)In
chatbot-eval-mongodb-public
package:Notes