Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log model responses directly to file and reuse them for debugging #181

Closed
2 tasks
bauersimon opened this issue Jun 12, 2024 · 1 comment
Closed
2 tasks
Assignees
Labels
duplicate This issue or pull request already exists enhancement New feature or request
Milestone

Comments

@bauersimon
Copy link
Member

Goal, be able to use exactly 1:1 responses from a previous run to debug the evaluation logic.

  • log model responses directly to files (either on provider query response level or generate test level)
  • add dummy model that takes these files and responds accordingly (essentially mimicking/replaying the original model responses)
@bauersimon bauersimon added the enhancement New feature or request label Jun 12, 2024
@bauersimon bauersimon added this to the v0.6.0 milestone Jun 12, 2024
@bauersimon bauersimon mentioned this issue Jun 12, 2024
45 tasks
@ahumenberger ahumenberger added the duplicate This issue or pull request already exists label Jul 4, 2024
@ahumenberger
Copy link
Contributor

Duplicate of #204. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
duplicate This issue or pull request already exists enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants