Skip to content

Issues: symflower/eval-dev-quality

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

47 Open 49 Closed
47 Open 49 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Docker runtime is using the wrong container image bug Something isn't working
#242 opened Jul 5, 2024 by zimmski v0.6.0
@Munsio
Dump the assessments in the CSV files once they happen and not in the end of all executions enhancement New feature or request
#237 opened Jul 3, 2024 by ruiAzevedo19
5 of 7 tasks
v0.6.0
@ruiAzevedo19
1
Assess failing tests enhancement New feature or request
#235 opened Jul 2, 2024 by zimmski v0.6.0
Weight "executed code" more prominently enhancement New feature or request
#233 opened Jul 2, 2024 by zimmski v0.6.0
@bauersimon
4
Make the Knapsack.java case easier to solve for models bug Something isn't working
#230 opened Jul 1, 2024 by zimmski v0.6.0
@ahumenberger
Do not start the ollama server if not needed refactor Make code cleaner/better
#225 opened Jun 27, 2024 by Munsio
Multiple model parameter with same value result in multiple evaluations bug Something isn't working question Further information is requested
#220 opened Jun 26, 2024 by Munsio
@Munsio
unable to create temporary repository path: exec: WaitDelay expired before I/O complete bug Something isn't working flaky Nondeterministic behavior
#219 opened Jun 26, 2024 by bauersimon v0.6.0
2
Report the maximum theoretically reachable #files-executed
#215 opened Jun 25, 2024 by bauersimon
Extract model costs into log and CSVs enhancement New feature or request
#210 opened Jun 24, 2024 by bauersimon
1 of 2 tasks
v0.6.0
@ruiAzevedo19
1
Do an evaluation run for all "good open weight models" with all available quantizations and different GPUs enhancement New feature or request
#209 opened Jun 22, 2024 by zimmski v0.6.0
Interactive result comparison enhancement New feature or request
#208 opened Jun 21, 2024 by bauersimon
9 tasks
v0.6.0
Add the current commit revision to the binary, Docker image and reports enhancement New feature or request
#207 opened Jun 21, 2024 by zimmski
4 tasks
v0.6.0
@Munsio
Tool/command to combine multiple evaluations into one enhancement New feature or request
#205 opened Jun 20, 2024 by bauersimon v0.6.0
@bauersimon
1
Keep individual coverage files and LLM query/responses enhancement New feature or request
#204 opened Jun 19, 2024 by zimmski v0.6.0
@ruiAzevedo19
3
Evaluation task: Transpile enhancement New feature or request
#201 opened Jun 18, 2024 by ruiAzevedo19
26 of 29 tasks
v0.6.0
@ruiAzevedo19
2
Follow-up "Code repairing task to enable models to fix code with compilation errors" enhancement New feature or request
#200 opened Jun 17, 2024 by ruiAzevedo19
1 of 2 tasks
v0.6.0
@bauersimon
Isolation of evaluations enhancement New feature or request
#198 opened Jun 17, 2024 by Munsio
23 of 27 tasks
v0.6.0
@Munsio
Roadmap for v0.6.0 roadmap Collection of issues for a release
#195 opened Jun 17, 2024 by zimmski 100+ v0.6.0
@zimmski
Evaluation task: TDD
#194 opened Jun 17, 2024 by ahumenberger
Coverage for Java is tracked for lines, while Go is tracked for ranges bug Something isn't working
#193 opened Jun 14, 2024 by bauersimon v0.6.0
4
Deal with dependencies requested by LLMs
#174 opened Jun 10, 2024 by ahumenberger
2
Fixed timeouts for symflower unit-tests and symflower test enhancement New feature or request
#167 opened Jun 6, 2024 by Munsio v0.6.0
@Munsio
2
Support multiple evaluation tasks enhancement New feature or request
#165 opened Jun 6, 2024 by ahumenberger
7 of 9 tasks
v0.6.0
@ahumenberger
ollama_llama_server and other background processes we start must be killed on CTRL+C bug Something isn't working
#164 opened Jun 6, 2024 by zimmski v0.6.0
3
ProTip! Find all open issues with in progress development work with linked:pr.