Skip to content

Issues: aws-samples/foundation-model-benchmarking-tool

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

28 Open 29 Closed
28 Open 29 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Fails when testing Haiku on AWS Bedrock
#247 opened Nov 28, 2024 by NidPlays
Allow backup/ failover regions to be specified
#232 opened Nov 7, 2024 by jimburtoft
Add support for Bedrock Custom Import Models
#224 opened Oct 26, 2024 by dickren123
@madhurprash
1
miss warm up phase
#213 opened Oct 8, 2024 by lxning
Add support for non AWS models to be benchmarked
#203 opened Sep 28, 2024 by madhurprash
1
@madhurprash
add hallucination metric to model evals on FMBench
#170 opened Aug 14, 2024 by madhurprash
Add retry handling for throttling errors during inference
#157 opened Aug 1, 2024 by madhurprash
UI Support for FMBench
#155 opened Jul 31, 2024 by madhurprash
@madhurprash
TypeError: unsupported format string passed to NoneType.__format__
#154 opened Jul 24, 2024 by prasad-nair-amd
Unable to locate credential issue
#153 opened Jul 23, 2024 by prasadwrites
1
Remove the config_filepath.txt as a config reference for developer workflow
#150 opened Jul 18, 2024 by madhurprash
@madhurprash
Is it possible to run FMBench on an EC2 instance on endpoints currently not supported on Sagemaker.
#120 opened Jun 14, 2024 by prasad-nair-amd
@aarora79
5
Reports only show information per instance instead of per endpoint
#114 opened Jun 7, 2024 by jimburtoft
Loosen dependencies for more flexible installation
#105 opened May 31, 2024 by athewsey
@aarora79
1
FMBench to support benchmarking for embedding models enhancement New feature or request
#33 opened Mar 25, 2024 by madhurprash
@madhurprash
Need for Integration of FMBench Tool with External 3rd party Synthetic Data Generation Tools for Benchmarking usecase.
#32 opened Mar 16, 2024 by akramIOT
Compare different models for the same dataset enhancement New feature or request
#30 opened Feb 17, 2024 by aarora79
@aarora79
Try FMBench with instance count set to > 1 to see how scaling impacts latency and transactions per minute enhancement New feature or request
#29 opened Feb 17, 2024 by aarora79
Provide config file for FLAN-T5 out of the box enhancement New feature or request
#28 opened Feb 15, 2024 by aarora79
@madhurprash
Allow different experiments in the same config file to use different regions good first issue Good for newcomers
#27 opened Feb 15, 2024 by aarora79
@4mp
The business summary plot in the report needs to have a caption for disclaimer bug Something isn't working
#26 opened Feb 15, 2024 by aarora79
The README gives an impression that this tool is comparing instance types when it should be serving stacks documentation Improvements or additions to documentation
#25 opened Feb 15, 2024 by aarora79
@aarora79
Add support for models hosted on platforms other than SageMaker enhancement New feature or request
#22 opened Feb 15, 2024 by aarora79
Add documentation for the config file on readthedocs.io good first issue Good for newcomers
#21 opened Feb 15, 2024 by aarora79
@4mp
Code cleanup needed to replace the notebooks with regular python files help wanted Extra attention is needed
#20 opened Feb 15, 2024 by aarora79
@aarora79
ProTip! Type g i on any issue or pull request to go back to the issue listing page.