[Feature Request] Directly model costs in LLMeter

[athewsey]

While LLMeter currently measures a range of latency & throughput metrics, it doesn't yet relate these performance attributes to the actual cost of running different model types. Users are interested in connecting performance test results with financial metrics for a more holistic view of trade-offs between models and solutions.

This feature requires some careful design because pricing is complicated (even for one cloud, let alone across all the different options out there we support). The most likely solution would be providing flexible tools for users to inject their own models of cost drivers, with strong examples / documentation showing how to use it. Going further to dynamically look prices up from APIs would be nice, but likely difficult for many scenarios.