Standardize object (de)-serialization in LLMeter #7
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available: N/A
While exploring possible cost model implementations, I was slowed/frustrated by the lack of standardisation in how LLMeter serializes objects to file & loads them back.
There are various objects we'd like to be able to save to file and reconstruct: including data like Run
Result
s,RunConfig
s, individualInvocationResponse
s - but also more complex, implementation-dependent objects like anEndpoint
,Tokenizer
, and hopefully maybe one day a cost model?Description of changes:
This change proposes to define a standard
from_dict()
,from_json()
,from_file()
,to_dict()
,to_json()
,to_file()
interface (seeserde.py
), stick to it as closely as possible, and re-use implementations as much as possible between serializable objects in LLMeter. In particular at a high level, it proposes:_type
attribute, normally corresponding to the__class__.__name__
.Endpoint.from_file("endpoint.json", alt_classes={"MyCoolGeminiEndpoint": MyCoolGeminiEndpoint})
, where endpoint.json could also refer to one of the LLMeter built-in typesTokenizer
to the same unknown-implementation-loading pattern asEndpoint
, because they're solving the same problem (defining a few built-in implementations, but wanting flexible support for custom ones too)JSONable
classes nesting theirJSONable
fields directly in their output JSON by default, and treat separating things out as an override behaviour. For example: A cost model would serialize to one JSON file even though it's composed of multiple (built-in or custom-defined) cost dimensions... But there are some cases (like RunConfig.responses) that might continue to be extracted to avoid undue bulk.This is an early draft: I haven't narrowed down the exact extent of breaking change in file output formats yet, or fixed tests, or added tests for the new components... But raising for visibility so we can discuss 😄
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.