v0.0.3-alpha

This release adds a simple pipeline abstraction for existing ModelWrapper, Metric and Evaluator.

Changelog

Major

• evalem.pipelines.SimpleEvaluationPipeline is added that wraps existing model wrappers, metrics and evaluators to run in single coherent abstraction. see PR
• More semantic metrics like Bleu, ROUGE, METEOR are added. see PR

Minor

• Test suites are refactored. For example, the model and pipeline tests suites are parameterized through conftest.py paradigm.

Usage

from evalem.pipelines import SimpleEvaluationPipeline
from evalem.models import TextClassificationHFPipelineWrapper
from evalem.evaluators import TextClassificationEvaluator

# can switch to any implemented wrapper
model = TextClassificationHFPipelineWrapper()

# can switch to other evaluator implementation
evaluator = TextClassificationEvaluator()

# initialize
eval_pipe = SimpleEvaluationPipeline(model=model, evaluators=evaluator)

results = pipe(inputs, references)

# or
results = pipe.run(inputs, references)