Build comparison table

[NISH1001]

Changelog

Major

• now all the Metric.compute(...) method returns evalem._base.structures.MetricResult object
• evalem._base.pipelines.NamedSimpleEvaluationPipeline is added that -- verbose -- provides naming to each pipeline. (If name is not provided, auto-naming scheme is done)
• evalem.misc.utils.build_comparison_table(...) utility function is added (see usage section). (In the future, need to figure out a better way to package this comparison. Currently, the comparison is very basic/naive and assumes the same set of inputs and references are passed through all the evaluation pipes.)

Minor

• evalem.nlp.metrics.RougeMetric actual score is now computed via averaging (rouge1, rougeL, rougeLsum) in the MetricResult.score object.
• tests cases are refactored to remove redundancies
• evalem._base.abc.InstantCountMixin mixin is added to provide usability for auto-naming/auto-id objects of any downstream class.
• GitHub action now includes pytest coverage via python -m pytest --cov evalem --verbose tests/
• evalem._base.evaluators.Evauator calls/invocations now returns only a list of results instead of dictionary since we're now using evalem.MetricResult data structure for downstream result ingestion.

Usage

from evalem import NamedSimpleEvaluationPipeline

from evalem.nlp.models import QuestionAnsweringHFPipelineWrapper
from evalem.nlp.evaluators import QAEvaluator
from evalem.nlp.metrics import BertScore, RougeMetric, MeteorMetric

from evalem.misc.utils import build_comparison_table

# vanilla model
wrapped_model = QuestionAnsweringHFPipelineWrapper(device="mps")

# nasa-v6 model
wrapped_model_2 = QuestionAnsweringHFPipelineWrapper.from_onnx(
    model="tmp/onnx/nasa-v6/",
    tokenizer="tmp/onnx/nasa-v6/",
    device="mps"
)

# common metrics
evaluators_common = [
    QAEvaluator(),
    BertScore(device="mps"),
#     BartScore(device="mps"),
    RougeMetric(),
    MeteorMetric()
]

# pipeline no. 1
eval_pipe = NamedSimpleEvaluationPipeline(
    model=wrapped_model,
    evaluators=evaluators_common,
    name="distilbert"
)

# pipeline no. 2
eval_pipe_2 = NamedSimpleEvaluationPipeline(
    model=wrapped_model_2,
    evaluators=evaluators_common,
    name="nasa-v6-onnx"
)

# get comparison table
results = build_comparison_table(
    eval_pipe, eval_pipe_2,
    inputs=list(data[["context", "question"]].T.to_dict().values()),
    references=data["answer_text"].to_list(),
)

print(results)

[NISH1001]

Re:
We used the changes to evaluate askathon dataset at:
#35

[muthukumaranR]

how are bertscore negative scores to be interpreted? does that affect the averaging done here?

[NISH1001]

The final actual bertscore isn't compute via mean. Only the f1/precision/recall are averaged -- hence no negative scores for these -- because of how Jury returns per-instance scores for these as well. (We could possibly also remove these as these will also be computed by evalem.F1Metric. etc)