Skip to content

Commit

Permalink
Bump up elinor-cli to v0.1.1 (#51)
Browse files Browse the repository at this point in the history
kampersanda authored Oct 26, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
1 parent 60e15ab commit a35b6be
Showing 3 changed files with 21 additions and 8 deletions.
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -4,6 +4,8 @@
[![Documentation](https://docs.rs/elinor/badge.svg)](https://docs.rs/elinor)
[![Build Status](https://github.com/kampersanda/elinor/actions/workflows/ci.yml/badge.svg)](https://github.com/kampersanda/elinor/actions)

**News: The CLI tools are now available in the [elinor-cli](./elinor-cli) directory!**

Elinor is a Rust library for evaluating information retrieval (IR) systems.
It provides a comprehensive set of metrics and statistical tests for evaluating and comparing IR systems.

2 changes: 1 addition & 1 deletion elinor-cli/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "elinor-cli"
version = "0.1.0"
version = "0.1.1"
edition = "2021"
authors = ["Shunsuke Kanda (kampersanda) <[email protected]>"]
description = "Command line tools for elinor."
25 changes: 18 additions & 7 deletions elinor-cli/README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,22 @@
# elinor-cli

[![Crates.io](https://img.shields.io/crates/v/elinor-cli)](https://crates.io/crates/elinor-cli)
[![Build Status](https://github.com/kampersanda/elinor/actions/workflows/ci.yml/badge.svg)](https://github.com/kampersanda/elinor/actions)

elinor-cli is a set of command-line tools for evaluating IR systems:

- [elinor-evaluate](#elinor-evaluate) evaluates the ranking metrics of the system.
- [elinor-compare](#elinor-compare) compares the metrics of multiple systems with statistical tests.
- [elinor-convert](#elinor-convert) converts the TREC format into the JSONL format for elinor-evaluate.

## Installation

Simply use cargo to install from crates.io.

```sh
cargo install elinor-cli
```

## elinor-evaluate

elinor-evaluate evaluates the ranking metrics of the system.
@@ -52,7 +63,7 @@ Here is example usage with sample JSONL files in the [`test-data/sample`](../tes
If you want to evaluate the Precision@3, Average Precision (AP), Reciprocal Rank (RR), and nDCG@3 metrics, run:

```sh
cargo run --release -p elinor-cli --bin elinor-evaluate -- \
elinor-evaluate \
--true-jsonl test-data/sample/true.jsonl \
--pred-jsonl test-data/sample/pred_1.jsonl \
--metrics precision@3 ap rr ndcg@3
@@ -77,7 +88,7 @@ ndcg@3 0.8286
The detailed results can be saved to a CSV file by specifying the `--output-csv` option:

```sh
cargo run --release -p elinor-cli --bin elinor-evaluate -- \
elinor-evaluate \
--true-jsonl test-data/sample/true.jsonl \
--pred-jsonl test-data/sample/pred_1.jsonl \
--output-csv test-data/sample/pred_1.csv \ # Specify output CSV path
@@ -132,7 +143,7 @@ Here is example usage with sample CSV files in the [`test-data/sample`](../test-
If you want to compare the metrics of two systems, run:

```sh
cargo run --release -p elinor-cli --bin elinor-compare -- \
elinor-compare \
--input-csvs test-data/sample/pred_1.csv \
--input-csvs test-data/sample/pred_2.csv
```
@@ -209,7 +220,7 @@ See the following documentation for more details about the statistical tests:
If you want to compare the metrics of three (or more) systems, run:

```sh
cargo run --release -p elinor-cli --bin elinor-compare -- \
elinor-compare \
--input-csvs test-data/sample/pred_1.csv \
--input-csvs test-data/sample/pred_2.csv \
--input-csvs test-data/sample/pred_3.csv
@@ -285,7 +296,7 @@ If you set `--print-mode raw`, the tables will be printed in a tab-separated for
enabling you to copy and paste them into a spreadsheet:

```sh
cargo run --release -p elinor-cli --bin elinor-compare -- \
elinor-compare \
--input-csvs test-data/sample/pred_1.csv \
--input-csvs test-data/sample/pred_2.csv \
--print-mode raw
@@ -298,7 +309,7 @@ elinor-convert converts the TREC format into the JSONL format for elinor-evaluat
For [Qrels](https://trec.nist.gov/data/qrels_eng/) files:

```sh
cargo run --release -p elinor-cli --bin elinor-convert -- \
elinor-convert \
--input-trec qrels.trec \
--output-jsonl qrels.jsonl \
--rel-type true
@@ -307,7 +318,7 @@ cargo run --release -p elinor-cli --bin elinor-convert -- \
For [Run](https://faculty.washington.edu/levow/courses/ling573_SPR2011/hw/trec_eval_desc.htm) files:

```sh
cargo run --release -p elinor-cli --bin elinor-convert -- \
elinor-convert \
--input-trec run.trec \
--output-jsonl run.jsonl \
--rel-type pred

0 comments on commit a35b6be

Please sign in to comment.