Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Risk score logic explanation #138

Closed
kadamsolanki opened this issue May 15, 2024 · 5 comments
Closed

Risk score logic explanation #138

kadamsolanki opened this issue May 15, 2024 · 5 comments
Labels
duplicate This issue or pull request already exists

Comments

@kadamsolanki
Copy link

Hey, can anyone explain me the logic of risk score calculation in toxicity scanner in input scanner, as the formula in util does not give justice to the model generated scores.

If possible please provide a detailed explanation behind adding risk_score as a metric/indicator.

Thanks,
Kadam

@asofter
Copy link
Collaborator

asofter commented May 15, 2024

Hey @kadamsolanki , thanks for reaching out.

We use threshold configured and only calculate the risk score if it's above that threshold. Then the risk score is basically how far above the confidence score from the threshold.

Hope it makes sense

@kadamsolanki
Copy link
Author

Hey @asofter, it does make sense and I was aware of this, but I wanted to say use the risk score for evaluation and there it does not make sense in case of all the scanners where we have sentence level scores. Because it will take the max score of all sentences of any 1 of the labels.

now to use the same max score for risk score calculation, does not help me as I am not sure on which sentence, or which label it is failing. So, I wanted to understand that is there a way for some sort of aggregation calculation or some confidence score at overall level for me to be clear with the model output.

@asofter
Copy link
Collaborator

asofter commented May 16, 2024

I see, your use-case is sentence-level matching instead of overall text. Do you mean something which provides avg score across all sentences instead of the highest?

@kadamsolanki
Copy link
Author

Yes.

@asofter
Copy link
Collaborator

asofter commented Jul 29, 2024

Marking as duplicate of #111

@asofter asofter closed this as completed Jul 29, 2024
@asofter asofter added the duplicate This issue or pull request already exists label Jul 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
duplicate This issue or pull request already exists
Projects
None yet
Development

No branches or pull requests

2 participants