Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reproducing Knowedit results #440

Open
gkruddl opened this issue Dec 3, 2024 · 3 comments
Open

Reproducing Knowedit results #440

gkruddl opened this issue Dec 3, 2024 · 3 comments
Labels
question Further information is requested

Comments

@gkruddl
Copy link

gkruddl commented Dec 3, 2024

          Hi, after updating the code, there is no ['post']['fluency'] results in the output file. 

I am not getting the fluency results . Also, the results doesn't match with the table..Below is my output for ICE on Wikibio dataset

Metrics Summary: {'pre': {'rewrite_acc': 0.6363403936421949, 'locality': {'Relation_Specificity_acc': 1.0}}, 'post': {'rewrite_acc': 0.9999999961042716, 'locality': {'Relation_Specificity_acc': 0.8011074301515478}}}

Originally posted by @gkruddl in #433 (comment)

@zxlzr zxlzr added the question Further information is requested label Dec 4, 2024
@zxlzr
Copy link
Contributor

zxlzr commented Dec 4, 2024

We will handle this asap.

@zxlzr
Copy link
Contributor

zxlzr commented Dec 5, 2024

We are fixing this issue and will update the code this week.

We apologize for any inconvenience caused. Our team is limited in size, and this tool is quite extensive. Additionally, new models and methods are emerging continuously.

@littlefive5
Copy link
Collaborator

Hello, I have updated the code so that it can compute fluency. I also fixed some bugs when computing the evaluation metric for portability and locality.
I will update the results in the paper.
Thanks a lot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

3 participants