Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hendrycks Math extraction rule seems too strict #2552

Open
fzyzcjy opened this issue Dec 8, 2024 · 1 comment
Open

Hendrycks Math extraction rule seems too strict #2552

fzyzcjy opened this issue Dec 8, 2024 · 1 comment
Labels
good first issue Good for newcomers validation For validation of task implementations.

Comments

@fzyzcjy
Copy link
Contributor

fzyzcjy commented Dec 8, 2024

Hi thanks for the library! It seems that the way how math answer is extracted, i.e.

indices = [pos for pos, char in enumerate(results[0]) if char == "$"]
if len(indices) <= 1:
answer = results[0]
else:
answer = results[0][indices[0] + 1 : indices[-1]]
, may be too strict.

For example, the following answer: ... some reasoning logic ... Thus the answer is \[ \boxed{42} \] is not extracted, because it is not a $.

@baberabb
Copy link
Contributor

baberabb commented Dec 9, 2024

Hi! This is based on the original code, but we can add another metric to show alongside by adding a flexible-extract filter, as gsm8k does it. PR welcome!

@baberabb baberabb added good first issue Good for newcomers validation For validation of task implementations. labels Dec 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers validation For validation of task implementations.
Projects
None yet
Development

No branches or pull requests

2 participants