Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add word filters #116

Merged
merged 3 commits into from
Jul 1, 2023
Merged

Add word filters #116

merged 3 commits into from
Jul 1, 2023

Conversation

mocobeta
Copy link
Owner

@mocobeta mocobeta commented Jul 1, 2023

This adds two new token filters.

  1. WordStopFilter removes tokens whose surface form is listed in the stop words list.
  2. WordKeepFilter keeps tokens whose surface form is listed in the keep words list.

See the tests for the usage.

@coveralls
Copy link

Coverage Status

coverage: 87.054% (+0.2%) from 86.878% when pulling 02279ee on feat/stop-keep-token-filters into 88f1d66 on master.

@coveralls
Copy link

coveralls commented Jul 1, 2023 •
edited
Loading

Coverage Status

coverage: 87.054% (+0.2%) from 86.878% when pulling f2556c3 on feat/stop-keep-token-filters into 88f1d66 on master.

@mocobeta mocobeta merged commit 8f92b61 into master Jul 1, 2023
@mocobeta mocobeta deleted the feat/stop-keep-token-filters branch July 1, 2023 09:59
@mocobeta mocobeta added this to the 0.5.0 milestone Jul 1, 2023
mocobeta added a commit that referenced this pull request Jul 1, 2023
@mocobeta
Revert "Add word filters (#116)"
a8d8b60 
This reverts commit 8f92b61.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
0.5.0
Development

Successfully merging this pull request may close these issues.
2 participants
@mocobeta @coveralls