Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Suggestion: parallelize low complexity masking #39

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Suggestion: parallelize low complexity masking #39

wants to merge 3 commits into from

Commits on Sep 26, 2018

  1. Suggestion: parallelize repeat masking 

    I was trying to build `nt` database and discovered that it would take roughly 12 hours to mask all of it in one thread. This example modification uses GNU Parallel to speed up repeat masking in proportion to the number of CPUs. Perhaps this code will need to be changed before being merged (e. g. checking that GNU Parallel is available in PATH, of taking into account user setting of whether to use multiple threads), so this is only a draft and a suggestion.
    @tskir
    tskir committed Sep 26, 2018
    Configuration menu
    Copy the full SHA
    582338f View commit details
    Browse the repository at this point in the history

  2. Do not keep order during parallelization 

    Since keeping order of records (when comparing before/after low complexity masking) is not necessary, it is better to disable this option, since it degrades parallelism somewhat.
    @tskir
    tskir committed Sep 26, 2018
    Configuration menu
    Copy the full SHA
    c621b4c View commit details
    Browse the repository at this point in the history

  3. Fix error when passing $MASKER variable

    @tskir
    tskir committed Sep 26, 2018
    Configuration menu
    Copy the full SHA
    62f2eb2 View commit details
    Browse the repository at this point in the history