Error in creating the standard Kraken2 database

[Nathalia-Cavichiolli]

Hello I'm trying to create the standard Kraken 2 database, with the following command:

kraken2-build --standard --threads 16 --db kraken-Sdb

I'm getting this:

Step 1/2: Performing rsync file transfer of requested files
Rsync file transfer complete.
Step 2/2: Assigning taxonomic IDs to sequences
All files processed, cleaning up extra sequence files... done, library complete.
Masking low-complexity regions of downloaded library... done.
Step 1/2: Performing rsync file transfer of requested files
Rsync file transfer complete.
Step 2/2: Assigning taxonomic IDs to sequences
All files processed, cleaning up extra sequence files... done, library complete.
Masking low-complexity regions of downloaded library... done.
Step 1/2: Performing rsync file transfer of requested files
Rsync file transfer complete.
Step 2/2: Assigning taxonomic IDs to sequences
All files processed, cleaning up extra sequence files... done, library complete.
Masking low-complexity regions of downloaded library... done.
Downloading plasmid files from FTP...
gzip: plasmid.6.1.genomic.fna.gz: invalid compressed data--format violated

gzip: plasmid.7.1.genomic.fna.gz: invalid compressed data--format violated

gzip: plasmid.8.1.genomic.fna.gz: invalid compressed data--format violated

gzip: plasmid.9.1.genomic.fna.gz: invalid compressed data--format violated

Any thoughts how I can solve this problem?

Thank you Nathalia

[saras224]

Hi!
Even I am trying to build Kraken2 standard database but I got the similar error.
command:
kraken2-build --standard --db standard

error:
Downloading taxonomy tree data...rsync: failed to connect to ftp.ncbi.nlm.nih.gov (130.14.250.7): Connection timed out (110)
rsync: failed to connect to ftp.ncbi.nlm.nih.gov (2607:f220:41e:250::10): Network is unreachable (101)
rsync: failed to connect to ftp.ncbi.nlm.nih.gov (2607:f220:41e:250::7): Network is unreachable (101)
rsync: failed to connect to ftp.ncbi.nlm.nih.gov (2607:f220:41e:250::11): Network is unreachable (101)
rsync: failed to connect to ftp.ncbi.nlm.nih.gov (2607:f220:41e:250::12): Network is unreachable (101)
rsync: failed to connect to ftp.ncbi.nlm.nih.gov (2607:f220:41e:250::13): Network is unreachable (101)
rsync error: error in socket IO (code 10) at clientserver.c(127) [Receiver=3.1.3]

I notice other people too getting the same error but could not find any solution for it.
Thank You
Saraswati

[jenniferlu717]

@saras224 thats a connectivity issue. It may be that ncbi's ftp site is not working at the moment so you might just have to wait, or you can try using --use-ftp

@Nathalia-Cavichiolli thats an issue i havent seen before. Are you using the newest kraken2 version to download the files?

[saras224]

Thank you for responding Jennifer! I tried --use-ftp with kraken2 version 2.0.8 and it did not work so I read other issues and figured out that was a version issue so I downloaded latest kraken2 version which is 2.1.1 and installed then used --use-ftp flag while building and this time it started to build but while processing it was giving this:
.
.
.
Processed 10579/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_001275535.1_ViralProj167781_genomic.fna.gz: No such file or directory
Processed 10580/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_001966295.1_ViralProj362114_genomic.fna.gz: No such file or directory
Processed 10581/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_001646275.1_ViralProj321681_genomic.fna.gz: No such file or directory
Processed 10582/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_002593665.1_ASM259366v1_genomic.fna.gz: No such file or directory
Processed 10583/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_001960515.1_ViralProj361763_genomic.fna.gz: No such file or directory
Processed 10584/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_002820225.1_ASM282022v1_genomic.fna.gz: No such file or directory
Processed 10585/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_000883875.1_ViralProj38097_genomic.fna.gz: No such file or directory
Processed 10586/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_000973475.1_ViralProj281405_genomic.fna.gz: No such file or directory
Processed 10587/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_002830305.1_ASM283030v1_genomic.fna.gz: No such file or directory
Processed 10588/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_002709945.1_ASM270994v2_genomic.fna.gz: No such file or directory
Processed 10589/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_012271565.1_ASM1227156v1_genomic.fna.gz: No such file or directory
Processed 10590/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_004133365.1_ASM413336v1_genomic.fna.gz: No such file or directory
Processed 10591/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_004790435.1_ASM479043v1_genomic.fna.gz: No such file or directory
Processed 10592/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_002997575.1_ASM299757v1_genomic.fna.gz: No such file or directory
Processed 10593/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_000861365.1_ViralProj15329_genomic.fna.gz: No such file or directory
Processed 10594/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_000891575.1_ViralProj62661_genomic.fna.gz: No such file or directory
Processed 10595/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_001958715.1_ViralMultiSegProj361855_genomic.fna.gz: No such file or directory
Processed 10596/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_011757035.1_ASM1175703v1_genomic.fna.gz: No such file or directory
Processed 10597/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_000910255.1_ViralProj209209_genomic.fna.gz: No such file or directory
Processed 10598/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_002277885.1_ViralProj309565_genomic.fna.gz: No such file or directory
Processed 10599/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_003330045.1_ASM333004v1_genomic.fna.gz: No such file or directory
Processed 10600/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_000916595.1_ViralMultiSegProj240572_genomic.fna.gz: No such file or directory
Processed 10601/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_002149745.1_ViralProj308007.1_genomic.fna.gz: No such file or directory
Processed 10602/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_001755205.1_ViralProj344620_genomic.fna.gz: No such file or directory
Processed 10603/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_000854565.1_ViralProj14963_genomic.fna.gz: No such file or directory
Processed 10604/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_002830565.1_ASM283056v1_genomic.fna.gz: No such file or directory
Processed 10605/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_001722825.1_ViralProj342451_genomic.fna.gz: No such file or directory
Processed 10606/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_003613295.1_ASM361329v1_genomic.fna.gz: No such file or directory
Processed 10607/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_001461385.2_ASM146138v2_genomic.fna.gz: No such file or directory
Processed 10608/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_002003795.1_ViralProj371622_genomic.fna.gz: No such file or directory
Processed 10609/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_000906795.1_ViralProj193978_genomic.fna.gz: No such file or directory
Processed 10610/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_000867765.1_ViralProj18131_genomic.fna.gz: No such file or directory
Processed 10611/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_002029595.1_ViralProj378771_genomic.fna.gz: No such file or directory
Processed 10612/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_000880495.1_ViralProj31287_genomic.fna.gz: No such file or directory
Processed 10613/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_003571865.1_ASM357186v1_genomic.fna.gz: No such file or directory
Processed 10614/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_001964275.1_ViralProj362039_genomic.fna.gz: No such file or directory
Processed 10615/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_001881935.1_ViralProj353669_genomic.fna.gz: No such file or directory
Processed 10616/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_004129715.1_ASM412971v1_genomic.fna.gz: No such file or directory
Processed 10617/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_000886075.1_ViralProj42145_genomic.fna.gz: No such file or directory
Processed 10618/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_001924095.1_ViralProj358927_genomic.fna.gz: No such file or directory
Processed 10619/10620 projects (118 sequences, 3.22 Mbp)...gzip: all/GCF_004134145.1_ASM413414v1_genomic.fna.gz: No such file or directory
Processed 10620 projects (118 sequences, 3.22 Mbp)... done.
All files processed, cleaning up extra sequence files... done, library complete.
Masking low-complexity regions of downloaded library... done.
Downloading plasmid files from FTP...
gzip: plasmid.2.1.genomic.fna.gz: invalid compressed data--format violated

gzip: plasmid.3.1.genomic.fna.gz: invalid compressed data--format violated

gzip: plasmid.4.1.genomic.fna.gz: invalid compressed data--format violated

I gave this command: kraken2-build --standard --threads 12 --db STANDARD-DB --use-ftp

Thanks

[Nathalia-Cavichiolli]

@Nathalia-Cavichiolli thats an issue i havent seen before. Are you using the newest kraken2 version to download the files?

Hi @jenniferlu717

Thank you for your replay

I've installed using conda (conda install -c bioconda kraken2). The version is v2.1.1

[jenniferlu717]

Just letting you guys know that I'm checking the standard build right now. I'll comment again if I figure out what is going on

[jenniferlu717]

@Nathalia-Cavichiolli im not seeing the same issue with the plasmid files. It could be that they had messed up the plasmid download at the time.....I would try running the standard download again, but I don't have a solution for you otherwise.

Since it looks like its only happening with plasmid, I wouldn't restart from scratch but check to see which of these does not exist:
archaea bacteria human plasmid UniVec_Core viral

and download that:
kraken2-build --download-library plasmid --db kraken-Sdb
and then run the build from there:

kraken2-build --build --db kraken-Sdb --threads 16

That should save you from re-doing the full build.

[jenniferlu717]

@saras224 Can you retry downloading the viral and plasmid libraries? I'm sorry about all of the issues but I suspect the plasmid errors should be fixed by now as my standard build did not have those problems.

The errors with downloading the viral library seems to do with the ftp-download script which I'm still trying to debug at the moment. I'm not certain why it is doing that but its having a problem downloading the files correctly and then unzipping them.

[puva]

Hi, I'm having the same issue. As archaea, bacteria and viral were downloaded correctly, I tried with:
kraken2-build --download-library plasmid --db K2standardDB --use-ftp
and
kraken2-build --download-library plasmid --db K2standardDB
but they both failed with:

gzip: plasmid.1.1.genomic.fna.gz: invalid compressed data--format violated
gzip: plasmid.2.1.genomic.fna.gz: invalid compressed data--format violated
gzip: plasmid.3.1.genomic.fna.gz: invalid compressed data--format violated
gzip: plasmid.4.1.genomic.fna.gz: invalid compressed data--format violated
gzip: plasmid.5.1.genomic.fna.gz: invalid compressed data--format violated
gzip: plasmid.6.1.genomic.fna.gz: invalid compressed data--format violated
gzip: plasmid.7.1.genomic.fna.gz: invalid compressed data--format violated
gzip: plasmid.8.1.genomic.fna.gz: invalid compressed data--format violated
gzip: plasmid.9.1.genomic.fna.gz: invalid compressed data--format violated

Could it be that the original file at NCBI is corrupted?

[Nathalia-Cavichiolli]

@saras224 Can you retry downloading the viral and plasmid libraries? I'm sorry about all of the issues but I suspect the plasmid errors should be fixed by now as my standard build did not have those problems.

The errors with downloading the viral library seems to do with the ftp-download script which I'm still trying to debug at the moment. I'm not certain why it is doing that but its having a problem downloading the files correctly and then unzipping them.

Thank you @saras224 .
Unfortunately I'm still getting the same error message.

[theo-allnutt-bioinformatics]

I'm having this problem with all downloads using kraken2. I suspect ncbi has changed something and the download script no longer works.

[wresch]

I have the same problem with the plasmid database with kraken 2.1.1. To debug, here is the output from the wget
run by download_genomic_library.sh:

 4► wget --no-remove-listing --spider ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/plasmid/
Spider mode enabled. Check if remote file exists.
--2021-02-24 20:53:27--  ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/plasmid/
Resolving dtn04-e0 (dtn04-e0)... 10.1.200.240
Connecting to dtn04-e0 (dtn04-e0)|10.1.200.240|:3128... connected.
Proxy request sent, awaiting response... 200 Gatewaying
Length: unspecified [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

this is with wget 1.14 behind a squid proxy (using ftp-over-http)

My workaround, which is admittedly ugly:

curl -s ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/plasmid/ | perl -nle 'print  "$1"  while(/<a\s+href\s*=\s*"([^"]+.gz)"/g)' | sort -u

[jenniferlu717]

Apologies everyone, for the rsync issue, it may be a temporary connection issue that MAY resolve if you try again at a later time. However, if not, there is some sort of server setting that is preventing your server from connecting to NCBI via rsync.

In that case, the only option is to use the "--use-ftp" download option. However, we recently discovered an issue with the ftp-downloads that we are working to resolve alongside the NCBI staff.

[saras224]

Hi @jenniferlu717
I tried re-building the standard database using the --use-ftp flag but it did not help. The same error keeps on coming "no file or directory" while doing for plasmids and viruses. Is it due to the network issue or some technical issue that will get resolved in near future?

Thanks
Saraswati Awasthi

[laugon10]

HI @jenniferlu717
I am having just the same problem.

Downloading plasmid files from FTP...
gzip: plasmid.4.1.genomic.fna.gz: invalid compressed data--format violated

gzip: plasmid.7.1.genomic.fna.gz: invalid compressed data--format violated

gzip: plasmid.8.1.genomic.fna.gz: invalid compressed data--format violated

I hope there is a solution soon,
Thanks!
Laura

[wresch]

I didn't really explain my workaround very well, so here is a patch:

--- download_genomic_library.sh.orig    2021-03-04 07:47:23.979836000 -0500
+++ download_genomic_library.sh 2021-02-24 22:06:31.238936000 -0500
@@ -62,7 +62,12 @@
     rm -f library.f* plasmid.*
     ## This is staying FTP only D/L for now
     1>&2 echo -n "Downloading plasmid files from FTP..."
-    wget -q --no-remove-listing --spider $FTP_SERVER/genomes/refseq/plasmid/
+    ## WR: fixing the broken wget
+    # wget -q --no-remove-listing --spider $FTP_SERVER/genomes/refseq/plasmid/
+    curl -s $FTP_SERVER/genomes/refseq/plasmid/ \
+        | perl -nle 'print  "$1"  while (/<a\s+href\s*=\s*"([^"]+.gz)"/g)' \
+        | sort -u \
+        > .listing
     if [ -n "$KRAKEN2_PROTEIN_DB" ]; then
       awk '{ print $NF }' .listing | perl -ple 'tr/\r//d' | grep '\.faa\.gz' > manifest.txt
     else

For me the issue was that the script wasn't generating the list of plasmid files to download. the --use-ftp flag doesn't actually seem to affect the plasmid sequence fetch step. The patch i came up with uses curl + perl to generate the list instead. Worked for me, might not be generally the best way of doing it.

[mirifax]

I actually have the same problem as @saras224 for the bacteria and viral but not the human database. Even though I can use them further, they are not complete ( when I compare to refseq). Thanks for all the effort!

[laugon10]

This just has worked for me. I download this libraries independently. And then build. The building is going on but it seems that the problem is solved.

Thanks a lot!
Laura

@jenniferlu717

@Nathalia-Cavichiolli im not seeing the same issue with the plasmid files. It could be that they had messed up the plasmid download at the time.....I would try running the standard download again, but I don't have a solution for you otherwise.

Since it looks like its only happening with plasmid, I wouldn't restart from scratch but check to see which of these does not exist:
archaea bacteria human plasmid UniVec_Core viral

and download that:
kraken2-build --download-library plasmid --db kraken-Sdb
and then run the build from there:

kraken2-build --build --db kraken-Sdb --threads 16

That should save you from re-doing the full build.

[fanninpm]

Would it be possible to do a sanity check against the md5 hash they provide?

[zhou-sumei]

Hi, I met the same problem:

gzip: plasmid.1.1.genomic.fna.gz: invalid compressed data--format violated

gzip: plasmid.10.1.genomic.fna.gz: invalid compressed data--format violated

gzip: plasmid.3.1.genomic.fna.gz: invalid compressed data--format violated

Here I have two questions:

**The first one: about kraken2-build --standard ** I read these comments and replies, I tried again ,but it takes long time to run again( --thread 84, take >10 hour then the progress faild ), is there any solution to avoid the the full build if I already have the result of last time? By the way, the --use-ftp doesn't work either for me:
here is the error info

Step 1/2: Performing ftp file transfer of requested files
Step 2/2: Assigning taxonomic IDs to sequences
All files processed, cleaning up extra sequence files... done, library complete.
Masking low-complexity regions of downloaded library... done.
Step 1/2: Performing ftp file transfer of requested files
Timeout at /picb/evolgen/users/zhousumei/miniconda/envs/meta_read_als/lib/5.26.2/Net/FTP.pm line 583.

Can I do something to fix these error and run the kraken2-build smoothly?

**The second one: about kraken2-build --download-library ; kraken2-build --build **
At the same time, I tried to download and built five libraries seperately,It taked long time to finish plasmid library,and I did see the three .k2d files, but they don‘t have any prefix to indicate that they are the plasmid library, If I download the virus, bacterial libraries in the same folder, will the plasmid library to be covered?

Any suggestions mean a lot for me ! Thank you very much !

[jflucier]

I didn't really explain my workaround very well, so here is a patch:

--- download_genomic_library.sh.orig    2021-03-04 07:47:23.979836000 -0500
+++ download_genomic_library.sh 2021-02-24 22:06:31.238936000 -0500
@@ -62,7 +62,12 @@
     rm -f library.f* plasmid.*
     ## This is staying FTP only D/L for now
     1>&2 echo -n "Downloading plasmid files from FTP..."
-    wget -q --no-remove-listing --spider $FTP_SERVER/genomes/refseq/plasmid/
+    ## WR: fixing the broken wget
+    # wget -q --no-remove-listing --spider $FTP_SERVER/genomes/refseq/plasmid/
+    curl -s $FTP_SERVER/genomes/refseq/plasmid/ \
+        | perl -nle 'print  "$1"  while (/<a\s+href\s*=\s*"([^"]+.gz)"/g)' \
+        | sort -u \
+        > .listing
     if [ -n "$KRAKEN2_PROTEIN_DB" ]; then
       awk '{ print $NF }' .listing | perl -ple 'tr/\r//d' | grep '\.faa\.gz' > manifest.txt
     else

For me the issue was that the script wasn't generating the list of plasmid files to download. the --use-ftp flag doesn't actually seem to affect the plasmid sequence fetch step. The patch i came up with uses curl + perl to generate the list instead. Worked for me, might not be generally the best way of doing it.

This patch worked well for me! tks @wresch wresch

[icotto25]

Someone solved this issue? I have tried to do all the suggested changes here and in #529 but nothing works.

[SergeyBaikal]

Someone solved this issue? I have tried to do all the suggested changes here and in #529 but nothing works.

All databases are here
https://benlangmead.github.io/aws-indexes/k2

[ichauchcc]

Someone solved this issue? I have tried to do all the suggested changes here and in #529 but nothing works.

All databases are here https://benlangmead.github.io/aws-indexes/k2

Thanks Sergey, this is really helpful.

[quliping]

Hello I'm trying to create the standard Kraken 2 database, with the following command:

kraken2-build --standard --threads 16 --db kraken-Sdb

I'm getting this:

Step 1/2: Performing rsync file transfer of requested files Rsync file transfer complete. Step 2/2: Assigning taxonomic IDs to sequences All files processed, cleaning up extra sequence files... done, library complete. Masking low-complexity regions of downloaded library... done. Step 1/2: Performing rsync file transfer of requested files Rsync file transfer complete. Step 2/2: Assigning taxonomic IDs to sequences All files processed, cleaning up extra sequence files... done, library complete. Masking low-complexity regions of downloaded library... done. Step 1/2: Performing rsync file transfer of requested files Rsync file transfer complete. Step 2/2: Assigning taxonomic IDs to sequences All files processed, cleaning up extra sequence files... done, library complete. Masking low-complexity regions of downloaded library... done. Downloading plasmid files from FTP... gzip: plasmid.6.1.genomic.fna.gz: invalid compressed data--format violated

gzip: plasmid.7.1.genomic.fna.gz: invalid compressed data--format violated

gzip: plasmid.8.1.genomic.fna.gz: invalid compressed data--format violated

gzip: plasmid.9.1.genomic.fna.gz: invalid compressed data--format violated

Any thoughts how I can solve this problem?

Thank you Nathalia

You can try this, and it's worked for me:
#515 (comment)
Here is an example for the path of the script file which need to be modified:
/public/apps/anaconda/envs/kraken2/share/kraken2-2.1.3-1/libexec/download_genomic_library.sh

[ChillarAnand]

After running into a lot of issues, I build a small tool to easily create any kraken2 database index with a single command.

You can build a standard db with single command.

$ pip install kraken-db-builder

$ kdb --db-type standard

You can stop the process at any point and when you re-run it, it resumes from where it has stopped.

It is also way faster than kraken2-build.

You can give it a try and I will be happy answer any questions.

[jarrodscott]

Hello @ChillarAnand . Thanks for sharing this amazing tool! I am running like this:

kdb --db-type standard --threads 30 --cache-dir .

And I continue to get an error like this:

subprocess.CalledProcessError: Command 'ncbi-genome-download --section refseq \
--format fasta --assembly-level complete --retries 3 --parallel 30 --progress-bar plasmid' \
returned non-zero exit status 254.

In line 23 of kdb.py I tried moving plasmid to the end of the list, so human is next. But I get the same error. Any thoughts. And is the a better place to post this question? Thanks!!

[jarrodscott]

Ah I found the repo. Adding my question there.