Skyline transfer

.github/workflows/dryrun.yaml

@@ -6,6 +6,25 @@ on:
 
 jobs:
   dry_run_paired_end:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+      with:
+        submodules: 'true'
+    - uses: docker://snakemake/snakemake:v7.32.4
+    - name: Dry Run with test data
+      run: |
+        docker run -h cn0000 -v $PWD:/opt2 -w /opt2 snakemake/snakemake:v7.32.4 /bin/bash -c \
+        "source get_submods.sh; pip install -r requirements.txt; ./weave run --sheetname paired_end.csv -s /opt2/.tests/paired_end -o /opt2/.tests/dry_run_out --local --dry-run /opt2/.tests/paired_end"
+    - name: View the pipeline config file
+      run: |
+        echo "Generated config file for pipeline...." && cat $PWD/.tests/dry_run_out/paired_end/.config/config_job_0.json
+    - name: Lint Snakefile
+      continue-on-error: true
+      run: |
+        docker run -e SNK_CONFIG='/opt2/.tests/dry_run_out/paired_end/.config/config_job_0.json' -v $PWD:/opt2 snakemake/snakemake:stable snakemake --lint -s /opt2/workflow/Snakefile -d /opt2/workflow || \
+        echo 'There may have been a few warnings or errors. Please read through the log to determine if its harmless.'
+  dry_run_latest:
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v2
@@ -30,10 +49,10 @@ jobs:
     - uses: actions/checkout@v2
       with:
         submodules: 'true'
-    - uses: docker://snakemake/snakemake:stable
+    - uses: docker://snakemake/snakemake:v7.32.4
     - name: Dry Run with test data
       run: |
-        docker run -h cn0000 -v $PWD:/opt2 -w /opt2 snakemake/snakemake:stable /bin/bash -c \
+        docker run -h cn0000 -v $PWD:/opt2 -w /opt2 snakemake/snakemake:v7.32.4 /bin/bash -c \
         "source get_submods.sh; pip install -r requirements.txt;./weave run --sheetname single_end.csv -s /opt2/.tests/single_end -o /opt2/.tests/dry_run_out --local --dry-run /opt2/.tests/single_end"
     - name: View the pipeline config file
       run: |

.gitignore

@@ -1,6 +1,6 @@
 logs
-.tests/illumnia_demux/dry_run_out
 .snakemake
 site
 output
-.tests/dry_run_out/*
+.tests/illumnia_demux/dry_run_out
+.tests/dry_run_out

config/skyline.json

@@ -0,0 +1,20 @@
+{
+    "sif": "/data/openomics/SIFs/",
+    "mounts": {
+        "kaiju": {
+            "to": "/opt/kaiju",
+            "from": "/data/openomics/references/weave/kaiju/kaiju_db_nr_euk_2023-05-10",
+            "mode": "ro"
+        },
+        "kraken2" : {
+            "to": "/opt/kraken2",
+            "from": "/data/openomics/references/weave/kraken2/k2_pluspfp_20230605",
+            "mode": "ro"
+        }, 
+        "fastq_screen" : {
+            "to": "/fdb/fastq_screen/FastQ_Screen_Genomes",
+            "from": "/data/openomics/references/weave/FastQ_Screen_Genomes",
+            "mode": "ro"
+        }
+    }
+}

docs/execution.md

@@ -0,0 +1,117 @@
+**weave** is capable of automatically distributing its pipeline jobs across a slurm cluster. The context for it's initial execution can be varied as well.
+
+The context is also centrally related to the configuration and setup of a particular cluster. Right now weave is configured to work with NIH clusters **skyline**, **biowulf**, and **bigsky**.
+
+
+Typical contexts of execution include:
+
+# srun (real time execution) (non-interactive)
+
+The **weave** pipeline can be triggered from a head node in a non-interactive fashion:
+
+## Bigsky/Skyline
+
+!!! Note
+    Dependency files for skyline and bigsky differ <br />
+    **Bigsky: `/gs1/RTS/OpenOmics/bin/dependencies.sh`** <br />
+    **Skyline: `/data/openomics/bin/dependencies.sh`**
+
+```bash
+source ${dependencies}
+srun --export=ALL "weave run [keyword args] ${run_id}"
+```
+
+!!! Note
+    srun <a href="https://slurm.schedmd.com/srun.html#OPT_export">by default</a> exports all environmental variables from the executing environment and `--export=ALL` can be left off
+
+## Biowulf
+
+```bash
+srun --export=ALL "module load snakemake singularity; weave run [keyword args] ${run_id}"
+```
+
+# srun (real time execution) (interactive)
+
+## Bigsky/Skyline
+
+!!! Note
+    Dependency files for skyline and bigsky differ <br />
+    **Bigsky: `/gs1/RTS/OpenOmics/bin/dependencies.sh`** <br />
+    **Skyline: `/data/openomics/bin/dependencies.sh`**
+
+```bash
+> # <head node>
+srun --pty bash
+> # <compute node>
+source ${dependencies}
+weave run [keyword args] ${run_id}
+```
+
+## Biowulf
+
+```bash 
+> # <head node>
+sinteractive
+> # <compute node>
+module purge
+module load snakemake singularity
+weave run [keyword args] ${run_id}
+```
+
+Biowulf uses environmental modules to control software. After executing the above you should see a message similar to:
+
+> [+] Loading snakemake  7.XX.X on cnXXXX<br />
+> [+] Loading singularity  4.X.X  on cnXXXX<br />
+
+# sbatch (later time execution)
+
+## Bigsky/Skyline
+
+### sbatch tempalte
+```bash title="<b>bigsky-skyline sbatch template</b>"
+#!/bin/bash
+#SBATCH --job-name=<job_name>
+#SBATCH --export=ALL
+#SBATCH --time=01-00:00:00
+#SBATCH --cpus-per-task=1
+#SBATCH --ntasks=1
+#SBATCH --mem=8g
+#SBATCH --output=<stdout_file>_%j.out
+source ${dependencies}
+weave run \
+-s /sequencing/root/dir \
+-o output_dir \
+<run_id>
+```
+
+This above script can serve as a template to create an sbatch script for weave. Update the psuedo-variables in the script to suit your particular needs then execute using sbatch command:
+
+```bash
+sbatch weave_script.sbatch
+```
+
+## Biowulf
+
+### sbatch tempalte
+```bash title="<b>biowulf sbatch template</b>"
+#!/bin/bash
+#SBATCH --job-name=<job_name>
+#SBATCH --export=ALL
+#SBATCH --time=01-00:00:00
+#SBATCH --cpus-per-task=1
+#SBATCH --ntasks=1
+#SBATCH --mem=8g
+#SBATCH --output=<stdout_file>_%j.out
+module purge
+module load snakemake singularity
+weave run \
+-s /sequencing/root/dir \
+-o output_dir \
+<run_id>
+```
+
+Same sbatch execution as bigsky/skyline.
+
+```bash
+sbatch weave_script.sbatch
+```

mkdocs.yml

@@ -109,5 +109,6 @@ nav:
     - weave run: usage/run.md
     - weave cache: usage/cache.md
   - Installation: install.md
+  - Execution context: execution.md
   - Reference: ref/reference.md
   - License: license.md

requirements.txt

@@ -1,6 +1,3 @@
-pandas
-requests
-terminaltables
 pyyaml
-tabulate
 progressbar
+python-dateutil

scripts/cache.py

@@ -7,7 +7,6 @@
 import subprocess
 import json
 import urllib.request
-import progressbar
 from argparse import ArgumentTypeError
 from pathlib import Path
 from urllib.parse import urlparse
@@ -81,7 +80,14 @@ def handle_download(output_dir, resource, protocol, url):
     if protocol in ('http', 'https', 'ftp'):
         info_download(f"Getting web resource {resource}...")
         fnurl = Path(urlparse(url).path).stem
-        urllib.request.urlretrieve(uri, filename=Path(output_dir, fnurl), reporthook=DownloadProgressBar())
+        try:
+            import progressbar
+            urllib.request.urlretrieve(uri, filename=Path(output_dir, fnurl), reporthook=DownloadProgressBar())
+        except ModuleNotFoundError:
+            print('Downloading resources....')
+            urllib.request.urlretrieve(uri, filename=Path(output_dir, fnurl))
+            print('....done.')
+
     elif protocol in ('docker'):
         info_download(f"Getting docker resource {resource}...")
         docker_tag = url.split('/')[-1]