chore(l2g): update inclusion features list with newer QTLs

…apping results (#518)

* feat: dataflow decompress prototype (#501)

* chore: commit susie results gist

* feat(study_index): add `tissueFromSourceId` to schema and make `traitFromSource` nullable

* fix: bug and linting fixes in new eqtl ingestion step

* perf: config bugfixes and performance improvements

* perf: remove data persistance to avoid executor failure

* perf: load susie results for studies of interest only

* perf: collect locus for leads only and optimise partitioning cols

* feat: parametrise methods to include

* feat: run full dag

* test: add tests

* fix: reorder test inputs

* docs: update eqtl catalogue docs

* fix: correct typos in tests docstrings

* refactor: change mqtl_quantification_methods to mqtl_quantification_methods_blacklist

* feat: studyId is based on measured trait and not on gene

* feat: credible set lead is the variant with highest pip

* feat(studies): change logic in _identify_study_type to extract qtl type based on quantization method

* refactor: externalise reading logic to source classes

* chore: add mqtl_quantification_methods_blacklist to yaml config

* docs: update docs

* fix(dag): pass bucket name to GCSDeleteBucketOperator

* refactor(coloc): move get_logsum function to common utils

* feat(studylocus): add calculate_credible_set_log10bf and use it for eqtlcat credible sets

* fix: credible sets dataset is too large and cant be broadcasted

* fix(dag): use GCSDeleteObjectsOperator instead of GCSDeleteBucketOperator

* fix: correct typo

* fix: correct typo

* chore: update ruff pre-commit and rules

* refactor: to address None type and remove typeignore

---------

Co-authored-by: Daniel Suveges <[email protected]>

* fix: pr labeler patterns

* fix: change bug for fix

* revert: as per reviewers will

* feat: include single snp overlaps in coloc calcs

* chore: added unit test for coloc

* feat: include single snp overlaps in coloc calcs

* chore: added unit test for coloc

* fix: check for differences in test results

* fix: revert changes to single snp coloc tests for other pr

* fix: revert single snp change

* chore: restructure test folder

* fix: compare differences in results rather than exact match

* chore: test description

* fix: allow for coloc when only one snp is overlapping

* chore: include tests for h0 and h1/2

* fix: added pseudocount variable to coloc

* fix: pseudocount as a Coloc attribute

---------

Co-authored-by: David Ochoa <[email protected]>

chore: small updates to accomodate GWAS Catalog for feb release

Bumps [ruff](https://github.com/astral-sh/ruff) from 0.2.0 to 0.3.2.
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](astral-sh/ruff@v0.2.0...v0.3.2)

---
updated-dependencies:
- dependency-name: ruff
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [mypy](https://github.com/python/mypy) from 1.8.0 to 1.9.0.
- [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md)
- [Commits](python/mypy@v1.8.0...1.9.0)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat(sumstat imputation): adding class for sumstat imputation

* feat: added test

* docs: update summary statistics imputation description

* Update src/gentropy/method/sumstat_imputation.py

Co-authored-by: Irene López <[email protected]>

* Update src/gentropy/method/sumstat_imputation.py

Co-authored-by: Irene López <[email protected]>

* Update src/gentropy/method/sumstat_imputation.py

Co-authored-by: Irene López <[email protected]>

* Update src/gentropy/method/sumstat_imputation.py

Co-authored-by: Irene López <[email protected]>

* Update docs/python_api/methods/sumstat_imputation.md

Co-authored-by: Irene López <[email protected]>

* Update src/gentropy/method/sumstat_imputation.py

Co-authored-by: Irene López <[email protected]>

* Update src/gentropy/method/sumstat_imputation.py

Co-authored-by: Irene López <[email protected]>

* fix: changes according to comment

* Update src/gentropy/method/sumstat_imputation.py

Co-authored-by: Irene López <[email protected]>

* Update src/gentropy/method/sumstat_imputation.py

Co-authored-by: Irene López <[email protected]>

* Update src/gentropy/method/sumstat_imputation.py

Co-authored-by: Irene López <[email protected]>

* Update src/gentropy/method/sumstat_imputation.py

Co-authored-by: Irene López <[email protected]>

* Update src/gentropy/method/sumstat_imputation.py

Co-authored-by: Irene López <[email protected]>

* fix: fixing tests

---------

Co-authored-by: Irene López <[email protected]>

Co-authored-by: Irene López <[email protected]>

* chore: pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.3.0 → v0.3.2](astral-sh/ruff-pre-commit@v0.3.0...v0.3.2)
- [github.com/alessandrojcm/commitlint-pre-commit-hook: v9.11.0 → v9.13.0](alessandrojcm/commitlint-pre-commit-hook@v9.11.0...v9.13.0)
- [github.com/pre-commit/mirrors-mypy: v1.8.0 → v1.9.0](pre-commit/mirrors-mypy@v1.8.0...v1.9.0)
- [github.com/python-poetry/poetry: 1.8.2 → 1.8.0](python-poetry/poetry@1.8.2...1.8.0)

* ci: keep poetry s latest release

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Irene López <[email protected]>

…ts/gentropy into ds_3238_update_ETL_DAG

feat(dag): add data transfer task group for release process

Bumps [deptry](https://github.com/fpgmaas/deptry) from 0.12.0 to 0.14.0.
- [Release notes](https://github.com/fpgmaas/deptry/releases)
- [Changelog](https://github.com/fpgmaas/deptry/blob/main/CHANGELOG.md)
- [Commits](fpgmaas/deptry@0.12.0...0.14.0)

---
updated-dependencies:
- dependency-name: deptry
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [mkdocstrings-python](https://github.com/mkdocstrings/python) from 1.8.0 to 1.9.0.
- [Release notes](https://github.com/mkdocstrings/python/releases)
- [Changelog](https://github.com/mkdocstrings/python/blob/main/CHANGELOG.md)
- [Commits](mkdocstrings/python@1.8.0...1.9.0)

---
updated-dependencies:
- dependency-name: mkdocstrings-python
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* refactor(l2g): select features of interest in data outside trainer

* chore(l2g): log annotated gold standards in w&b

* fix: update test_train

* refactor(l2g): streamline coloc feature factory

* perf(l2g): make joins in _get_vep_features lighter

* fix(l2g): use weighted scores for _get_vep_features

* refactor(l2g): minor improvements

* perf(l2g): adapt session to set partition number for shuffling to 800

---------

Co-authored-by: Daniel Suveges <[email protected]>

* chore: pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.3.2 → v0.3.3](astral-sh/ruff-pre-commit@v0.3.2...v0.3.3)
- [github.com/python-poetry/poetry: 1.8.2 → 1.8.0](python-poetry/poetry@1.8.2...1.8.0)

* fix: poetry lock adjustments

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: David Ochoa <[email protected]>

* refactor(l2g): streamline coloc feature factory

* perf(l2g): make joins in _get_vep_features lighter

* fix(l2g): use weighted scores for _get_vep_features

* refactor(l2g): minor improvements

* perf(l2g): adapt session to set partition number for shuffling to 800

* feat(l2g): distance features based on weighted score

* docs: update docs

* refactor(l2g): select features of interest in data outside trainer

* chore(l2g): log annotated gold standards in w&b

* fix: update test_train

* fix(l2g): typo in _get_tss_distance_features with no consequences

* perf: refactored window based clumping

* fix: gwas catalog clump step adjusted

* fix: broadcast logic

* revert: coalesce instruction

* fix: add alias

* docs: enhance docs

* test: accidentally removed test

* test: rescue missing test

* chore: remove unused step

* refactor: increased modularisation

* feat: restructure data model

* test: structural test for annotate function

* fix: prevent locus column could be duplicated

* chore: up-to-date with pre-commit

* feat: notebook to run qc metrics each release

* feat: added additional checks as discussed

* feat: additional checks

* fix: remove matplotlib from dependencies

* fix: address comments

* fix: pytest for coloc unit tests

* fix: removed unused coloc tests

* test(coloc): add coloc semantic test (#538)

* test(coloc): define fixtures and parametrise coloc tests

* test(coloc): compare dfs with assert_frame_equal

* fix: remove unused threshold variable

---------

Co-authored-by: Irene López <[email protected]>
Co-authored-by: David Ochoa <[email protected]>

#553)

* perf(l2g): rewrite ColocalisationFactory._get_max_coloc_per_credible_set

* perf(l2g): more elegant rewrite ColocalisationFactory._get_max_coloc_per_credible_set

* chore: adapt tests to changes

* refactor: import colocalisation attributes

* fix(coloc): fillna doesnt fill nested data

* test(coloc): added test_coloc_no_logbf (semantic)

* revert(ecaviar): revert accidental changes

* feat: add the step class for fine-mapping

* test: adding test for susie to studylocus converter

* chore: fix the class description

* chore: answering comments

---------

Co-authored-by: Daniel Suveges <[email protected]>

* test: adding test for pairwiseLD

* feat: adding ld matrix extraction

* chore: merge from dev

* feat: index and block matrix extraction for studyLocus

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* chore: updating some test files to gentropy

* chore: updating tests

* chore: updating pairwise_ld_schema for tests

* chore: updating pairwise_ld tests

* chore: fix ld_pairwise tests

* chore: fix pairwise_ld tests

* chore: fix tests

* chore: fix tests

* chore: fixing typing for tests

* chore: fixing tests

* chore: fixing ld tests

* Update src/gentropy/dataset/study_index.py

Co-authored-by: Daniel Suveges <[email protected]>

* feat: moving functions to their appropriate locations and improving logic

* fix: optimise conversion of BM to NumPy

* feat: updating get_locus_index to allow for just chromosome and position inputs

* fix: suggested changes

* Update study_index.py

* fix: changes to datasource/gnomad/ld.py

* feat: updated method for ld_index extraction

* fix: sorting idx in hail

---------

Co-authored-by: Daniel Suveges <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* test: adding test for pairwiseLD

* feat: adding ld matrix extraction

* chore: merge from dev

* feat: index and block matrix extraction for studyLocus

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* chore: updating some test files to gentropy

* chore: updating tests

* chore: updating pairwise_ld_schema for tests

* chore: updating pairwise_ld tests

* chore: fix ld_pairwise tests

* chore: fix pairwise_ld tests

* chore: fix tests

* chore: fix tests

* chore: fixing typing for tests

* chore: fixing tests

* chore: fixing ld tests

* Update src/gentropy/dataset/study_index.py

Co-authored-by: Daniel Suveges <[email protected]>

* feat: moving functions to their appropriate locations and improving logic

* fix: optimise conversion of BM to NumPy

* feat: updating get_locus_index to allow for just chromosome and position inputs

* fix: suggested changes

* Update study_index.py

* fix: changes to datasource/gnomad/ld.py

* feat: add the draft of finemapper fucntion

* feat: updated method for ld_index extraction

* fix: changing input

* fix: adding fillter by studyId

* fix: sorting idx in hail

* feat: add fine-mapping of one study_locus_row

* fix: small fix in majpop

* fix: small fixes in function

* fix: using more spark before converting to pandas

* fix: fix in test

---------

Co-authored-by: Daniel Suveges <[email protected]>
Co-authored-by: Daniel Considine <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel-Considine <[email protected]>

* feat: susie_finemapper_ss_gathered()

* fix: docstring updates

* feat: add fm step with carma and sumstat imputation

* fix: adding log

* fix: fixing carma

* fix: resolving conflict

* fix: resolve conflicts with dev v2

* fix: siliencing FutureWarning in Carma

)

* feat: add benchmarking for fine-mapping using Alzheimer as example

* fix: small fix in notebook

…ctions (#574)

)

* feat: adding init to finemapping step

* fix: removing some commented lines

* chore: fixing indents

* fix: schema

* feat: changing output path to include studyLocusId mapped

---------

Co-authored-by: Yakov <[email protected]>

Co-authored-by: David Ochoa <[email protected]>

…er functions (#582)

* feat(sumstat qc): class for GWAS sumstat QC

* feat: small fixes

* feat: modifying sumarry_statistics class

* feat: adding qc_pz_check

* fix: test fix

* feat: adding se_n check

* chore: fixing functions

* feat: adding gc lambda check

* test: adding tests

* fix: fixing tests

* fix: fixing tests

* fix: fixing tests and adding rounding

* chore: pre-commit auto fixes [...]

* feat(sumstat qc): adding new method class

* test: fix tests

* fix(sumstat qc): fixing tests

* fix: remove number_of_snps from susmtat class

* test: add multiple studyId test

* docs: adding documentation for sumstat QC

* fix: fixes according to comments

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

…ithin-study overlaps (#587)

* feat: functionality added to StudyLocus.find_overlaps() for finding within-study overlaps

* feat: removal of secondary credible sets at the same region as overlaps

* fix: defining the join condition to make the code tidier

Co-authored-by: Irene López <[email protected]>

Bumps [deptry](https://github.com/fpgmaas/deptry) from 0.14.0 to 0.16.1.
- [Release notes](https://github.com/fpgmaas/deptry/releases)
- [Changelog](https://github.com/fpgmaas/deptry/blob/main/CHANGELOG.md)
- [Commits](fpgmaas/deptry@0.14.0...0.16.1)

---
updated-dependencies:
- dependency-name: deptry
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Ochoa <[email protected]>

Bumps [lxml](https://github.com/lxml/lxml) from 5.1.0 to 5.2.1.
- [Release notes](https://github.com/lxml/lxml/releases)
- [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt)
- [Commits](lxml/lxml@lxml-5.1.0...lxml-5.2.1)

---
updated-dependencies:
- dependency-name: lxml
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Ochoa <[email protected]>

* chore: pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.3.3 → v0.4.1](astral-sh/ruff-pre-commit@v0.3.3...v0.4.1)
- [github.com/pre-commit/pre-commit-hooks: v4.5.0 → v4.6.0](pre-commit/pre-commit-hooks@v4.5.0...v4.6.0)
- [github.com/alessandrojcm/commitlint-pre-commit-hook: v9.13.0 → v9.15.0](alessandrojcm/commitlint-pre-commit-hook@v9.13.0...v9.15.0)
- [github.com/econchick/interrogate: 1.5.0 → 1.7.0](econchick/interrogate@1.5.0...1.7.0)

* fix: poetry lock is not up-to-date

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: David Ochoa <[email protected]>
Co-authored-by: David Ochoa <[email protected]>

Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 3.6.0 to 3.7.0.
- [Release notes](https://github.com/pre-commit/pre-commit/releases)
- [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md)
- [Commits](pre-commit/pre-commit@v3.6.0...v3.7.0)

---
updated-dependencies:
- dependency-name: pre-commit
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Ochoa <[email protected]>

Bumps [pytest-cov](https://github.com/pytest-dev/pytest-cov) from 4.1.0 to 5.0.0.
- [Changelog](https://github.com/pytest-dev/pytest-cov/blob/master/CHANGELOG.rst)
- [Commits](pytest-dev/pytest-cov@v4.1.0...v5.0.0)

---
updated-dependencies:
- dependency-name: pytest-cov
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Ochoa <[email protected]>

Bumps [python-semantic-release](https://github.com/python-semantic-release/python-semantic-release) from 9.1.0 to 9.4.1.
- [Release notes](https://github.com/python-semantic-release/python-semantic-release/releases)
- [Changelog](https://github.com/python-semantic-release/python-semantic-release/blob/master/CHANGELOG.md)
- [Commits](python-semantic-release/python-semantic-release@v9.1.0...v9.4.1)

---
updated-dependencies:
- dependency-name: python-semantic-release
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.4.1 → v0.4.2](astral-sh/ruff-pre-commit@v0.4.1...v0.4.2)
- [github.com/alessandrojcm/commitlint-pre-commit-hook: v9.15.0 → v9.16.0](alessandrojcm/commitlint-pre-commit-hook@v9.15.0...v9.16.0)
- [github.com/pre-commit/mirrors-mypy: v1.9.0 → v1.10.0](pre-commit/mirrors-mypy@v1.9.0...v1.10.0)

* chore: it bumps poetry version

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: David Ochoa <[email protected]>

* feat: add purity qc metrics to fine-mapping

* fix: adding cs_lbf_thr to susie_finemapper_one_studylocus_row

* fix: schema

* feat: check for if no overlapping variants in LD index

* fix: removing unnecessary else statement

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.4.2 → v0.4.3](astral-sh/ruff-pre-commit@v0.4.2...v0.4.3)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Bumps [pymdown-extensions](https://github.com/facelessuser/pymdown-extensions) from 10.7 to 10.8.1.
- [Release notes](https://github.com/facelessuser/pymdown-extensions/releases)
- [Commits](facelessuser/pymdown-extensions@10.7...10.8.1)

---
updated-dependencies:
- dependency-name: pymdown-extensions
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Irene López <[email protected]>

Bumps [interrogate](https://github.com/econchick/interrogate) from 1.5.0 to 1.7.0.
- [Release notes](https://github.com/econchick/interrogate/releases)
- [Changelog](https://github.com/econchick/interrogate/blob/master/docs/changelog.rst)
- [Commits](econchick/interrogate@1.5.0...1.7.0)

---
updated-dependencies:
- dependency-name: interrogate
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [pytest](https://github.com/pytest-dev/pytest) from 8.1.0 to 8.2.0.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](pytest-dev/pytest@8.1.0...8.2.0)

---
updated-dependencies:
- dependency-name: pytest
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) from 3.5.0 to 3.6.1.
- [Release notes](https://github.com/pytest-dev/pytest-xdist/releases)
- [Changelog](https://github.com/pytest-dev/pytest-xdist/blob/master/CHANGELOG.rst)
- [Commits](pytest-dev/pytest-xdist@v3.5.0...v3.6.1)

---
updated-dependencies:
- dependency-name: pytest-xdist
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>