…elease_version

fix: fix python-semantic-release version

…c_version

fix: fix python-semantic-release version (again)

…ts#613)

* feat: fine-mapping simulations class

* fix: fix of function descriptions

* Update finemapping_simulations.py

---------

Co-authored-by: Daniel-Considine <[email protected]>

…pentargets#620)

* feat: drop .coverage files from tracked files
* feat: new configuration variables for DAGs
* build(linting): resolved ruff warnings in make check
* build(airflow_config): extract additional input parameters for gnomad steps
* feat(step_config): extracted new input parameters from gnomad step configs

Configuration updates for:
- [x] ld_index_step
- [x] ld_variant_annotation_step

Both steps and underlying classes use default values derived from
StepConfig data classes as defaults, while preserving the ability
to set inputs at each stage, in case end user want to use step function
API, step cli or datasource function from API.

* refactor(types): added a file for storing library types
* feat(version_engine): add version engine to infer datasource versions
* docs: added version engine to documentation

---------

Signed-off-by: Szymon Szyszkowski <[email protected]>
Co-authored-by: Szymon Szyszkowski <[email protected]>

* Update roadmap.md -- link to gentropy tickets

* fix: fix in link

---------

Co-authored-by: Yakov Tsepilov <[email protected]>

…gets#625)

* fix(susie_finemapper): correct syntax for saving the logs df

* fix(susie_finemapper): use parquet instead of csv

---------

Co-authored-by: Yakov <[email protected]>

…utation is False (opentargets#627)

* fix(susie_finemapper): fix in the FMer in case ss_imp=False

* fix: small fix in docs

* fix(SummaryStatistics): fix in sanity_filter

* fix: adding prune of inf values

* fix(dataset): removal of inf values from beta and stderr

* fix: fix in test and sanity filter

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>
Co-authored-by: Szymon Szyszkowski <[email protected]>

Co-authored-by: Yakov <[email protected]>

Co-authored-by: Szymon Szyszkowski <[email protected]>

…on (opentargets#628)

* feat(clumping): lower p-value significance threshold for clumping step
* feat(config): extract gwas_significance to config
* feat(config): synced p-value in association parsing

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>
Co-authored-by: Yakov <[email protected]>

* feat(airflow): drop finngen PICS results from etl
* feat(l2g): allow to publish feature matrix
* fix(types): typo in types
* chore(data_release): bumped release date

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

…s#632)

Bumps [typing-extensions](https://github.com/python/typing_extensions) from 4.11.0 to 4.12.1.
- [Release notes](https://github.com/python/typing_extensions/releases)
- [Changelog](https://github.com/python/typing_extensions/blob/main/CHANGELOG.md)
- [Commits](python/typing_extensions@4.11.0...4.12.1)

---
updated-dependencies:
- dependency-name: typing-extensions
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Szymon Szyszkowski <[email protected]>

)

Bumps [pep8-naming](https://github.com/PyCQA/pep8-naming) from 0.13.3 to 0.14.1.
- [Release notes](https://github.com/PyCQA/pep8-naming/releases)
- [Changelog](https://github.com/PyCQA/pep8-naming/blob/main/CHANGELOG.rst)
- [Commits](PyCQA/pep8-naming@0.13.3...0.14.1)

---
updated-dependencies:
- dependency-name: pep8-naming
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…rgets#630)

* chore: prototype ingestion of sceqtls

* chore: use credible sets from ftp

* feat(study_index): rename tissuefromsourceid to biosamplefromsourceid to accommodate cell type ids

* chore: update output paths to final destination

)

* feat(spark-helpers): enforce schema of returned objects by other function

* fix: fixing docstring

* test: adding test for array sorter

* chore: pre-commit auto fixes [...]

* fix: addressing a typo

* chore: pre-commit auto fixes [...]

* fix: removing duplicated rows

* fix: addressing review comments

* fix: adding StructType hint

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* feat(schema): adding locus start/end position to the study locus schema

* feat(clumping) adding clumping logic + docs

* test(clumpin): adding tests for locus breaker method

* test(clumping): adding semantic tests for locus breaker clumping

* fix(config): reverting gwas significance threshold

Bumps [wandb](https://github.com/wandb/wandb) from 0.16.2 to 0.17.0.
- [Release notes](https://github.com/wandb/wandb/releases)
- [Changelog](https://github.com/wandb/wandb/blob/main/CHANGELOG.md)
- [Commits](wandb/wandb@v0.16.2...v0.17.0)

---
updated-dependencies:
- dependency-name: wandb
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Bumps [ipython](https://github.com/ipython/ipython) from 8.24.0 to 8.25.0.
- [Release notes](https://github.com/ipython/ipython/releases)
- [Commits](ipython/ipython@8.24.0...8.25.0)

---
updated-dependencies:
- dependency-name: ipython
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [scikit-learn](https://github.com/scikit-learn/scikit-learn) from 1.4.0 to 1.5.0.
- [Release notes](https://github.com/scikit-learn/scikit-learn/releases)
- [Commits](scikit-learn/scikit-learn@1.4.0...1.5.0)

---
updated-dependencies:
- dependency-name: scikit-learn
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat: enable interface with gcp secrets manager

* chore: add google-cloud-secret-manager

* chore: merge

* feat: credible set quality filtering

* fix: purity threshold

Bumps [dbldatagen](https://github.com/databrickslabs/data-generator) from 0.3.5 to 0.4.0.
- [Release notes](https://github.com/databrickslabs/data-generator/releases)
- [Changelog](https://github.com/databrickslabs/dbldatagen/blob/master/CHANGELOG.md)
- [Commits](databrickslabs/dbldatagen@release/v0.3.5...release/v0.4.0)

---
updated-dependencies:
- dependency-name: dbldatagen
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat(config): typos fix
* fix(config): moved ld_index dataset to static assets

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

* feat(dataset): schema mismatch issue
* feat(L2GPrediction): schema unification
* fix: swapped data types

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

…ets#645)

* feat(SusieFineMapperStep): add new fucntion that takes boundaries as input

* fix: typo in function

* refactor: remove ot_pics

* refactor: gwas_catalog_sumstat_preprocess config removed

* refactor: ot_finngen_studies removed

* refactor: ot_finngen_studies removed

* refactor: window_based_clumping cleanup

…ets#650)

* feat(ld_annotator): apply r2 threshold

* feat(ld_annotator): apply r2 threshold

* chore(ldannotator): change threshold to 0.5

…argets#644)

* feat(stydyLocus): adding new locus collection using boundaries

* fix: fix in test

* Update tests/gentropy/dataset/test_study_locus.py

Co-authored-by: Szymon Szyszkowski <[email protected]>

* chore: pre-commit auto fixes [...]

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

…targets#651)

Sometimes it is very useful to be able to spin up a Jupyter notebook on a Dataproc cluster which has access to Gentropy and all its configured environment. Previously, I already added the “enable_component_gateway” option; however, it doesn't really take any effect unless you also specify the list of components to enable, which is what I'm doing in this PR.

…ets#576)

* chore: checkpoint

* chore: checkpoint

* chore: deprecate spark evaluator

* chore: checkpoint

* chore: resolve conflicts with dev

* chore: resolve conflicts with dev

* chore(model): add parameters class property

* feat: add module to export model to hub

* refactor: make model agnostic of features list

* chore: add wandb to gitignore

* feat: download model from hub

* chore(model): adapt predict method

* feat(trainer): add hyperparameter tuning

* chore: deprecate trainer tests

* refactor: modularise step

* feat: download model from hub by default

* fix: convert omegaconfig defaults to python objects

* fix: write serialised model to disk and then upload to gcs

* fix(matrix): drop goldStandardSet when in predict mode

* chore: pass token to access private model

* chore: pass token to access private model

* fix: pass right schema

* chore: pre-commit auto fixes [...]

* chore: fix mypy issues

* build: remove xgboost

* chore: merge

* chore: pre-commit auto fixes [...]

* chore: address comments

)

* feat: implement UKB PPP (EUR) ingestion & harmonisation

* fix: correct module name for docs

* fix: definitely correct module name for docs

* test: update output of neglog_pvalue_to_mantissa_and_exponent

* fix: test syntax with <BLANKLINE>

* Update src/gentropy/datasource/ukb_ppp_eur/summary_stats.py

Co-authored-by: Szymon Szyszkowski <[email protected]>

* fix: code review updates for docs and version

* fix: syntax for concat_ws

* style: list harmonisation steps in the docstring

* style: rename freq to MAF

* style: use concat_ws

* style: use two distinct parameters for study index and summary stats output paths

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

Code inspection shows that it is not used anymore.

…entargets#656)

Bumps [python-semantic-release/python-semantic-release](https://github.com/python-semantic-release/python-semantic-release) from 9.6.0 to 9.8.3.
- [Release notes](https://github.com/python-semantic-release/python-semantic-release/releases)
- [Changelog](https://github.com/python-semantic-release/python-semantic-release/blob/master/CHANGELOG.md)
- [Commits](python-semantic-release/python-semantic-release@v9.6.0...v9.8.3)

---
updated-dependencies:
- dependency-name: python-semantic-release/python-semantic-release
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Kirill Tsukanov <[email protected]>

Bumps [pydoclint](https://github.com/jsh9/pydoclint) from 0.4.1 to 0.5.1.
- [Release notes](https://github.com/jsh9/pydoclint/releases)
- [Changelog](https://github.com/jsh9/pydoclint/blob/main/CHANGELOG.md)
- [Commits](jsh9/pydoclint@0.4.1...0.5.1)

---
updated-dependencies:
- dependency-name: pydoclint
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Ochoa <[email protected]>

* fix: remove check merge conflict

* fix: remove line for now

…#654)

* feat(variant annotation): new variant annotation schema + logic to extract from VEP

* fix: typehints in function

* refactor(variant annotation): migrating methods to the new schema

* chore: pre-commit auto fixes [...]

* refactor(variant index): sorting out new variant index dataset

* chore: pre-commit auto fixes [...]

* feature(vep): adding predictors to vep transcript object

* fix(schema): fixing schema missing fields

* fix(schema): fixing schema missing fields

* fix(schema): fixing schema missing fields

* fix(schema): fixing schema missing fields

* chore: pre-commit auto fixes [...]

* fix(annotation): array union under condition

* fix: merging dbxref objects

* feat(variants): updating variants to make more robust

* feat: migrating methods to new variant index

* adjusting variant index methods

* some updates

* rename v2g to variant to gene

* chore: pre-commit auto fixes [...]

* adding test

* chore: pre-commit auto fixes [...]

* fix(precommit): json file needed to rename to jsonl

* fix(precommit): removing steps depending on old data model

* fix(coftest): fixing variant index mock generation

* fix: typo in package import

* fix: sorting out conftest

* refactor(gwas ingest): Updating GnomAD handling

* refactor(gnomad): variant annotation removed, changed to variant index, steps updated

* refactor: shuffling around gnomad logic

* fix: references in tests

* refactor: sorting out gnomad variant dag

* refactor: cleaning configs and tests

* docs(vep): adding datasource description

* test(vep): adding more test to the vep parser

* test(vep): tests are now running

* fix: removing version suffix from pyproject and airflow config

* fix: reverting DAGs - removing temporary modifications I added for testing

* fix: addressing reviewer comments

* refactor: fiddling with variant index annotation logic

* chore: addressing comments

* fix: variant cross-ref snake case

* fix: correcting join strategy

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fix: typo in cs_lbf_thr parameter name

* fix: removing two parameters

---------

Co-authored-by: Yakov Tsepilov <[email protected]>

…ets#668)

As it is, when the susie_finemapper step is triggered, nothing happens because the configuration class is not linked via _target_ to the step class. This commit addresses the problem

…ts#608)

* feat: custom dockerfile to run ensembl vep

* ci: automate vep image build and artifact registry

* chore: update airflow google operators (not required)

* feat: working version of google batch airflow vep job

* feat: working version of google batch airflow vep job

* feat(VEP): adding CADD plugin

* feat: local loftee file

* feat: working with input bucket full of input files

* feat: prevent writing html

* fix: minor adjustments to retry strategy

* feat(airflow): separating mounting points for input/output and cache

* fix: typo in airflow dag

* fix: pre-commit pain

* chore: rename airflow dag file

---------

Co-authored-by: DSuveges <[email protected]>
Co-authored-by: Szymon Szyszkowski <[email protected]>

* feat: locus_breaker_clumping

* fix: dosctring

* feat: _process_locus_breaker function

* feat: locus breaker clumping step

* fix: tidying parameters

* feat: option to remove MHC region

* fix: description for LocusBreakerClumpingStep

* fix: removing division of distance

* fix: adding new parameters for wbc distance separate from large_loci_size

* fix: resolving comments

* refactor: refactored code in process_locus_breaker_output

* fix: removing superfluous variable

* fix: persisting sumstats parquet to improve analysis plan

---------

Co-authored-by: Yakov <[email protected]>

* feat: add qc step

* fix: remove .df

* fix: fix in name

* fix: fix v3

* Update src/gentropy/sumstat_qc_step.py

Co-authored-by: Daniel Suveges <[email protected]>

* Update src/gentropy/sumstat_qc_step.py

Co-authored-by: Daniel Suveges <[email protected]>

* fix: optimisation of code

---------

Co-authored-by: Daniel Suveges <[email protected]>

…targets#677)

* feat: adding sanity filter to GWASCatalogSumstatsPreprocessStep

* fix: adding description

* fix: improving locus_breaker_step logic

* fix: updating susie_finemapper.py to deal with new I/O logic

* chore: removing unused log output path

…ts#682)

Two values were recently removed from the step. This brings config in alignment with these changes.

* feat: credible set quality control step

* feat: credible_set_qc.py step

* chore: updating genotropy/config.py to include CredibleSetQCConfig()

* chore: updated LocusBreakerClumpingConfig(StepConfig)

* fix: changing default configs in code

* feat: added jq to the docker image dependencies
* feat: added gsutil to gentropy image
* fix: added missing env vars

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

…ntargets#672)

* chore: snapshot

* feat(StudyIndex): adding valiation methods

* feat(studyIdex): adding disease validation

* fix: typo in test

* fix: moving import under the type checking condition

* fix: some columns might need to be dropped

* fix(study index): preventing [null] arrays in the cohorts object

* fix(study index): more context is provided for disease normalisation

Bumps [ruff](https://github.com/astral-sh/ruff) from 0.4.10 to 0.5.1.
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](astral-sh/ruff@v0.4.10...0.5.1)

---
updated-dependencies:
- dependency-name: ruff
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Ochoa <[email protected]>

* chore: pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.4.3 → v0.5.1](astral-sh/ruff-pre-commit@v0.4.3...v0.5.1)
- [github.com/pre-commit/mirrors-mypy: v1.10.0 → v1.10.1](pre-commit/mirrors-mypy@v1.10.0...v1.10.1)
- [github.com/jsh9/pydoclint: 0.4.1 → 0.5.3](jsh9/pydoclint@0.4.1...0.5.3)

* chore: update lock

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: David Ochoa <[email protected]>
Co-authored-by: David Ochoa <[email protected]>

* build(deps): bump skops from 0.9.0 to 0.10.0

Bumps [skops](https://github.com/skops-dev/skops) from 0.9.0 to 0.10.0.
- [Release notes](https://github.com/skops-dev/skops/releases)
- [Changelog](https://github.com/skops-dev/skops/blob/main/docs/changes.rst)
- [Commits](skops-dev/skops@v0.9.0...v0.10)

---
updated-dependencies:
- dependency-name: skops
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* fix: update `trusted` in load to avoid typeerror

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Irene López <[email protected]>

…ntargets#684)

Bumps [google-cloud-dataproc](https://github.com/googleapis/google-cloud-python) from 5.9.3 to 5.10.1.
- [Release notes](https://github.com/googleapis/google-cloud-python/releases)
- [Changelog](https://github.com/googleapis/google-cloud-python/blob/main/packages/google-cloud-documentai/CHANGELOG.md)
- [Commits](googleapis/google-cloud-python@google-cloud-dataproc-v5.9.3...google-cloud-dataproc-v5.10.1)

---
updated-dependencies:
- dependency-name: google-cloud-dataproc
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

…pentargets#665)

* feat(variant index): calculate hash as variant id for variants with long alleles

* fix: adjusting hashing method

* fix: painful iteration to get gnomad running with the new schema + fine tune with variant index

* chore: fixing variant index step

* fix: reverting changes

* fix: minor updates

* feat(variant index): null values of array columns are enforced to be replaced with empty arrays

* fix: array union vs coalese fix

* fix(variant index): annotation logic made more robust

* fix: addressing review comments

* fix: renaming helper function for better readability

* fix(liftover): fixing docstring

Bumps [ipython](https://github.com/ipython/ipython) from 8.25.0 to 8.26.0.
- [Release notes](https://github.com/ipython/ipython/releases)
- [Commits](ipython/ipython@8.25.0...8.26.0)

---
updated-dependencies:
- dependency-name: ipython
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Ochoa <[email protected]>
Co-authored-by: David Ochoa <[email protected]>

Co-authored-by: Szymon Szyszkowski <[email protected]>

added tags to the artifact github actions triggers, so the
images produced by gentropy will be versionned by tag name

Co-authored-by: Szymon Szyszkowski <[email protected]>

* fix: moving hl.init

* fix: missed a hl.init

* fix: removing the hl.init from notebooks

…nd Finngen (opentargets#693)

* chore: remove finngen prefix from studyIds

* feat(eqtl_catalogue): changes from #3357

* fix: update study table

* fix: update _identify_study_type example

…taset (opentargets#686)

* feat(studyLocus validation): adding validation logic to studyLocus dataset

* feat: making QC more granular

* Update src/gentropy/dataset/study_locus.py

Co-authored-by: David Ochoa <[email protected]>

---------

Co-authored-by: David Ochoa <[email protected]>

Co-authored-by: Szymon Szyszkowski <[email protected]>

* feat(session): replace `read_parquet` with more general `load_data`

* chore: checkpoint

* feat(variant_index): overwrite variant id for longer variants

* feat(variant_index): copy vcf from local filesystem to gcs

* build: remove request dependency for urllib

* feat(variant idx): integrate all components into dag

* test: fix `fetch_coordinates` typo

* Update src/airflow/dags/common_airflow.py

Co-authored-by: Daniel Suveges <[email protected]>

* refactor: move fetch_coordinates_from_rsids to independent module

* fix: update fetch_coordinates_from_rsids example

---------

Co-authored-by: Daniel Suveges <[email protected]>

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.5.1 → v0.5.2](astral-sh/ruff-pre-commit@v0.5.1...v0.5.2)
- [github.com/jsh9/pydoclint: 0.5.3 → 0.5.4](jsh9/pydoclint@0.5.3...0.5.4)

fix(variants): explode loci to generate list of variants + tests

…ate-config

chore: pre-commit autoupdate

Co-authored-by: Irene López Santiago <[email protected]>

…s-for-gentropy

…of-the-docker-release-process-for-gentropy

ci: Refactoring of the docker release process for gentropy

…ping-typo

fix(finemapping): typo in "elapsed_time"

…push

feat: allow image push

…ets#706)

Bumps [pytest](https://github.com/pytest-dev/pytest) from 8.2.2 to 8.3.1.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](pytest-dev/pytest@8.2.2...8.3.1)

---
updated-dependencies:
- dependency-name: pytest
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [deptry](https://github.com/fpgmaas/deptry) from 0.16.1 to 0.17.0.
- [Release notes](https://github.com/fpgmaas/deptry/releases)
- [Changelog](https://github.com/fpgmaas/deptry/blob/main/CHANGELOG.md)
- [Commits](fpgmaas/deptry@0.16.1...0.17.0)

---
updated-dependencies:
- dependency-name: deptry
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>