…li (#716)

* feat: expose summary statistics qc to hydra cli

* feat: expose locus breaker clumping step

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

Co-authored-by: Szymon Szyszkowski <[email protected]>

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.5.2 → v0.5.5](astral-sh/ruff-pre-commit@v0.5.2...v0.5.5)
- [github.com/pre-commit/mirrors-mypy: v1.10.1 → v1.11.0](pre-commit/mirrors-mypy@v1.10.1...v1.11.0)
- [github.com/jsh9/pydoclint: 0.5.4 → 0.5.6](jsh9/pydoclint@0.5.4...0.5.6)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Bumps [pymdown-extensions](https://github.com/facelessuser/pymdown-extensions) from 10.8.1 to 10.9.
- [Release notes](https://github.com/facelessuser/pymdown-extensions/releases)
- [Commits](facelessuser/pymdown-extensions@10.8.1...10.9)

---
updated-dependencies:
- dependency-name: pymdown-extensions
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Szymon Szyszkowski <[email protected]>

Bumps [deptry](https://github.com/fpgmaas/deptry) from 0.17.0 to 0.18.0.
- [Release notes](https://github.com/fpgmaas/deptry/releases)
- [Changelog](https://github.com/fpgmaas/deptry/blob/main/CHANGELOG.md)
- [Commits](fpgmaas/deptry@0.17.0...0.18.0)

---
updated-dependencies:
- dependency-name: deptry
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat: notebook for locus breaker and susie finemapping benchmark

* fix: won't pass tests with matplotlib imported

* feat: ukb_ppp_fm benchmark notebook

* fix: check for nulls as well as nans

* fix: removing matplotlib

* fix: update notebooks

* chore: re-run ukb ppp notebook

* chore: rerun notebook

* fix: updating config paths and fine-mapping methods

* Update ot_locus_to_gene_train.yaml

* docs: macos fix for some functions

* docs: formatting

* build(deps-dev): bump ruff from 0.5.1 to 0.6.1

Bumps [ruff](https://github.com/astral-sh/ruff) from 0.5.1 to 0.6.1.
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](astral-sh/ruff@0.5.1...0.6.1)

---
updated-dependencies:
- dependency-name: ruff
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* fix: linting issues associated with ruff 0.6

* chore: fixing imports for notebooks

* chore: removing old notebook

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Ochoa <[email protected]>
Co-authored-by: Daniel Considine <[email protected]>

* chore: pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.5.5 → v0.6.1](astral-sh/ruff-pre-commit@v0.5.5...v0.6.1)
- [github.com/pre-commit/mirrors-mypy: v1.11.0 → v1.11.1](pre-commit/mirrors-mypy@v1.11.0...v1.11.1)

* chore: pre-commit auto fixes [...]

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: David Ochoa <[email protected]>

Bumps [deptry](https://github.com/fpgmaas/deptry) from 0.18.0 to 0.19.1.
- [Release notes](https://github.com/fpgmaas/deptry/releases)
- [Changelog](https://github.com/fpgmaas/deptry/blob/main/CHANGELOG.md)
- [Commits](fpgmaas/deptry@0.18.0...0.19.1)

---
updated-dependencies:
- dependency-name: deptry
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Ochoa <[email protected]>

Bumps [lxml](https://github.com/lxml/lxml) from 5.2.2 to 5.3.0.
- [Release notes](https://github.com/lxml/lxml/releases)
- [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt)
- [Commits](lxml/lxml@lxml-5.2.2...lxml-5.3.0)

---
updated-dependencies:
- dependency-name: lxml
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps-dev): bump pre-commit from 3.7.1 to 3.8.0

Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 3.7.1 to 3.8.0.
- [Release notes](https://github.com/pre-commit/pre-commit/releases)
- [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md)
- [Commits](pre-commit/pre-commit@v3.7.1...v3.8.0)

---
updated-dependencies:
- dependency-name: pre-commit
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* fix: update lock

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Szymon Szyszkowski <[email protected]>
Co-authored-by: David Ochoa <[email protected]>
Co-authored-by: David Ochoa <[email protected]>

* feat: update variant sources for variant idx and drop rsid mapping

* chore: remove variant_index step from etl dag

* chore: move gnomad annotation from dev to static assets

* chore: change gentropy docker image

* feat(convert_to_vcf): write data partitioned

* feat: add vcf merging step

* chore: assert merge is succesful

* fix: remove non canonical variants from vcfs

* chore: pass vcf types to merging rule

* fix: match vep parser order of columns to variant index schema

* fix: ditch `assign_variant_id` in favour of `hash_long_variant_ids`

* chore: update `test_as_vcf_df_without_variant_id`

* fix: correct test mock data

* chore: pre-commit auto fixes [...]

* chore: merge from dev

* chore: pre-commit auto fixes [...]

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Szymon Szyszkowski <[email protected]>

Co-authored-by: Szymon Szyszkowski <[email protected]>

Co-authored-by: Szymon Szyszkowski <[email protected]>

* feat: finngen r11 study index paths

* feat: simplification of finngen susie finemapping ingestion

* feat: efos in finngne study index

* docs: updated docstring

* fix: pass spark_session not gentropy session to step

* fix: read uploaded file  requires file:///

* fix: parallelize csv from url

* fix: typo in docstring

* chore: pre-commit auto fixes [...]

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Suveges <[email protected]>

* fix: adding carma_tau parameter to susie_finemapper

* fix: changing default

* fix: defaults

* refactor: generalised validation logic

* refactor: generalised validation logic

* fix(airflow): fine-tuning DAG for data validation

* fix(validation): study locus uniqueness fixed

* feat: add invalid/valid dataset generation in validation steps/dag (#734)

* fix: does not belong in this PR

* refactor: to be moved to orchestration repo

* docs: respective docs pages for the steps

* docs: ammend docstrings

* revert: maintain dag for now but it should be removed eventually

* feat: validate study_locus dataset to produce valid or invalid df

* feat: adjust DAG to parametrise dataset validation

* fix: duplicated row

* feat: increase abstraction of Dataset validation of rows

* docs: increase clarity of what the function does

* fix: error message

* revert: unintended change

* test: testing dataset filtering by quality flag

---------

Co-authored-by: DSuveges <[email protected]>

* chore: pre-commit auto fixes [...]

---------

Co-authored-by: David Ochoa <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Szymon Szyszkowski <[email protected]>

* feat(variant index): upating schema and logic

* feat(variant index): vep transcript annotation improved with consequence score + transcript index

* refactor: removed hardcoded schema definition from parser

* fix(test): fixing mock variant index

* fix: schema extraction fixed

* fix: doctest issue resolved

* fix: updating vep command in the DAG

* fix: removing un-used sequence ontology terms

* fix: adding example to sorter method + changing variable names

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

* fix(finngen_r11): preserve all studyIds

Preserve all studyIds, even if EFO mapping is missing, so mapping
between studyIndex and StudyLocus is by studyId column is accurate.

* fix: typo in docstring

Co-authored-by: Irene López Santiago <[email protected]>

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>
Co-authored-by: Irene López Santiago <[email protected]>

* feat: adding finemapping method to studylocusid hash

* chore: updating function in needed files

* test: update doc test

* fix: mistakes

* fix: tidying logic

* chore: adding studyLocusId recalculation to pics finemapping method

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

Bumps [ipython](https://github.com/ipython/ipython) from 8.26.0 to 8.27.0.
- [Release notes](https://github.com/ipython/ipython/releases)
- [Commits](ipython/ipython@8.26.0...8.27.0)

---
updated-dependencies:
- dependency-name: ipython
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Szymon Szyszkowski <[email protected]>

Bumps [deptry](https://github.com/fpgmaas/deptry) from 0.19.1 to 0.20.0.
- [Release notes](https://github.com/fpgmaas/deptry/releases)
- [Changelog](https://github.com/fpgmaas/deptry/blob/main/CHANGELOG.md)
- [Commits](fpgmaas/deptry@0.19.1...0.20.0)

---
updated-dependencies:
- dependency-name: deptry
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Szymon Szyszkowski <[email protected]>

* fix: use mapping instead of enum values in valid_rows

* fix: typos

* fix: swap valid and invalid paths

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

Co-authored-by: Szymon Szyszkowski <[email protected]>

* refactor: move VA preparation logic into a separate module

* refactor: generalise per-chromosome summary stats ingestion

* Update src/gentropy/common/per_chromosome.py

Co-authored-by: Szymon Szyszkowski <[email protected]>

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

* refactor: move harmonisation into a separate module

* refactor: make column names configurable

* feat: make INFO and A1FREQ columns optional

* docs: expand comments on tmp_variant_annotation_path and variant types

* feat: implement FinnGen UKB meta-analysis ingestion and harmonisation

* chore: remove ot_finngen_ukb_meta.yaml

* chore: remove raw_study_index_path to raw_study_index_path_from_tsv

* fix: use session.write_mode

* style: rename class to FinngenUkbMetaIngestionStep

* feat(validation): adding logic to validate credible sets against variant index

* fix: tidying docstrings

* refactor: drop gentropy config

* feat(airflow): removal of airflow orchestration

BREAKING CHANGE: see https://github.com/opentargets/orchestration

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

Bumps [wandb](https://github.com/wandb/wandb) from 0.17.2 to 0.18.0.
- [Release notes](https://github.com/wandb/wandb/releases)
- [Changelog](https://github.com/wandb/wandb/blob/main/CHANGELOG.md)
- [Commits](wandb/wandb@v0.17.2...v0.18.0)

---
updated-dependencies:
- dependency-name: wandb
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: prevent multiple filters to override spark plan

* feat: mhc quality control flag

* fix: prevent multiple filters to override spark plan

* Revert "fix: prevent multiple filters to override spark plan"

This reverts commit a358781.

* revert: wrong commit

* fix: missing changes due to git chaos

* chore: merge dev toml, update lock

* chore: update lock after upgrading poetry to 1.8.3 v 2

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

* fix: prevent multiple filters to override spark plan

* feat: mhc quality control flag

* fix: typo

* docs: genomic region added to docs

* docs: info added to common index (and title)

* docs: fix paths

* Update src/gentropy/study_locus_validation.py

* fix: remove unnecessary comment

---------
Closes opentargets/issues#3469
Co-authored-by: Szymon Szyszkowski <[email protected]>

* fix: turning OR to beta harmonisation on for GWAS Catalog

* fix(ingestion): addressing beta harmonisation bug

* fix(ingestion): addressing beta harmonisation bug

* test: adding test for effect harmonisation

* test: adding doctests for beta harmonisation

* fix: fixing value checking bug: effect value needs to be cast to double type

* fix: typos in gwas ingestion script

* perf: cache Docker image layers

* chore: temporary add branch to test

* perf: separate AMD and ARM builds for gentropy

* perf: temporarily test only VEP image separately

* perf: test ARM/AMD gentropy/VEP separately

* perf: introduce quick build for all commits

* perf: quick build for all tags

* fix: use correct syntax for tags

* fix: revert accidental changes for the VEP step

Co-authored-by: Szymon Szyszkowski <[email protected]>

Co-authored-by: Daniel Suveges <[email protected]>

* fix(ld clumping): a revised logic allows a more accurate identification of linked loci

* test: adding some test

* chore: pre-commit auto fixes [...]

* test: adding more test cases

* fix: fixing test data

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* refactor(L2GFeatureMatrix): remove schema validation

* refactor(FeatureFactory): reshape feature generation WIP

* chore: pre-commit auto fixes [...]

* chore: set l2gfeature properties with decorator

* chore(l2gfeature): make credible_set and input_dependency instance attributes

* chore(l2gfeature): make credible_set and input_dependency instance attributes

* chore(featurefactory): distanceTssMeanFeature working

* refactor(l2g): improve step dependency management

* feat: implement

* chore: fix mypy issues

* feat: l2gfeaturematrix.from_features_list working

* chore: comment out obsolete refs

* chore(L2GFeatureMatrix): change `mode` attribute to `with_gold_standard`

* refactor(l2g): move feature matrix writing to training module

* feat(L2GFeatureMatrix): accept L2GGoldStandard or StudyLocus as inputs

* feat: implement methods to build a feature matrix based on a studylocus/L2GGoldStandard instance

* feat: coloc logic prototype

* feat(l2g): filter non gwas credible sets at the start of the step

* feat: rewrite colocalisation feature factory

* test: add `test_colocalisation_feature_type`

* test(colocalisation): add test_extract_maximum_coloc_probability_per_region_and_gene

* feat(L2GFeatureInputLoader): support multiple deps by passing loader as kwarg

* test: add integration tests `test_build_feature_matrix`

* chore: drop config yamls

* refactor: move feature classes to datasets module

* docs: update feature docs

* refactor(colocalisation): cleaner joins in `append_right_study_metadata`

* chore: better logging abstract methods

* test: add `L2GFeatureMatrix.test_from_features_list` unit tests

* fix: add goldStandardSet when a gs instance is passed to `from_features_list`

* fix: lowercase colocalisation type and add semantic test

* test: add semantic test for `append_right_study_metadata`

* feat(colocalisation): make `append_right_study_metadata` extensible to left metadata

* fix(colocalisation): append_study_metadata cant take a gold standard

* fix(colocalisation): extract_maximum_coloc_probability_per_region_and_gene cant take a gold standard

* feat: add `StudyLocus` as a dependency of colocalisation features

* fix: add studylocus to input loader in test

* fix: add studylocus to input loader in test

* fix: add studylocus to input loader in test

…yLocusOverlap`) (#782)

* feat: add studyType to StudyLocus schema

* feat: add annotate_study_type function to add studyType to StudyLocus

* fix: remove lines for retrieving studyType as StudyLocus now contains studyType

* fix: add studyType to test input data as StudyLocus now contains studyType

* feat: add leftStudyType and rightStudyType to Colocalisation and StudyLocusOverlap schemas

* feat: update _convert_to_square_matrix and its test with leftStudyType and rightStudyType

* feat: update test_find_overlaps_semantic inputs with leftStudyType and rightStudyType

* feat: update tests in test_colocalisation_method.py with leftStudyType and rightStudyType

* feat: add leftStudyType and rightStudyType when creating StudyLocusOverlap

* feat: add leftStudyType and rightStudyType to Colocalisation results

* fix: remove redundant study_index parameter from filter_by_study_type function def and calls

* fix: remove redundant study_index parameter from find_overlaps function def and calls

* fix: remove leftStudyType from Colocalisation (not needed as always gwas)

* fix: remove leftStudyType from StudyLocusOverlap (not needed as always gwas)

* fix: missing comma

* feat: update tests in test_locus_to_gene.py with studyType and rightStudyType

* feat: update tests (colocalisation, l2g, l2g feature matrix) with rightStudyType

* fix: remove studyType from metadata_cols in append_study_metadata function call

* feat: adding window based clumping to locus

* fix: reverting some changes

* chore: pre-commit auto fixes [...]

* fix: fixing probem introduced by merge conflict

* fix: addressing review comment

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Initial commit of biosample index

* Make minimal class

* Tidy up first draft of adding biosample index

* Add beginning of logic for checking if biosample from a studyindex is in biosample index

* Make early file for merging multiple biosample indices into one

* Finish adding basic iteration of biosample index, needs debugging

* Tweak slightly

* Modified the parser to accept JSON files

* Update biosample index

* Tests and docs

* Updating tests

* Revert GWAS catalog file

* fix(biosample index): update to match pre-commit standards

* fix(biosample index): merging indices fix

* fix(biosample index): update study index qc logic

* fix(biosample index): fix missing mock_biosample_index

* chore(biosample index): change datasource name from ontologies

* fix(biosample index): add dataset doc

* fix(biosample index): change dbXrefs to xrefs

* chore (biosample index): better commenting

Co-authored-by: Daniel Suveges <[email protected]>

* fix(biosample index): various minor tweaks to biosample index

* fix(biosample index): minor bug

* fix(biosample index): fix merge shift to method

* feat(biosample index): make biosampleName not nullable

---------

Co-authored-by: Daniel Suveges <[email protected]>

* feat: study locus validation filters for 95% credible sets

* revert: no longer needed to filter for credible set interval

* feat: annotate credible sets before filter them

* docs: adding more context here

* feat: flag PICS top hits in studies with credset sumstats

* feat: flag credible sets explained by SuSiE region

* feat: consider unresolved LD cases

* refactor: improve readability of code

* refactor: improve code readability

---------

Co-authored-by: Daniel Suveges <[email protected]>

Bumps [pymdown-extensions](https://github.com/facelessuser/pymdown-extensions) from 10.9 to 10.10.1.
- [Release notes](https://github.com/facelessuser/pymdown-extensions/releases)
- [Commits](facelessuser/pymdown-extensions@10.9...10.10.1)

---
updated-dependencies:
- dependency-name: pymdown-extensions
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Ochoa <[email protected]>

* fix(finngen_study_index): improved tests for finngen study index

* chore(tests): added pytest mark to be able to isolate step tests

* chore: pr comments

* feat: revert mock.patch

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

* fix: remove study_index_path from coloc step

* fix(safe_array_union): sort struct fields in array

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

#790)

* fix(schema): recursive validation of arbitrariliy deep nested structure

* chore: pre-commit auto fixes [...]

* fix: docstring issues

* fix: failing tests are fixed

* test: skipping tests that are failing because bug in vep parser

* test(schemas): adding tests for schema comparison functions

* test: removing skip as the vep parser logic is fixed

* fix: addressing review comments

* fix(schemas): removing schema flattening function

* chore: pre-commit auto fixes [...]

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Szymon Szyszkowski <[email protected]>

* fix: adding data specific fillters

* fix: removing variables

* fix: adding options to init

---------

Co-authored-by: Daniel Suveges <[email protected]>

…ocusId` to string type) (#783)

* feat: change studyLocusId to string in schema

* feat: change studyLocusId of example data to string in tests

* feat: change hashing method to md5

* test: remove test_assign_study_locus_id__null_variant_id as validation will have removed null ids

* fix: change studyLocusId to string in remaining files

* fix: ensure inputs to assign_study_locus_id are columns and not strings

* fix: change studyLocusId to string in remaining files

* chore: update assign_study_locus_id docstring with updated output

* chore: update assign_study_locus_id docstring with updated output (again)

* fix: change studyLocusId to string in recently merged files

* feat: move hashing logic to generate_identifier function in Dataset class

---------

Co-authored-by: Daniel Suveges <[email protected]>

* refactor(L2GFeatureMatrix): remove schema validation

* refactor(FeatureFactory): reshape feature generation WIP

* chore: pre-commit auto fixes [...]

* chore: set l2gfeature properties with decorator

* chore(l2gfeature): make credible_set and input_dependency instance attributes

* chore(l2gfeature): make credible_set and input_dependency instance attributes

* chore(featurefactory): distanceTssMeanFeature working

* refactor(l2g): improve step dependency management

* feat: implement

* chore: fix mypy issues

* feat: l2gfeaturematrix.from_features_list working

* chore: comment out obsolete refs

* chore(L2GFeatureMatrix): change `mode` attribute to `with_gold_standard`

* refactor(l2g): move feature matrix writing to training module

* feat(L2GFeatureMatrix): accept L2GGoldStandard or StudyLocus as inputs

* feat: implement methods to build a feature matrix based on a studylocus/L2GGoldStandard instance

* feat: coloc logic prototype

* feat(l2g): filter non gwas credible sets at the start of the step

* feat: rewrite colocalisation feature factory

* test: add `test_colocalisation_feature_type`

* test(colocalisation): add test_extract_maximum_coloc_probability_per_region_and_gene

* feat(L2GFeatureInputLoader): support multiple deps by passing loader as kwarg

* test: add integration tests `test_build_feature_matrix`

* feat(variant_index): add `get_distance_to_gene` and deprecate `get_distance_from_tss`

* feat(variant_index): deprecate `get_most_severe_transcript_consequence`

* feat(variant_index): add `get_loftee` and deprecate `get_plof_v2g`

* chore: reduce v2g assesments to intervals

* feat(feature_factory): add distance to footprint features

* test: refactor `test_feature_factory_return_type`

* feat(feature_factory): add all distance neighbourhood features

* chore: delete v2g

* feat(feature_factory): add all colocalisation neighbourhood features

* chore: final v2g deletion

* chore: drop config yamls

* refactor: move feature classes to datasets module

* docs: update feature docs

* fix: import

* test: add semantic `TestCommonColocalisationFeatureLogic`

* test: add semantic `TestCommonDistanceFeatureLogic`

* refactor: separate features into diff modules

* fix: documentation references

* feat: implement distance to sentinel and adapt definitions

* docs: update distance class names

* fix: add all variant index mandatory fields in tests

* fix(schema_validator): remove extra `[]` from parent prefix

* fix: convert studylocusid to string in tests

* revert: revert 72ea515 (it was ok)

* fix: adapt test

Co-authored-by: Daniel Suveges <[email protected]>

Co-authored-by: Szymon Szyszkowski <[email protected]>

* fix(ukb_ppp_study_index): update column name to match schema

* chore: add note that notebooks are not supported

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

Co-authored-by: Szymon Szyszkowski <[email protected]>

…tion time (#801)

* feat: adding credible set confidence

* feat: adding confidence assignment to the credible set validation step

* fix: adding confidence field to schema

* fix: schema

* fix: confidence criteria

* fix: finalising labels

* fix: docstring typo

* test: for credible set confidence assignment

Co-authored-by: Szymon Szyszkowski <[email protected]>

* feat: adding filltering to susie fine_mapper

* fix: correct options

* fix: fix simulations

* fix: correct clumping

* fix: fix options

* fix: fix for catching None

* fix: adding LD and gwas_df as output

* chore: changing defaults

* chore: v1

* fix: spelling in susie_finemapper.py

---------

Co-authored-by: Daniel-Considine <[email protected]>

* feat: add sumstat QC fields to schema

* fix: fix of type

* fix: fix for struct

* fix(biosample index): update biosample index validation step to only apply to non-gwas studies

* fix(biosample index): revert to dropping

* feat(biosample index): update test study index to test biosample index too

* feat(biosample index): add biosampleId to study index schema

…config (#811)

* refactor: set variant functional consequence to pathogenicity score as class attribute

* chore: drop `VariantIndex.get_most_severe_gene_consequence`

* chore(VariantIndex): make `CONSEQUENCE_TO_PATHOGENICITY_SCORE` a class attribute

* fix(vep): convert `id_to_score_map` to `label_to_score_map`

* chore: remove comment

* refactor: move `consequence_to_pathogenicity_score` to `VariantIndexConfig`

* feat(variant_index): set variant functional consequence to pathogenicity score as class attribute

* feat: add and test vep features

* feat: add and test vep neighbourhood features

* feat(vep_features): use only protein coding genes to look in the vicinity

* chore: update feature list and other bits

* fix: docs ref

* refactor: set variant functional consequence to pathogenicity score as class attribute

* chore: drop `VariantIndex.get_most_severe_gene_consequence`

* chore(VariantIndex): make `CONSEQUENCE_TO_PATHOGENICITY_SCORE` a class attribute

* fix(vep): convert `id_to_score_map` to `label_to_score_map`

* chore: remove comment

* chore: access max consequence score from variant index

* feat: optimisation of qc step

* fix: adding Z2 filter

* fix: v1

…s dataset when perfroming validation (#814)

* fix: coalesce qualityControl column in credible set

* fix: resolve missing qualityControls column when validating studyLocus

* chore: whitespace

* chore: use getter to infer qc column name

* chore: drop show

* chore: drop warnings for step_test(s)

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

* fix: fix of type error in schma checking

* fix: fix fix fix

* fix(biosample_index): remove biosampleId column if present in validation and tests

* fix(biosample index): update tests for biosample index

#821)

* feat: add functions to get the major population and calculate r2

* refactor: remove functions not needed anymore

* refactor: change r2Overall to r2Major

* test: add tests for both of the added functions

* test: remove tests no longer needed

* revert: change r2Major back to r2Overall

Bumps [ipython](https://github.com/ipython/ipython) from 8.27.0 to 8.28.0.
- [Release notes](https://github.com/ipython/ipython/releases)
- [Commits](ipython/ipython@8.27.0...8.28.0)

---
updated-dependencies:
- dependency-name: ipython
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Szymon Szyszkowski <[email protected]>

Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 3.8.0 to 4.0.0.
- [Release notes](https://github.com/pre-commit/pre-commit/releases)
- [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md)
- [Commits](pre-commit/pre-commit@v3.8.0...v4.0.0)

---
updated-dependencies:
- dependency-name: pre-commit
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: join between gold standard and credible set based on studyId and variantId

* fix: minor bugs to generate the feature matrix

* fix(colocalisation): safeguard existing rightStudyType when applying `append_study_metadata`

* feat: add feature generation step (working interactively)

* feat(l2g): remove feature generation from `LocusToGeneStep`

* fix: correct feature names

* feat: filter gwas credible sets in `L2GPrediction.from_credible_set`

* chore: update docs

* chore: pass credible set to  `L2GGoldStandard.build_feature_matrix` in test

* chore: uncomment code