Release schema + validator (+doc gen) on PyPi

[dosumis]

structure

src/schema/dosdp_schema.yaml  #  = spec/DOSDP_schema_full.yaml
src/schema/test/
src/schema/test/generic_test # Is it valid JSON schema.
src/schema/test/positive_test_set/ # Some list of passing files to test
src/schema/test/negative test_set/# Some list of failing files to test
src/dosdp/dosdp.py # wrapper script with argparse (or click)
src/dosdp/validator.py # sub module based on existing code Allow setting threshold for hard vs soft fail # based on src/dosdp/simple_pattern_tester.py
src/dosdp/document.py # sub module -  patterns as input and generates markdown doc (@matentzn has code)

# dosdp = package name

[matentzn]

Lets try to adhere to good CLI practices: https://clig.dev/

Mainly:

• use flags over args
• use verbs for commands

[matentzn]

In case test config is needed: https://github.com/mapping-commons/sssom-py/blob/master/tests/test_config.yaml

[matentzn]

Also, can we make sure we can use these as a library in python directly:

from dosdp import validate
validate("test.yaml")

[hkir-dev]

pypi package released: https://pypi.org/project/dosdp/0.1.0.dev1/

If ok, we can release a stable version and update usage scenarios at README.md

Usage

From the CLI

$ pip3 install dosdp==0.1.0.dev1
$ dosdp -v <test.yaml>

From Python

from dosdp import validator

validator.validate("test.yaml")

[matentzn]

Hey @hkir-dev! Great push forward; I would like to request one change. I am trying to coordinate a standardisation of parameter across all the tools we are involved in. I will share the full list with you when it is finalised. But generally I would like to always use the same grammar:

toolname function_verb {-x/--xxx}*

So in this case:

dosdp validate -i pattern

or in the case of documentation

dosdp document -i pattern

What do you think? Sorry to spring this on you after the pypi release, I was not paying enough attention :)

[dosumis]

It's clearly flagged as a dev & pre-release on PypI, so I think no expectation of stability yet

Note - Setup.py should be commited at root and should include link to doc

[dosumis]

Potentially relevant: Can use GitHub actions to standardise build and link to release. e.g. see
https://github.com/schlegelp/navis/blob/master/.github/workflows/publish-to-pypi.yml
or
https://github.com/pypa/gh-action-pypi-publish

[hkir-dev]

Cli interface updated. All code is in branch pypi_release

Usage

From the CLI

$ pip3 install dosdp==0.1.0.dev1

$ dosdp validate -i <test.yaml>

$ dosdp document -i <schema.yaml> -o <schema.md>

From Python

from dosdp import validator

validator.validate("test.yaml")

[matentzn]

Awesome! Thanks!

The document command will have to do something more than just documenting the schema, but we can discuss this next week. Thanks!

[hkir-dev]

Package release github action added.
Now releasing to test pypi repository. We can change repo and credentials when ready for stable release.

[hkir-dev]

New python package released

• Documentation support for pattern files added
• Schema documentation interface changed. No need to give schema as input, uses schema provided by the python package
• Generated schema documentation updated to increase readability: 'array' changed to 'list', 'object' changed to 'dict', 'Refer to *#/definitions/x' changed to 'Values from x'

Usage

From the CLI

$ dosdp validate -i <test.yaml or 'test folder'>

$ dosdp document --schema
$ dosdp document --schema -o <schema.md>

$ dosdp document -i <pattern.yaml> -o <pattern.md>

From Python

from dosdp import validator

validator.validate("test.yaml")

[matentzn]

Awesome! This is super cool, agree!

One thing missing:

dosdp document -i <test.yaml or 'test folder'> -o <'out folder'>

So that this overview table of all yamls in a folder and the overview is created

[hkir-dev]

Update implemented. If input is a folder, an "overview.md" file is generated in the output folder along with the pattern documentations.

[matentzn]

Given the document -i, how is the data preview created? shouldn't there be a link to a data directory as well?

[dosumis]

Final repo cleanup.

Move lib, patterns, spec & any stray unused files -> attic.

[hkir-dev]

@matentzn inorder to make pattern documentation interface compatible with the previous schema documentation interface, I had removed the sample data parameter. Now I added that sample data parameter back and made a new relase

https://test.pypi.org/project/dosdp/0.1.7.dev1/

Usage

From the CLI

$ dosdp validate -i <test.yaml or 'test folder'>

$ dosdp document --schema
$ dosdp document --schema -o <schema.md>

$ dosdp document -i <pattern.yaml or pattern folder> -o <pattern.md or output folder>
$ dosdp document -i <pattern.yaml or pattern folder> -d <sample data folder> -o <pattern.md or pattern folder>

From Python

from dosdp import validator

validator.validate("test.yaml")

[matentzn]

Awesome, thank you! I will have to integrate this at some point into ODK, when I make a larger push on pattern infrastructure.