[INF] Simplify environment #1133

Zeroto521 · 2022-07-14T02:11:31Z

PR Description

Please describe the changes proposed in the pull request:

To keep minimal dependencies.
Split environment
- required: base, work for functions
- optional: work for others
- testing: work for testing
- dev: work for the developer in local
- doc: work for doc building

this /environment-dev.yaml will be kept for the developer to quickly install pyjanitor environment.
And /ci/envs/* will be created work for CI to test pyjanitor compatibility.

This PR resolves #1127.

PR Checklist

Please ensure that you have done the following:

PR in from a fork off your branch. Do not PR from <your_username>:dev, but rather from <your_username>:<feature-branch_name>.

If you're not on the contributors list, add yourself to AUTHORS.md.

Add a line to CHANGELOG.md under the latest version header (i.e. the one that is "on deck") describing the contribution.
- Do use some discretion here; if there are multiple PRs that are related, keep them in a single line.

Automatic checks

There will be automatic checks run on the PR. These include:

Building a preview of the docs on Netlify
Automatically linting the code
Making sure the code is documented
Making sure that all tests are passed
Making sure that code coverage doesn't go down.

Relevant Reviewers

Please tag maintainers to review.

@ericmjl

Zeroto521 · 2022-07-14T02:14:14Z

environment-dev.yml

@@ -2,54 +2,43 @@ name: pyjanitor-dev
 channels:
  - conda-forge
 dependencies:
-  - python=3.9
+  - python>=3.6


Now minimal python version is 3.6
It may don't work in py3.6 with pandas 1.4.1.
it will be updated after setting ci/env multi-envs

pandas minimum python version is 3.8, I feel we should make it that, or at least 3.7 because of dictionary order

py3.6 base on setup.py.
As you said, this is the reason I want to build multi-envs
we could test pyjanitor compatible with python3.6 after ci/env/ was built.

pyjanitor/setup.py

Line 124 in 1908f35

python_requires=">=3.6",

@samukweku @Zeroto521 thanks for doing the PR and reviewing!

I think @samukweku's suggestion makes sense here. With pandas minimum version being 3.8, we should just follow along. I think setup.py should be updated.

Zeroto521 · 2022-07-14T02:17:09Z

environment-dev.yml

+  - pandas
+  - pandas-flavor
+  - multipledispatch
+  - scipy


scipy I thought also could as an optional dependency.
there are only two scripts that use scipy

math.py

functions/impute.py

Zeroto521 · 2022-07-14T02:19:30Z

environment-dev.yml

  - make
+  # release
+  - twine


I thought we could drop twine. we use Github action to build and realse.

Zeroto521 · 2022-07-14T02:23:37Z

environment-dev.yml

+  - pre-commit
+  - pylint
  - isort


We could drop pylint and isort since we start to use pre-commit.
once we drop these dependencies Makefile also should be updated.

Having pylint and isort binaries in the environment helps with those who use VSCode for code checking. Could we ensure that these stay in the environment while also annotating with comments that they are present for developer convenience?

I already keep IDE code style checking packages (black, pylint, isort)

I also made a PR #1231 making similar changes. I agree with keeping black/isort, for pylint: we don't enforce it via pre-commit, instead we use flake8 as our main linting tool. So I don't see the need for keeping it around.

codecov · 2022-07-14T02:29:35Z

Codecov Report

Merging #1133 (94effb2) into dev (6c43b9d) will decrease coverage by 55.36%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##              dev    #1133       +/-   ##
===========================================
- Coverage   97.34%   41.98%   -55.37%     
===========================================
  Files          77       77               
  Lines        3240     3244        +4     
===========================================
- Hits         3154     1362     -1792     
- Misses         86     1882     +1796

environment-dev.yml

Zeroto521 · 2022-07-14T02:48:17Z

@samukweku Hi, this problem (test_pivot_longer.py fail) needs you to have a look.
https://github.com/pyjanitor-devs/pyjanitor/runs/7332617267?check_suite_focus=true

samukweku · 2022-07-14T03:12:44Z

@Zeroto521 is it possible to extend this to what users can install? I feel it might help with #826 and #793

samukweku · 2022-07-14T03:15:03Z

@samukweku Hi, this problem (test_pivot_longer.py fail) needs you to have a look.
https://github.com/pyjanitor-devs/pyjanitor/runs/7332617267?check_suite_focus=true

I feel this is triggered because in pandas 3.6, the order of a dictionary is not assured; compared to 3.7 and above, where the order of a dictionary is assured. WIth this, I suggest we make the minimum python version 3.8, as pandas minimum version is 3.8

Zeroto521 · 2022-07-14T03:19:35Z

@Zeroto521 is it possible to extend this to what users can install? I feel it might help with #826 and #793

I'm in the progress to do this, see #1127.

Zeroto521 · 2022-07-14T03:36:45Z

I feel this is triggered because in pandas 3.6, the order of a dictionary is not assured; compared to 3.7 and above, where the order of a dictionary is assured. WIth this, I suggest we make the minimum python version 3.8, as pandas minimum version is 3.8

conda will install python from a high version to start
if there is an error happening, then install a lower version instead.
And the python version is 3.10.
Does it seem pyjanitor can't work well with py3.10?

 + python                               3.10.5  h582c2e5_0_cpython   conda-forge/linux-64      30MB

https://github.com/pyjanitor-devs/pyjanitor/runs/7332617267?check_suite_focus=true#step:3:834

Zeroto521 · 2022-08-05T11:56:45Z

I finally get some spare time to continue this work. Hope I finished this pr on weekends.

Simplify environment

2ead22e

Zeroto521 commented Jul 14, 2022

View reviewed changes

should be pip-tools

e43d6e7

Zeroto521 commented Jul 14, 2022

View reviewed changes

Zeroto521 added 3 commits July 14, 2022 10:27

Add blank lines

7e248f6

simplify a bit

22a9023

simplify a bit

f3f083a

Zeroto521 commented Jul 14, 2022

View reviewed changes

environment-dev.yml Show resolved Hide resolved

Zeroto521 commented Jul 14, 2022

View reviewed changes

environment-dev.yml Show resolved Hide resolved

samukweku assigned Zeroto521 Jul 14, 2022

samukweku added the infrastructure Infrastructure-related issues label Jul 14, 2022

Merge branch 'dev' into ci-environment

b0605fa

Zeroto521 mentioned this pull request Aug 6, 2022

[INF] Update mkdocstrings version to 0.19.0 #1138

Merged

3 tasks

ericmjl closed this in #1138 Aug 10, 2022

Zeroto521 reopened this Aug 10, 2022

Zeroto521 mentioned this pull request Aug 10, 2022

[INF] Set independent environment for building documentation #1141

Merged

3 tasks

Zeroto521 force-pushed the ci-environment branch from 229c9ee to b0605fa Compare August 11, 2022 03:40

Zeroto521 added 4 commits August 11, 2022 11:46

Merge branch 'dev' into ci-environment

67d96df

Remove 'twine'

5a956c0

keep IDE code style checking package

a9a607d

Add 'requests'

878fc32

Add marks

94effb2

Zeroto521 mentioned this pull request Aug 11, 2022

[INF] Set multi-test env to compat dependency and system #1143

Open

5 tasks

ericmjl mentioned this pull request Aug 15, 2022

[INF/DOC] Documentation bugfix by pinning mkdocstrings to 0.18.1 #1147

Merged

thatlittleboy mentioned this pull request Jan 23, 2023

[INF] update some mkdocs infrastructure #1231

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[INF] Simplify environment #1133

[INF] Simplify environment #1133

Zeroto521 commented Jul 14, 2022 •

edited

Loading

Zeroto521 Jul 14, 2022

samukweku Jul 14, 2022 •

edited

Loading

Zeroto521 Jul 14, 2022

ericmjl Aug 5, 2022

Zeroto521 Jul 14, 2022

Zeroto521 Jul 14, 2022

Zeroto521 Jul 14, 2022

ericmjl Aug 6, 2022

Zeroto521 Aug 11, 2022

thatlittleboy Jan 23, 2023

codecov bot commented Jul 14, 2022 •

edited

Loading

Zeroto521 commented Jul 14, 2022

samukweku commented Jul 14, 2022

samukweku commented Jul 14, 2022 •

edited

Loading

Zeroto521 commented Jul 14, 2022 •

edited

Loading

Zeroto521 commented Jul 14, 2022

Zeroto521 commented Aug 5, 2022

[INF] Simplify environment #1133

Are you sure you want to change the base?

[INF] Simplify environment #1133

Conversation

Zeroto521 commented Jul 14, 2022 • edited Loading

PR Description

PR Checklist

Automatic checks

Relevant Reviewers

Choose a reason for hiding this comment

samukweku Jul 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jul 14, 2022 • edited Loading

Codecov Report

Zeroto521 commented Jul 14, 2022

samukweku commented Jul 14, 2022

samukweku commented Jul 14, 2022 • edited Loading

Zeroto521 commented Jul 14, 2022 • edited Loading

Zeroto521 commented Jul 14, 2022

Zeroto521 commented Aug 5, 2022

Zeroto521 commented Jul 14, 2022 •

edited

Loading

samukweku Jul 14, 2022 •

edited

Loading

codecov bot commented Jul 14, 2022 •

edited

Loading

samukweku commented Jul 14, 2022 •

edited

Loading

Zeroto521 commented Jul 14, 2022 •

edited

Loading