Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bulk deletion of site-package/tests #951

Merged
merged 1 commit into from
Dec 22, 2023
Merged

Bulk deletion of site-package/tests #951

merged 1 commit into from
Dec 22, 2023

Conversation

MukuFlash03
Copy link
Contributor

Added a one line pipe command to remove all occurrences of site-packages/tests occurring in miniconda main directory.
The pipeline works like:

  1. Find all packages containing "tests" and having path as "site-packages/tests".
  2. Grep only those which have tests immediately below site-package as a subfolder.
  3. Remove these folders.

Bulk deletion of site-package/tests
1159eac 
Added a one line pipe command to remove all occurrences of site-packages/tests occurring in miniconda main directory.
@shankari shankari merged commit 54659fb into e-mission:master Dec 22, 2023
5 checks passed
MukuFlash03 pushed a commit to MukuFlash03/e-mission-server that referenced this pull request Jan 13, 2024
Squashed commit of the following:
31626ba 
commit 54659fb
Merge: cf0c9e2 1159eac
Author: K. Shankari <[email protected]>
Date:   Thu Dec 21 20:17:15 2023 -0800

    Merge pull request e-mission#951 from MukuFlash03/fix-vuln

    Bulk deletion of site-package/tests

commit 1159eac
Author: Mahadik, Mukul Chandrakant <[email protected]>
Date:   Thu Dec 21 20:43:39 2023 -0700

    Bulk deletion of site-package/tests

    Added a one line pipe command to remove all occurrences of site-packages/tests occurring in miniconda main directory.

commit cf0c9e2
Merge: d2f38bc 3be2757
Author: K. Shankari <[email protected]>
Date:   Thu Dec 21 17:47:27 2023 -0800

    Merge pull request e-mission#950 from MukuFlash03/fix-vuln

    Remove obsolete package versions

commit 3be2757
Author: Mahadik, Mukul Chandrakant <[email protected]>
Date:   Thu Dec 21 18:05:23 2023 -0700

    Remove obsolete package versions

    Cleaned up older versions for two packages:
    urllib3 - deleted stale version folders
    python - deleted tests folder

commit d2f38bc
Merge: 978a719 c1b0889
Author: K. Shankari <[email protected]>
Date:   Wed Dec 20 14:31:09 2023 -0800

    Merge pull request e-mission#949 from MukuFlash03/fix-vuln

    Fixing latest Docker image vulnerabilities

commit c1b0889
Author: Mahadik, Mukul Chandrakant <[email protected]>
Date:   Mon Dec 18 11:04:25 2023 -0700

    Upgraded Ubuntu base image

    Latest Ubuntu base image was just released officially by Docker which contains updated version of libc6 and libc-bin.

commit 07747d0
Author: Mahadik, Mukul Chandrakant <[email protected]>
Date:   Fri Dec 15 18:38:12 2023 -0700

    Fixing latest Docker image vulnerabilities

    AWS Inspector found the following vulnerable packages:

    CRITICAL
    perl

    HIGH
    nghttp2, libnghttp2-14
    cryptography, libssl3
    cryptography
    libc6, libc-bin

    Upgraded perl, libssl3, nghttp2 packages by upgrading base Ubuntu image to latest of the same LTS version - jammy (22.04).

    Cryptography package was fixed by mentioning required version to be installed using conda.

    Libc6, Libc-bin can be fixed by using apt-get upgrade but this upgrades all packages which is not recommended as a blanket upgrade fix.
shankari added a commit that referenced this pull request Feb 10, 2024
@MukuFlash03 @shankari
Store limited number of models and clear older ones (#948)
911e1ec 
* Added details in logging statements

Completed marked future fixes for PR-944
Provided more detailed info regarding number of distinct users, as well  as number of predictions made for the trip_list.

* Implemented logic to clear models

Clearing or trimming number of models for a particular user from the Stage_updateable_models collection.

Constant K_MODELS_COUNT defines the number of models to keep.

Currently tested with model data available in stage dataset snapshot.

Pending - Complete testing with assertions to be added for testing.

* Added function call

Called the trim_model_entries function in upsert_model to delete models before inserting new ones.

* Squashed commit of the following:

commit 54659fb
Merge: cf0c9e2 1159eac
Author: K. Shankari <[email protected]>
Date:   Thu Dec 21 20:17:15 2023 -0800

    Merge pull request #951 from MukuFlash03/fix-vuln

    Bulk deletion of site-package/tests

commit 1159eac
Author: Mahadik, Mukul Chandrakant <[email protected]>
Date:   Thu Dec 21 20:43:39 2023 -0700

    Bulk deletion of site-package/tests

    Added a one line pipe command to remove all occurrences of site-packages/tests occurring in miniconda main directory.

commit cf0c9e2
Merge: d2f38bc 3be2757
Author: K. Shankari <[email protected]>
Date:   Thu Dec 21 17:47:27 2023 -0800

    Merge pull request #950 from MukuFlash03/fix-vuln

    Remove obsolete package versions

commit 3be2757
Author: Mahadik, Mukul Chandrakant <[email protected]>
Date:   Thu Dec 21 18:05:23 2023 -0700

    Remove obsolete package versions

    Cleaned up older versions for two packages:
    urllib3 - deleted stale version folders
    python - deleted tests folder

commit d2f38bc
Merge: 978a719 c1b0889
Author: K. Shankari <[email protected]>
Date:   Wed Dec 20 14:31:09 2023 -0800

    Merge pull request #949 from MukuFlash03/fix-vuln

    Fixing latest Docker image vulnerabilities

commit c1b0889
Author: Mahadik, Mukul Chandrakant <[email protected]>
Date:   Mon Dec 18 11:04:25 2023 -0700

    Upgraded Ubuntu base image

    Latest Ubuntu base image was just released officially by Docker which contains updated version of libc6 and libc-bin.

commit 07747d0
Author: Mahadik, Mukul Chandrakant <[email protected]>
Date:   Fri Dec 15 18:38:12 2023 -0700

    Fixing latest Docker image vulnerabilities

    AWS Inspector found the following vulnerable packages:

    CRITICAL
    perl

    HIGH
    nghttp2, libnghttp2-14
    cryptography, libssl3
    cryptography
    libc6, libc-bin

    Upgraded perl, libssl3, nghttp2 packages by upgrading base Ubuntu image to latest of the same LTS version - jammy (22.04).

    Cryptography package was fixed by mentioning required version to be installed using conda.

    Libc6, Libc-bin can be fixed by using apt-get upgrade but this upgrades all packages which is not recommended as a blanket upgrade fix.

* Updated Test file with dummy models

Was using pre-built models for testing that were present in stage_snapshot dataset but these won't be available in the normal dataset.

Hence, took reference from emission.tests.modellingTests.TestRunGreedyModel.py for creating test models using dummy trip data.

Creating multiple such models and then checking for trimming operation success.

* Reverted commit 31626ba + Removed log statements

Commit 31626ba included changes related to vulnerability fixes, hence reverted them.

Had added some log statements related to scalability fixes for model loading.
Removed them and will add in separate PR.

* Changed imports to access K_MODEL_COUNT

Removed redundant "esda" import added possibly during local testing.

Removed get_model_limit() as K_MODEL_COUNT is a class variable and can be accessed directly using class name.

* Revert "Squashed commit of the following:"

This reverts commit 31626ba.

Earlier commit f54e85f missed out the discarded reverted changes relating to the vulnerability fixes.
Hence removing them in these.

* Included maximum_stored_model_limit parameter in trip_model config file

Decided upon threshold value for model count above which redundant models will be deleted.
This was agreed upon to be 3 and is defined in the trip_model.config.json.sample file.

Necessary changes have been made in the related files to use this config value.

* Addressed latest review comments

- Improved logging statements to included whether model count has changed.
- Skipped check for saved test data in Test file as we clear teardown() related collections in the DB.

* Restored inferrers.py

Did this to revert a whitespace change which was possibly due to removing a line at the end of the file.

We don't want it to be modified as it is unrelated to this set of PR changes.

Restored using this command by pointing to specific commit hash for the first instance where the file was modified.

git restore --source=79ad1cc~1 emission/analysis/classification/inference/labels/inferrers.py

* Updated Test for Storing limited models

Added test to ensure that only the latest models are stored by comparing the write_ts times.

* Revert whitespace changes

* Revert whitespace changes

---------

Co-authored-by: Mahadik, Mukul Chandrakant <[email protected]>
Co-authored-by: K. Shankari <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@MukuFlash03 @shankari