Skip to content
This repository has been archived by the owner on Sep 20, 2023. It is now read-only.

reduce analyzer requirements installation time #67

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

xeaon
Copy link

@xeaon xeaon commented Aug 13, 2020

While your solution to install all analyzer requirements looks clean and easy to understand, i've always been annoyed a litte about how time consuming it is to watch pip & pip3 going through all requirements.txt, especially because a lot of analyzers share the same 'base' requirements like cortexutils or requests.

So by deduplicating all requirements and combining them into one single requirements file, you can speed p the hole process. As of today, the linecount of combined_requirements.txt is 71, compared to 277 overall requirements count.

root@cortex:/opt# for I in $(find Cortex-Analyzers -name 'requirements.txt'); do cat >> combined_requirements.txt < $I; sort -u -o combined_requirements.txt combined_requirements.txt; done'
root@cortex:/opt# for I in $(find Cortex-Analyzers -name 'requirements.txt'); do cat >> all_requirements.txt < $I; sort -o all_requirements.txt all_requirements.txt; done

root@cortex:/opt# wc combined_requirements.txt 
  71   78 1142 all_requirements.txt
root@cortex:/opt# wc all_requirements.txt 
 277  290 3427 combined_requirements.tx

It might not be the prettiest solution so feel free to edit, if you're considering to accept this pull request.

while your solution to install all analyzer requirements looks clean and easy to understand, i've always been annoyed about how time consuming it is to watch pip & pip3 going through all requirements.txt.

I've recently tried to reduce the actual steps by deduplicating all dependencies and combining them, which speeds up the process a lot.
@desnij
Copy link

desnij commented Aug 28, 2020

+1 agreed
python2 is supposed to be dead :) https://pythonclock.org/

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants