Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exclusively build for pwdata and filter out modules #3

Merged
merged 38 commits into from
May 1, 2024
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
fcefe35
Add `exclusives` and `ignored_mods` to filter out irrelevant modules
jacksongoode Apr 22, 2024
bc8ec6c
Linting
jacksongoode Apr 23, 2024
4484b96
Return even earlier
jacksongoode Apr 24, 2024
332ed7e
Hm
jacksongoode Apr 24, 2024
9145079
Hmm
jacksongoode Apr 24, 2024
3fbb617
Fix
jacksongoode Apr 24, 2024
09ace52
Hm
jacksongoode Apr 24, 2024
8f2107b
Remove print
jacksongoode Apr 24, 2024
227325d
Log
jacksongoode Apr 24, 2024
4668151
Hm
jacksongoode Apr 24, 2024
0c7f1ec
Fix
jacksongoode Apr 24, 2024
165a26d
Fix
jacksongoode Apr 24, 2024
9d70e92
Fix
jacksongoode Apr 24, 2024
0f498e2
Test
jacksongoode Apr 24, 2024
d150cb5
Remove
jacksongoode Apr 24, 2024
712ebb1
Add default
jacksongoode Apr 30, 2024
0f25a10
Print
jacksongoode Apr 30, 2024
cdf8c30
Edit
jacksongoode Apr 30, 2024
a478da3
Fix
jacksongoode Apr 30, 2024
7f944eb
Edit
jacksongoode Apr 30, 2024
7270fe7
Edit
jacksongoode Apr 30, 2024
9482506
Edit
jacksongoode Apr 30, 2024
3c36d6f
Edit
jacksongoode Apr 30, 2024
0036baf
Fix
jacksongoode Apr 30, 2024
9a5ce92
Fix
jacksongoode Apr 30, 2024
c482243
Edit
jacksongoode May 1, 2024
5713719
Fix
jacksongoode May 1, 2024
5d950da
Fix
jacksongoode May 1, 2024
a159b35
Fast re search
jacksongoode May 1, 2024
dc182c7
Edit
jacksongoode May 1, 2024
c5319c5
Opps
jacksongoode May 1, 2024
48f5731
Move imported_name down
jacksongoode May 1, 2024
b57b4c3
Fix
jacksongoode May 1, 2024
8946577
Tet
jacksongoode May 1, 2024
5cb4728
Skip
jacksongoode May 1, 2024
7daf94e
Remove prints
jacksongoode May 1, 2024
697c4ad
Cleanup
jacksongoode May 1, 2024
b2a5d15
Fix
jacksongoode May 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions pycg/processing/postprocessor.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ def __init__(
self,
input_file,
modname,
exclusives,
ignored_mods,
import_manager,
scope_manager,
def_manager,
Expand All @@ -38,6 +40,8 @@ def __init__(
modules_analyzed=None,
):
super().__init__(input_file, modname, modules_analyzed)
self.exclusives = exclusives
self.ignored_mods = ignored_mods
self.import_manager = import_manager
self.scope_manager = scope_manager
self.def_manager = def_manager
Expand Down Expand Up @@ -329,6 +333,8 @@ def update_parent_classes(self, defi):
def analyze_submodules(self):
super().analyze_submodules(
PostProcessor,
self.exclusives,
self.ignored_mods,
self.import_manager,
self.scope_manager,
self.def_manager,
Expand Down
21 changes: 21 additions & 0 deletions pycg/processing/preprocessor.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@
self,
filename,
modname,
exclusives,
ignored_mods,
import_manager,
scope_manager,
def_manager,
Expand All @@ -40,6 +42,8 @@
super().__init__(filename, modname, modules_analyzed)

self.modname = modname
self.exclusives = exclusives
self.ignored_mods = ignored_mods
self.mod_dir = "/".join(self.filename.split("/")[:-1])

self.import_manager = import_manager
Expand Down Expand Up @@ -71,6 +75,8 @@
super().analyze_submodule(
PreProcessor,
modname,
self.exclusives,
self.ignored_mods,
self.import_manager,
self.scope_manager,
self.def_manager,
Expand All @@ -81,6 +87,10 @@

def visit_Module(self, node):
def iterate_mod_items(items, const):
items = [
i for i in items if not \

Check failure on line 91 in pycg/processing/preprocessor.py

View workflow job for this annotation

GitHub Actions / flake8

the backslash is redundant between brackets
any(ignored_mod in i for ignored_mod in self.ignored_mods)

Check failure on line 92 in pycg/processing/preprocessor.py

View workflow job for this annotation

GitHub Actions / flake8

continuation line unaligned for hanging indent
]
for item in items:
defi = self.def_manager.get(item)
if not defi:
Expand Down Expand Up @@ -205,6 +215,14 @@
tgt_name = import_item.asname if import_item.asname else import_item.name
imported_name = self.import_manager.handle_import(src_name, level)

# Limit to exclusive module
if src_name.split(".")[0] not in self.exclusives:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens here if self.exclusives is None?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, I'll add their default to be an empty array.

continue

# Skip ignored modules
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to skip ignored modules down here and remove them from times above?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Below is what happens when PyCG looks at an import declaration, the other ... in self.ignored_mods: case above is when we visit a module. I did this as a safety precaution more so I think PyCG could end up traversing through a ignored modules in both cases - though I could be wrong about this and we can test.

if tgt_name in self.ignored_mods:
continue

if not imported_name:
add_external_def(src_name, tgt_name)
continue
Expand All @@ -223,6 +241,9 @@

# handle all modules that were not analyzed
for modname in self.import_manager.get_imports(self.modname):
if modname.split(".")[0] not in self.exclusives:
continue

fname = self.import_manager.get_filepath(modname)

if not fname:
Expand Down
10 changes: 9 additions & 1 deletion pycg/pycg.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,13 @@


class CallGraphGenerator(object):
def __init__(self, entry_points, package, max_iter, operation):
def __init__(
self, entry_points, package, exclusives, ignored_mods, max_iter, operation
):
self.entry_points = entry_points
self.package = package
self.exclusives = exclusives
self.ignored_mods = ignored_mods
self.state = None
self.max_iter = max_iter
self.operation = operation
Expand Down Expand Up @@ -162,6 +166,8 @@ def analyze(self):
self.do_pass(
PreProcessor,
True,
self.exclusives,
self.ignored_mods,
self.import_manager,
self.scope_manager,
self.def_manager,
Expand All @@ -179,6 +185,8 @@ def analyze(self):
self.do_pass(
PostProcessor,
False,
self.exclusives,
self.ignored_mods,
self.import_manager,
self.scope_manager,
self.def_manager,
Expand Down
Loading