PythonMutator: propagate source locations #1783
Conversation
|
|
||
| // LoadLocations is a flag to enable loading Python source locations from the PyDABs. | ||
| // | ||
| // Locations are only supported since PyDABs 0.6.0, and because of that, |
There was a problem hiding this comment.
Shouldn't it just try to load locations and fail gracefully?
There was a problem hiding this comment.
We will enable it by default when we move out of the experimental section. It will be extra difficult to handle the case when the subprocess fails with an unknown argument error and not another error because we will need to parse stderr for that. Not sure it's worth it.
There was a problem hiding this comment.
What do you think about adding flags in the future?
It's not great if we have to break out a flag every time. Not a problem when using env vars.
There was a problem hiding this comment.
Newer versions will ignore and warn about unknown flags. We should have done it from the beginning.
| return nil, fmt.Errorf("failed to open locations file: %w", err) | ||
| } | ||
|
|
||
| defer locationsFile.Close() |
There was a problem hiding this comment.
Isn't the file going to be closed before parse function succeeds?
There was a problem hiding this comment.
You can just read the whole file and pass the bytes to parsePythonLocations
There was a problem hiding this comment.
I'm not 100% sure of the exact order. I think it works because defer is executed only on exit. I will rewrite the code in a way which doesn't require such knowledge :)
There was a problem hiding this comment.
All defer statements are executed upon exiting the frame after the return value is computed.
There was a problem hiding this comment.
|
|
||
| // LoadLocations is a flag to enable loading Python source locations from the PyDABs. | ||
| // | ||
| // Locations are only supported since PyDABs 0.6.0, and because of that, |
There was a problem hiding this comment.
What do you think about adding flags in the future?
It's not great if we have to break out a flag every time. Not a problem when using env vars.
| exists bool | ||
| } | ||
|
|
||
| type pythonLocationEntry struct { |
There was a problem hiding this comment.
I know this is used only for serialization/protocol, but good to have a comment explaining this (or move it to a separate file, together with the loader). If this context is missing it's weird to see the definition of pythonLocations above and see it not use the pythonLocationEntry.
|
|
||
| currentNode.location = location | ||
| currentNode.exists = true | ||
| } |
There was a problem hiding this comment.
Nit: this can be a member function of the type.
There was a problem hiding this comment.
I looked into changing but it doesn't make it obviously better. putPythonLocation/findPythonLocation is not intended to be used outside of merging code, so if we could limit visibility, we would only export mergePythonLocations and parsePythonLocations
| func loadLocationsFile(locationsPath string) (*pythonLocations, error) { | ||
| if _, err := os.Stat(locationsPath); os.IsNotExist(err) { | ||
| return newPythonLocations(), nil | ||
| } |
There was a problem hiding this comment.
You can check for this error as part of os.Open. Saves a syscall.
Also make sure to use errors.Is(err, fs.ErrNotExist) -- os.IsNotExists is marked deprecated.
kanterov
force-pushed
the
kanterov/python-locations
branch
from
def4744 to
43ce278
Compare
Changes
Add a mechanism to load Python source locations in the Python mutator. Previously, locations pointed to generated YAML. Now, they point to Python sources instead. Python process outputs "locations.json" containing locations of bundle paths, examples:
{"path": "resources.jobs.job_0", "file": "src/examples/job_0.py", "line": 3, "column": 5} {"path": "resources.jobs.job_0.tasks[0].task_key", "file": "src/examples/job_0.py", "line": 10, "column": 5} {"path": "resources.jobs.job_1", "file": "src/examples/job_1.py", "line": 5, "column": 7}Such locations form a tree, and we assign locations of the closest ancestor to each
dyn.Valuebased on its path. For example,resources.jobs.job_0.tasks[0].task_keyis located atjob_0.py:10:5andresources.jobs.job_0.tasks[0].email_notificationsis located atjob_0.py:3:5, because we use the location of the job as the most precise approximation.This feature is not yet enabled by default because PyDABs don't yet ignore unknown arguments, and this will be only supported with 0.6.0.
Note: for now, we don't update locations with relative paths, because it has a side effect in changing how these paths are resolved
Example
Tests
Unit tests and manually
Stacked on top of #1782