Query engine for CMF #131
Draft
Query engine for CMF #131
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit adds doc strings and type annotation for CmfQuery class methods.
- References to `client` in the source code (e.g., CmfClient instead of CmfQuery) are removed.
- Several bugs related to checking input dict parameters are fixed (e.g., `d = d or {}` where it should be `if d is None: d= {}`). Now, the corrent object is returned when input dict is just empty.
- Missing doc strings and type annotations are added.
- Additional checks and log messages are added.
Fixing one possible bug related to accessing a column in a data frame when this data frame is empty. Adding key mapper classes to help map source to target keys when copying dictionaries.
The API implements graph-like API to traverse CMF metadata in a graph-like manner. The entry point is the `MetadataStore` class that retrieves from metadata store pipelines, stages, executions and artifacts. Users can specify search query to specify what they want to return (the query is basically the value for the `filter_query` parameter of the `ListOptions` class ML Metadata (MLMD) library). Each node mentioned above (pipeline, stages, executions and artifacts) havs its own Python wrapper class that provides developer-friendly API to access node's parameters and travers graph of machine learning concepts (e.g., get all stages of a pipeline or get all executions of a stage). The graph API also provides the `Properties` wrapper for MLMD's properties and custom_propertied nodes' fields. This wrapper implements the `Mapping` API and automatically converts MLMD's values to Python values on the fly.
- Adding unit tests (97% coverage). - Renaming certain classes, redesigning implementation of several methods. - Adding `Type` class that represents concepts such as ContextType, ExecutionType and ArtifactType in MLMD library.
- Implementation of multiple analytic functions. - Unified mechanism to traverse graph of artifacts along their dependency paths.
- New traverse API. - Base methods. - Allow users to specify selection criteria for artifacts in some methods.
- Adding HPE header. - Improving doc strings for classes that are responsible for traversing MLMD graph of artifacts.
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related Issues / Pull Requests
Description
This PR introduces a
Query Engineanalytic engine, one of several analytic engines to be developed for CMF.The query engine proposed in this PR is one of the first steps towards achieving the above-mentioned goal. Its purpose is to provide query and search functionality that is considered to be high-level to be part of lower-level APIs (such as graph or tabular API to access ML and pipeline metadata).
What changes are proposed in this pull request?
Checklist: