[Docs] pydantic integration: create flytesnacks example page #4066
Comments
samhita-alla
added
documentation
|
Hi. Can I work on this issue? |
|
Yup, assigned to you, thanks! |
|
Hello @pingsutw , please assign this issue to me as I have already worked on this kind of problem in past and has a great experience. |
|
@pingsutw I'm still working on it |
|
I'm having trouble getting the plugin to behave as described. @pingsutw perhaps you can offer some guidance? I'm using Python 3.9.11 on macos. These are the python dependencies I've installed: I also installed I created this short script similar to the flytekit docs. I verified that I could get the example in the docs working, and then modified it to use the pydantic plugin. # train_logistic_regression.py
from pydantic import BaseModel
import pandas as pd
from sklearn.datasets import load_wine
from sklearn.linear_model import LogisticRegression
from flytekit import task, workflow
class Config(BaseModel):
C: float = 1.0
max_iter: int = 100
@task
def get_data() -> pd.DataFrame:
"""Get the wine dataset."""
return load_wine(as_frame=True).frame
@task
def process_data(data: pd.DataFrame) -> pd.DataFrame:
"""Simplify the task from a 3-class to a binary classification problem."""
return data.assign(target=lambda x: x["target"].where(x["target"] == 0, 1))
@task
def train_model(data: pd.DataFrame, config: Config) -> LogisticRegression:
"""Train a model on the wine dataset."""
features = data.drop("target", axis="columns")
target = data["target"]
return LogisticRegression(**config.dict()).fit(features, target)
@workflow
def training_workflow(config: Config) -> LogisticRegression:
"""Put all of the steps together into a single workflow."""
data = get_data()
processed_data = process_data(data=data)
return train_model(
data=processed_data,
config=config,
)I am running the script with the command I then get this error: |
|
@galbwe, could you initialize config in the workflow itself, i.e. provide a default value to |
|
@samhita-alla I changed the workflow definition to @workflow
def training_workflow(config: Config = Config()) -> LogisticRegression:
"""Put all of the steps together into a single workflow."""
data = get_data()
processed_data = process_data(data=data)
return train_model(
data=processed_data,
config=config,
)then ran Now I'm getting this error: Then I tried adding an empty So it seems like Then I tried hard coding the pydantic model in the workflow and that seemed to work. @workflow
def training_workflow() -> LogisticRegression:
"""Put all of the steps together into a single workflow."""
config = Config(C=0.1, max_iter=1000)
data = get_data()
processed_data = process_data(data=data)
return train_model(
data=processed_data,
config=config,
) |
|
I guess I was misled by this test that passes pydantic models directly to workflows. I think a |
|
@galbwe, |
|
Sure, here is the verbose output. Thanks. % pyflyte --verbose run ./pydantic_plugin/train_logistic_regression.py training_workflow
2023-10-04 12:41:49,366938 WARNING {"asctime": "2023-10-04 12:41:49,366", "name": "flytekit", "levelname": "WARNING", "message": "Unsupported Type <class 'sklearn.linear_model._logistic.LogisticRegression'> found, Flyte will default to type_engine.py:1141
use PickleFile as the transport. Pickle can only be used to send objects between the exact same version of Python, and we strongly recommend to use python type that flyte support."}
Verbose mode on
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ /Users/wes/code/projects/flyteorg/flytesnacks/examples/pydantic_plugin/venv/bin/pyflyte:8 in <module> │
│ │
│ ❱ 8 │ sys.exit(main()) │
│ │
│ /Users/wes/code/projects/flyteorg/flytesnacks/examples/pydantic_plugin/venv/lib/python3.9/site-packages/click/core.py:1157 in __call__ │
│ │
│ ❱ 1157 │ │ return self.main(*args, **kwargs) │
│ │
│ /Users/wes/code/projects/flyteorg/flytesnacks/examples/pydantic_plugin/venv/lib/python3.9/site-packages/rich_click/rich_group.py:21 in main │
│ │
│ ❱ 21 │ │ │ rv = super().main(*args, standalone_mode=False, **kwargs) │
│ │
│ /Users/wes/code/projects/flyteorg/flytesnacks/examples/pydantic_plugin/venv/lib/python3.9/site-packages/click/core.py:1078 in main │
│ │
│ ❱ 1078 │ │ │ │ │ rv = self.invoke(ctx) │
│ │
│ /Users/wes/code/projects/flyteorg/flytesnacks/examples/pydantic_plugin/venv/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/pyflyte.py:87 in invoke │
│ │
│ ❱ 87 │ │ │ │ raise e │
│ │
│ /Users/wes/code/projects/flyteorg/flytesnacks/examples/pydantic_plugin/venv/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/pyflyte.py:83 in invoke │
│ │
│ ❱ 83 │ │ │ return super().invoke(ctx) │
│ │
│ /Users/wes/code/projects/flyteorg/flytesnacks/examples/pydantic_plugin/venv/lib/python3.9/site-packages/click/core.py:1688 in invoke │
│ │
│ ❱ 1688 │ │ │ │ │ return _process_result(sub_ctx.command.invoke(sub_ctx)) │
│ │
│ /Users/wes/code/projects/flyteorg/flytesnacks/examples/pydantic_plugin/venv/lib/python3.9/site-packages/click/core.py:1688 in invoke │
│ │
│ ❱ 1688 │ │ │ │ │ return _process_result(sub_ctx.command.invoke(sub_ctx)) │
│ │
│ /Users/wes/code/projects/flyteorg/flytesnacks/examples/pydantic_plugin/venv/lib/python3.9/site-packages/click/core.py:1682 in invoke │
│ │
│ ❱ 1682 │ │ │ │ cmd_name, cmd, args = self.resolve_command(ctx, args) │
│ │
│ /Users/wes/code/projects/flyteorg/flytesnacks/examples/pydantic_plugin/venv/lib/python3.9/site-packages/click/core.py:1729 in resolve_command │
│ │
│ ❱ 1729 │ │ cmd = self.get_command(ctx, cmd_name) │
│ │
│ /Users/wes/code/projects/flyteorg/flytesnacks/examples/pydantic_plugin/venv/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/run.py:810 in get_command │
│ │
│ ❱ 810 │ │ │ │ to_click_option(ctx, flyte_ctx, input_name, literal_var, python_type, de │
│ │
│ /Users/wes/code/projects/flyteorg/flytesnacks/examples/pydantic_plugin/venv/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/run.py:478 in to_click_option │
│ │
│ ❱ 478 │ │ │ │ default_val = cast(DataClassJsonMixin, default_val).to_json() │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'Config' object has no attribute 'to_json' |
|
@pingsutw should this work? |
|
Hello 👋, this issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will engage on it to decide if it is still applicable. |
|
Hello @davidmirror-ops |
|
@sumana-2705 Thanks, looking forward to your contributions! |
|
@samhita-alla @davidmirror-ops Does the example page refer only to the .py file containing the Pydantic integration example, or should I include any additional files as well? |
|
@sumana-2705 you'll need to add two pages, similar to the integration examples in the docs, like this one: https://docs.flyte.org/en/latest/flytesnacks/examples/ollama_plugin/index.html |
|
Hello @samhita-alla, @davidmirror-ops, I have opened a pull request for this issue. Could you please review it and provide any necessary feedback or suggestions for changes. Thank you. |
Description
The Pydantic integration currently does not have an example page under the integrations section: https://github.com/flyteorg/flytesnacks/tree/master/examples
The purpose of this task is to add (a) a page describing the plugin and how to install it (see here) and (b) an example page on how to use it (see here)
Are you sure this issue hasn't been raised already?
Have you read the Code of Conduct?
The text was updated successfully, but these errors were encountered: