Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't drop null fields from request bodies #19068

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

Ichimonji10
Copy link
Contributor

The python API bindings drop null request field from HTTP request
bodies. This is especially problematic for HTTP PUT and PATCH requests,
where null values are semantically important.

Change the behavior of the bindings:

# before
model_dump(by_alias=True, exclude=excluded_fields, exclude_none=True)

# after
model_dump(by_alias=True, exclude_unset=True)

The effect of this change is to make the API send data to the remote
server that hews more closely to the original model. This is important
when making HTTP PUT and PATCH requests:

# model describing a movie you've watched
class LogEntry(BaseModel):
    name: str
    rating: int | None = None

# create log entry
log_entry = LogEntry(name="Blade", rating=5)
api_client.put_log_entry(1, log_entry)

# clear rating; PUT {"name": "Blade", "rating": null}
log_entry = LogEntry(name="Blade", rating=None)
api_client.put_log_entry(1, log_entry)

# clear rating; PATCH {"rating": null}
log_entry = LogEntry.model_construct(rating=None)
api_client.patch_log_entry(1, log_entry)

Before this change, the API client would omit "rating": null from
request bodies.

Fix: #19067

@Ichimonji10
Copy link
Contributor Author

ping python tech committee members:

@cbornet @tomplus @krjakbrjak @fa0311 @multani

elif isinstance(obj, dict):
obj_dict = obj
elif hasattr(obj, "model_dump") and callable(obj.model_dump):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unfortunately the way to_dict/from_dict integrate this breaks in the face of the from/to dict layer the codegen still ads

as all from_dict call convert missing fields to None, pydantic has no chance to distinguish passed in from missing

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If both the serialization and deserialization logic has issues, then do they need to be solved at the same time? That is, if this merge request successfully improves upon serialization behavior, then can we consider improving deserialization as a separate problem?

If we do need to improve both serialization and deserialization logic at the same time, then do you have a suggestion as for how to do that?

@wing328
Copy link
Member

wing328 commented Jul 4, 2024

Thanks for the PR but your commit (as shown in the Commits tab) is not linked to your Github account, which means this PR won't count as your contribution in https://github.com/OpenAPITools/openapi-generator/graphs/contributors.

Let me know if you need help fixing it.

Ref: https://github.com/OpenAPITools/openapi-generator/wiki/FAQ#how-can-i-update-commits-that-are-not-linked-to-my-github-account

@Ichimonji10
Copy link
Contributor Author

Thanks for the PR but your commit (as shown in the Commits tab) is not linked to your Github account, which means this PR won't count as your contribution in https://github.com/OpenAPITools/openapi-generator/graphs/contributors.

Hmm, I don't see this issue. The commits in the "Commits" tab point to my account. And per git log, the commit author is Author: Jeremy Audet <[email protected]>, which is the same as my primary GitHub email account.

Let me know if you need help fixing it.

Ref: https://github.com/OpenAPITools/openapi-generator/wiki/FAQ#how-can-i-update-commits-that-are-not-linked-to-my-github-account

@multani
Copy link
Contributor

multani commented Jul 5, 2024

@Ichimonji10 would you be able to also add one or more tests to test the new behavior and prevent regressions?

@Ichimonji10
Copy link
Contributor Author

@multani Sure, can do. Can you please look at #19091? Once that's merged, I'll rebase this and update the unit tests.

The python API bindings drop null request field from HTTP request
bodies. This is especially problematic for HTTP PUT and PATCH requests,
where null values are semantically important.

Change the behavior of the bindings:

```python
# before
model_dump(by_alias=True, exclude=excluded_fields, exclude_none=True)

# after
model_dump(by_alias=True, exclude_unset=True)
```

The effect of this change is to make the API send data to the remote
server that hews more closely to the original model. This is important
when making HTTP PUT and PATCH requests:

```python
# model describing a movie you've watched
class LogEntry(BaseModel):
    name: str
    rating: int | None = None

# create log entry
log_entry = LogEntry(name="Blade", rating=5)
api_client.put_log_entry(1, log_entry)

# clear rating; PUT {"name": "Blade", "rating": null}
log_entry = LogEntry(name="Blade", rating=None)
api_client.put_log_entry(1, log_entry)

# clear rating; PATCH {"rating": null}
log_entry = LogEntry.model_construct(rating=None)
api_client.patch_log_entry(1, log_entry)
```

Before this change, the API client would omit `"rating": null` from
request bodies.

Fix: OpenAPITools#19067
Done with:

```bash
./bin/generate-samples.sh ./bin/configs/python*.yaml
```
@Ichimonji10
Copy link
Contributor Author

I see that this test assertion is failing:

# samples/client/echo_api/python/tests/test_manual.py
self.assertEqual(api_response, "{'name': 'testing', 'photoUrls': ['http://1', 'http://2']}")

I can fix that test assertion easily enough, but I'm not quite sure how to go about it. There are three test_manual.py modules containing similar tests, making me think that they're all generated somehow. And the test cases are also incorrectly executed by pytest, instead of unittest, making me wonder how valid the test_manual.py test cases are. Does anyone know? The layout of the code base is a little mysterious to me.

@fa0311
Copy link
Contributor

fa0311 commented Jul 9, 2024

@Ichimonji10
The test_manual.py test cases are valid tests and were created manually. The tests are executed using pytest. For more details, you can refer to the GitHub workflow configuration here.
If there are errors in the tests, feel free to correct them as needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] Null fields dropped from HTTP request bodies
5 participants