Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

to_msgpack() error: E TypeError: can not serialize 'ExternalPartition' object #269

Open
gshank opened this issue Dec 10, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@gshank
Copy link
Contributor

gshank commented Dec 10, 2024

  • mashumaro version: 3.15
  • Python version: 3.11.8
  • Operating System: MacOS

Description

A to_msgpack() call that worked in 3.14 no longer works for a particular class in 3.15.

What I Did

test script:

from dataclasses import dataclass, field
from typing import Dict, Any, Optional, Union, List

from mashumaro.mixins.msgpack import DataClassMessagePackMixin

@dataclass
class ExternalPartition(DataClassMessagePackMixin):
    name: str = ""
    description: str = ""
    data_type: str = ""
    meta: Dict[str, Any] = field(default_factory=dict)

@dataclass 
class ExternalTable(DataClassMessagePackMixin):
    location: Optional[str] = None
    file_format: Optional[str] = None
    row_format: Optional[str] = None
    tbl_properties: Optional[str] = None
    partitions: Optional[Union[List[ExternalPartition], List[str]]] = None

def test_partitions_serialization():

    part1 = ExternalPartition(
        name="partition 1",
        description="partition 1",
        data_type="string",
    )

    part2 = ExternalPartition(
        name="partition 2",
        description="partition 2",
        data_type="string",
    )

    ext_table = ExternalTable(
        location="my_location",
        file_format="my file format",
        row_format="row format",
        partitions=[part1, part2],
    )

    ext_table_dict = ext_table.to_dict()
    assert isinstance(ext_table_dict["partitions"][0], dict)

    ext_table_msgpack = ext_table.to_msgpack()
    assert ext_table_msgpack

output

>       ext_table_msgpack = ext_table.to_msgpack()

tests/functional/artifacts/test_serialization.py:56: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
<string>:12: in __mashumaro_to_msgpack__
    ???
../mashumaro/mashumaro/mixins/msgpack.py:30: in default_encoder
    return msgpack.packb(data, use_bin_type=True)
env/lib/python3.11/site-packages/msgpack/__init__.py:36: in packb
    return Packer(**kwargs).pack(o)
msgpack/_packer.pyx:279: in msgpack._cmsgpack.Packer.pack
    ???
msgpack/_packer.pyx:276: in msgpack._cmsgpack.Packer.pack
    ???
msgpack/_packer.pyx:270: in msgpack._cmsgpack.Packer._pack
    ???
msgpack/_packer.pyx:213: in msgpack._cmsgpack.Packer._pack_inner
    ???
msgpack/_packer.pyx:270: in msgpack._cmsgpack.Packer._pack
    ???
msgpack/_packer.pyx:232: in msgpack._cmsgpack.Packer._pack_inner
    ???
msgpack/_packer.pyx:270: in msgpack._cmsgpack.Packer._pack
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   TypeError: can not serialize 'ExternalPartition' object

msgpack/_packer.pyx:257: TypeError

msgpack is 1.1.0

@gshank
Copy link
Contributor Author

gshank commented Dec 12, 2024

Actually, it looks like this fails with 3.14 also. In 3.14 with the order Union[List[str], List[ExternalPartition]] the List[ExternalPartition] doesn't always serialize properly (it stays an object). In other cases it's okay (not sure what the difference is). In 3.15 that Union order causes the output to be a stringified object instead of a dictionary, so when I changed it to Union[List[ExternalPartition], List[str]] it converted to a dict okay, but we got the msgpack failure.

@Fatal1ty Fatal1ty added the bug Something isn't working label Dec 17, 2024
@Fatal1ty
Copy link
Owner

Fatal1ty commented Dec 17, 2024

This is certainly a bug related to no_copy_collections parameter. The condition for no copying should be improved because it's now very naive at the moment:

elif spec.origin_type is list:
return f"{spec.expression}.copy()"

Current workaround is to use your own dialect with the empty no_copy_collections tuple (or just make it so right in MessagePackDialect before the first import of your dataclasses 👀):

from mashumaro.mixins.msgpack import (
    DataClassMessagePackMixin,
    MessagePackDialect,
)

class FixedMessagePackDialect(MessagePackDialect):
    no_copy_collections = ()

@dataclass
class ExternalTable(DataClassMessagePackMixin):
    location: Optional[str] = None
    file_format: Optional[str] = None
    row_format: Optional[str] = None
    tbl_properties: Optional[str] = None
    partitions: Optional[Union[List[ExternalPartition], List[str]]] = None

    class Config(BaseConfig):
        dialect = FixedMessagePackDialect

This fixed dialect can be used in other ways, for example, as the to_msgpack dialect argument with the dialect keyword argument enabled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants