Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch simplejson to orjson #168

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

Switch simplejson to orjson #168

wants to merge 4 commits into from

Conversation

neyberson
Copy link
Collaborator

@neyberson neyberson commented Nov 20, 2024

orjson simplifies and solves some issues:

So, just adding the option OPT_SERIALIZE_NUMPY in the orjson.dumps(), avoid the issues with numpy and using orjson simplifies the code

@ayjayt
Copy link
Collaborator

ayjayt commented Nov 21, 2024

👍 on my list for today

@ayjayt
Copy link
Collaborator

ayjayt commented Nov 21, 2024

@neyberson this isn't passing tests

Problem with {"id":2,"result":{}}{bye}
 in json: unexpected content after document: line 1 column 21 (char 20)

This message is correct but the new parser is reading it as one message. It's a json message followed by a control message. This needs to be resolved.

@ayjayt
Copy link
Collaborator

ayjayt commented Nov 21, 2024

@neyberson the problem is that with package json also we have it set up to receive partial json messages and also json messages that have junk on the end- it still pulls out the whole message correctly, thats why its read_jsons and not read_json

Copy link
Collaborator

@ayjayt ayjayt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

failing tests

@ayjayt
Copy link
Collaborator

ayjayt commented Nov 21, 2024

also @neyberson this is particularly an issue with windows which, on all other machines, the OS seems to fill and flush the buffer message per message but windows messages sometimes come in partially, sometimes grouped, etc

@neyberson neyberson marked this pull request as draft November 21, 2024 22:57
@ayjayt
Copy link
Collaborator

ayjayt commented Nov 24, 2024

This is paused- orjson is a better option fundamentally but also doesn't support some things we need (streams etc).

At the same time, I think this is 100% necessary for the plolty ecosystem to address some other errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants