Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON file containing single 81 MB object won't load in Zui #3025

Open
philrz opened this issue Mar 13, 2024 · 2 comments · Fixed by #3026
Open

JSON file containing single 81 MB object won't load in Zui #3025

philrz opened this issue Mar 13, 2024 · 2 comments · Fixed by #3026
Labels
bug Something isn't working

Comments

@philrz
Copy link
Contributor

philrz commented Mar 13, 2024

tl;dr

A community zync user reported failure to load the JSON file stored at s3://zui-issues/3025/mun.json (it's on S3 because it's too big to attach here in the issue) into Zui. Closer study shows evidence of multiple bugs, so this issue walks through the high-level behaviors. Other issues or PRs will be spawned from here as necessary.

Details

Repro is with Zui commit a7c3aed which uses Zed commit 38763f8.

The community zync user originally asked:

any idea why the following file mun.json would not load in the latest Zui?
I am guessing it may be the (large) size of some geometries, but not sure ...
I should mention this data is public, so it can be shared.

The JSON file is 81 MB in size and contains a single object. A summary:

$ ls -l mun.json 
-rw-r--r--  1 phil  staff  85114393 Mar  7 08:10 mun.json

$ zq -version
Version: v1.14.0-16-g38763f82

$ zq 'count()' mun.json 
1(uint64)

$ head -5 mun.json 
{
"type": "FeatureCollection",
"name": "mun",
"crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:EPSG::4269" } },
"features": [

The attached video shows my own walk through the repro. A summary of what we see:

  1. zq has no problem reading the file via auto-detect at the shell
  2. When dragged into the app, the "Loading..." messages in the Preview screen never disappear no matter how long we wait
  3. Two zq processes of the head 100 variety seem to stay open indefinitely as long as we remain at the Preview screen
  4. If we go forward with clicking Load, it fails and shows auto-detect failure messages
  5. Even after we've canceled out of the failed load and deleted the pool, a zq process of the zq -i auto * variety seems to stay open indefinitely
  6. I never see any apparent error messages on the DevTools console
Repro.mp4
@philrz
Copy link
Contributor Author

philrz commented Mar 13, 2024

A summary of what's been spawned from this issue:

@philrz philrz linked a pull request Mar 15, 2024 that will close this issue
@philrz
Copy link
Contributor Author

philrz commented Mar 15, 2024

As shown in the attached video, the main symptom of this issue has been addressed by the changes in #3026. The mun.json file can now be loaded into Zui without complaint.

Verify.mp4

This issue will remain open as a reminder for us to look into the zq processes that were left behind when the load was failing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant