Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gadget interpolation speeds #3

Open
chrishavlin opened this issue Jan 27, 2022 · 2 comments
Open

Gadget interpolation speeds #3

chrishavlin opened this issue Jan 27, 2022 · 2 comments

Comments

@chrishavlin
Copy link
Collaborator

chrishavlin commented Jan 27, 2022

In playing around with the current json loader and datasets beyond IsolatedGalaxy, I've found that large sph particle datasets are pretty slow. e.g., the following json:

{"$schema": "https://raw.githubusercontent.com/data-exp-lab/yt-napari/main/src/yt_napari/schemas/yt-napari_0.0.1.json",
  "dataset": "snapshot_033",
  "field_type": "PartType0",
  "field_name": "density",
  "left_edge": [0.0, 0.0, 0.0],
  "right_edge": [25.0, 25.0, 25.0],
  "edge_units": "Mpc",
  "resolution": [600, 600, 600],
  "take_log": true}

takes around 2.5 minutes to load in napari. Most of this time is spent in yt, interpolating the SPH field. While alot of the slowness here may be unavoidable, there are at least three points that are worth looking at I think:

  1. Is there a way to add some napari notification that the interpolation is in progress? The terminal log shows the usual yt notifications, but the napari GUI hangs while it waits for yt to return the interpolated data. Some sort of loading symbol would be nice...
  2. There seems to be a significant differences in loading time between the yt code when executed in napari vs. executing the same yt code in a python shell: executing in napari was almost a whole minute slower. Need to do some more careful profiling to compare.
  3. It might be worth considering a cache system -- the interpolated FRB would be fairly cheap to store on disk so that if you subsequently load an identical json it could reload the previously interpolated FRB. Could have an optional "use_cache" argument in json model. Not an immediate item to do, but may be worth considering in the future.
@chrishavlin
Copy link
Collaborator Author

For reference, the terminal log while loading:

Screenshot from 2022-01-27 09-26-10

@chrishavlin chrishavlin mentioned this issue Feb 7, 2022
8 tasks
@chrishavlin
Copy link
Collaborator Author

with the jupyter interaction from #22, I think this is less of an immediate issue, will remove from #7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant