save json

[chrishavlin]

Should add a button to the GUI loader to save a yt-napari json so that you can easily re-load the configuration you've typed up in the GUI.

might be able to add an equivalent functionality to the notebook helper functions as well.

[TomasPetro]

Hi I would like to work on this issue but i have problem with installing enviroment.

[chrishavlin]

Hi @TomasPetro , glad you're interested!

I can look in more detail tomorrow, but can you make sure pip is up to date (pip install --upgrade pip). And what version of python are you using?

[TomasPetro]

Thanks for answer. Yes, I checked it yesterday and it was up to date and my python version is 3.9.9 so a bit older. I was looking for requirmens on it in project and did not found any.

[chrishavlin]

The tests run with 3.8-3.10 right now, so 3.9.9 should be fine.

Are you using a virtual environment (conda, pyenev-virtualenv) or system level python? I don't use windows myself, so there may be something funny going on with the editable install with windows that I haven't caught (though the GitHub tests do use windows).

Are you able to install the pypi version (not from the local copy of the source)?

[chrishavlin]

One other quick thing to try: use python -m pip install -e . to make sure you're picking up the right python (if you've got multiple installed and/or are using virtual environments)

[chrishavlin]

OH, actually, is there more to the error message that was cut off in your screenshot? Looks like there might be. that'd be helpful to see.

And, try updating setuptools: python -m pip install --upgrade setuptools (EDIT: I think this is most likely the issue, since we're not currently pinning the min setuptools version in the build environment, and I think we should be, will fix it when I get a chance, but try upgradeing it then building).

[TomasPetro]

I use system level python i think and yes on windows. Do you think that it could run better on Linux?

I tried that second advice python -m pip install -e but still same problem.
Here are rest lines from that error.

I updated setuptools it was bit older 58.1.0 now I have 68.2.2 ...

Do you think that it could run without problems if I just install python on my virtual machine - linux ubuntu ?
Sorry for late messages but it´s weekend and I am currently not at home.

[chrishavlin]

Installing on a virtual machine would be worth a try - I develop on Mac and Ububtu machines, and haven't had this problem. So it'd be good to confirm if it's just happening on windows for you. That said I see no reason why it shouldn't work for you...

Can you send over the contents of pip list?

Sorry this has been so annoying. Hopefully we get it sorted quickly (though I wont be online ievr the weekend, I'll set aside time on Monday to help more with this).

[TomasPetro]

Currently working on Ubuntu. I will answer next day on previous messages.

[TomasPetro]

So, finnaly I installed ubuntu and also that command pip install -e . runned. However I got same error as before on Windows. Solution was to update pip.

When I compared pip on ubuntu and windows, versions were same. So it did not solve issues on windows. Should i compare that pip list and check if there are any differences ?

I´ve one more question. How to run this plugin in developer mode? You know to see changes and so on.

[chrishavlin]

Great! That's a relief.

What version of python are you running on Ubuntu?

And yes, comparing pip list would be useful if you don't mind.

As for development mode -- I don't actually think there is a great way. Because of the way napari loads plugins, you have to restart the python kernel and napari between code changes (I would love to be wrong about this...). So I find it easiest to start napari from a jupyter notebook. That way after you make code changes you can just hit the restart kernel -> run all cells to restart napari. It still takes a bit of extra time but it cuts down on clicking.

And later today I'll send along a few quick notes on where to look in the code for making this change (I'm on my phone at the moment)-- the saving should be very easy as the widget validates entries against the same pydantic model as the json loader, so should be pretty quick to be able to save off a json after that validation step. Loading might take a bit more thought.

[chrishavlin]

Sorry for the delay, @TomasPetro , but here's a quick overview of how I'd recommend that you get started...

if you're new to yt

In the docs I recommend doing a full installation pip install yt[full] -- this, however, will try to install mpi4py, and if you don't have OpenMPI installed you will get an error noting that. MPI is not required for yt-napari. So i'd recommend doing:

pip install "yt[enzo]"

that should get you the dependencies needed to follow the yt-napari notebook examples.

To actually load in data you'll need to download sample data. The example notebooks uses data from the IsolatedGalaxy and enzo_tiny_cosmology datasets, so you'll want to download and unpack those. Here are the direct links from https://yt-project.org/data/ :

• enzo_tiny_cosmology
• IsolatedGalaxy

widget overview

Both the ReaderWidget and TimeSeriesReader are in /src/yt_napari/_widget_reader.py. Both classes have a load_sample load_data method that takes the current values from the widget fields and instantiates a yt_napari._data_model.InputModel (line 142 for ReaderWidget, line 262 for TimeSeriesReader). That instantiated model is what I'd want to save off as a json file.

So I think what you'd need to do is refactor each load_sample load_data method: add a _validate_data_model() function that does all of the field validation and returns the instantiated InputModel and use that within load_data. And then write a new save_json() function that also uses _validate_data_model() and tie that to a new qt widget button so that you can easily get the instantiated and validated pydantic model.

[TomasPetro]

Thanks for info. I plan to work on it today, I hope I won't have to reinstall linux and everything with it again

[chrishavlin]

you could also try using conda with windows -- there's likely some environment issue that I haven't encountered since I don't develop on windows. might be that using an environment manager like conda would help.

[TomasPetro]

I know, I worked in conda before but I also need linux for othes school projects. Can you give me your email addres to send you comparison of pip lists from ubuntu and windows?

[chrishavlin]

I'd rather keep discussion here -- just save them into text files and attach them here (pip list > ubuntu_packages.txt to quickly save to a text file)

[chrishavlin]

you could also join the yt slack (instructions and invite link are at https://yt-project.org/community.html) and send me a direct message with the files.

[TomasPetro]

@chrishavlin I found another error. This time when I starded example from documentation.

I changed the codec to utf-8 and when I did it there were no errors but napari did not start either.

[chrishavlin]

Looks like you're running from a plain python shell within pycharm or vscode? These examples are notebooks, so you'll need to use either ipython or jupyter. I know you can do this from pycharm/vscode but at least to get started you should try from a standard jupyter lab browser environment (sometimes running notebooks within pycharm/vscode can have weird side effects).

So do that first.

And if that doesn't work, check that you can run a plain yt command from the notebook (the error you're getting is coming from yt, not yt-napari or napari):

import yt
ds = yt.load_sample("IsolatedGalaxy")
slc = yt.SlicePlot(ds, 'x', ('gas', 'density'))
slc.show()

[chrishavlin]

also, seems like you should walk through https://napari.org/stable/tutorials/fundamentals/getting_started.html to make sure you have installed napari succesfully

[TomasPetro]

Thanks a lot for info, yes this error ocured in VScode i will try jupiter notebook

[chrishavlin]

@TomasPetro load_sample in my comment (#92 (comment)) was a typo -- I should have written load_data (the widget classes indeed do not have a load_sample method). EDIT: I've updated the comment.

[TomasPetro]

Hi again.
So I've been working a bit on that save json and currently I have a finished save button where it saves to a file all the values I enter into the YT reader (fieldtype, field name left edge values and others). But the problem arises when I want to use load Selection at which I have to edit. Even after watching the tutorials on youtube, the napari doesn't do what it should (and I haven't done anything in load data yet). Do you happen to know why it doesn't want to load my data and where the problem could be then ?

[chrishavlin]

hmm. those just look like warnings to me from QT, don't think that would impact things. Though the last one about the event loop might be an issue...

Some ideas:

threading?

The timeseries loader that you have the screenshot of puts the load into a separate thread, so there might be something going wrong with the QT threading (either due to a bug on the yt-napari side or in napari or an issue with QT?). You can turn off the threading by toggling this line

yt-napari/src/yt_napari/_widget_reader.py

Line 211 in c90261c

_use_threading = True

and then the load will block the QT thread.

You can also try just the base reader: that one does everything within the main QT thread (the one in the screenshot here https://yt-napari.readthedocs.io/en/latest/quickstart.html#the-yt-napari-yt-reader ).

dataset setup

I would think this would raise an error... but how did you download enzo_tiny_cosmology? If you just used yt.load_sample, then I think that gets you just one timestep. Here's what that directory should look like:

chavlin@lism01-chavlin yt_data % cd enzo_tiny_cosmology 
chavlin@lism01-chavlin enzo_tiny_cosmology % ls
32Mpc_32.enzo	DD0007		DD0016		DD0025		DD0034		DD0043		RD0003
32Mpc_32.inits	DD0008		DD0017		DD0026		DD0035		DD0044		RD0004
DD0000		DD0009		DD0018		DD0027		DD0036		DD0045		RD0005
DD0001		DD0010		DD0019		DD0028		DD0037		DD0046		RD0006
DD0002		DD0011		DD0020		DD0029		DD0038		Enzo_Build	RD0007
DD0003		DD0012		DD0021		DD0030		DD0039		OutputLog	RD0008
DD0004		DD0013		DD0022		DD0031		DD0040		RD0000		RD0009
DD0005		DD0014		DD0023		DD0032		DD0041		RD0001
DD0006		DD0015		DD0024		DD0033		DD0042		RD0002

If you've only got one of those DD* folders, you should manually download the full dataset: https://yt-project.org/data/enzo_tiny_cosmology.tar.gz

BUT, I'd be surprised if this was this issue -- if no files were found when you hit Load Selection then it should raise an error? But I don't think I actually have a unit test for that case, so it is possible.

[TomasPetro]

hm it looks as if it is the same ie there is no problem in missing files.

When I open it from another folder it throws me this error again. We've dealt with it before but apparently the problem remains.

It also shows the same error when I use YtReader.

[chrishavlin]

Looks like that error is coming for your filepath: I can reproduce the error with just the following (note, the problematic character is this one: \u010d, https://www.fileformat.info/info/unicode/char/10d/index.htm)

>>> b = u"\u010D"
>>> b.encode('latin-1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'latin-1' codec can't encode character '\u010d' in position 0: ordinal not in range(256)

I couldn't tell you why yt enforces that encoding but there is likely a good reason. Is it possible for you to move the dataset into a different folder to avoid having that character in your path?

[TomasPetro]

Thanks, that solved the problem. But I don't know currently if I understand the task correctly. The Load Selection button loads the selection I specify. I've done a save button which saves that selection to JSON,

But currently I don't understand if the Load Selection has to load also that JSON when currently the button has a completely different functionality - to load what I manually enter there.

[chrishavlin]

But currently I don't understand if the Load Selection has to load also that JSON when currently the button has a completely different functionality

With yt-napari, you can also load from a JSON file by clicking "File -> Open": https://yt-napari.readthedocs.io/en/latest/quickstart.html#loading-a-json-file-from-the-napari-gui

So the initial aim is to simply save off a JSON file that will work when you re-load using "File -> Open". Note that it's not just as simple as creating a JSON from the fields, it has to follow the yt-napari schema. That's why I suggested earlier to:

add a _validate_data_model() function that does all of the field validation and returns the instantiated InputModel and use that within load_data. And then write a new save_json() function that also uses _validate_data_model() and tie that to a new qt widget button so that you can easily get the instantiated and validated pydantic model.

The JSON you have above is missing fields, so it will not be a valid InputModel. It looks like you just pulled the entries for the selections manually so it's missing the dataset fields. For reference, on main, the spot where the reader widgets creates an InputModel instance:

yt-napari/src/yt_napari/_widget_reader.py

Line 142 in c90261c

model = _data_model.InputModel.parse_obj(py_kwargs)

Seems like you've made some progress though! feel free to submit a work-in-progress PR if you want me to look over what you've done so far.