One issue when run pyMatchSeries with the latest HyperSpy

[PhysXu]

Hi, thanks very much for this package.

I have an issue when I run the pyMatchSeries with my multi-frame EELS SI dataset. As I applied the the deformation_matrix on specta_data, it reported an error, see below:

I checked the version of my HyperSpy, it is 2.1.1, and the setup.py of pyMatchSeries told me that install_requires=hyperspy>=1.6.1.

So, the higher version of HyperSpy would be the cause of this issue, because the EELS/EDS functions have been moved to the eXSpy package after HyperSpy=2.x.

However, the compiling of match-series (conda install -c conda-forge match-series) will install HyperSpy with the latest version defaultly.

So, I need to compile match-series with the installation of an older HyperSpy, but I do not know how to process it.

Could you give me some suggestions?

Any reply will be appreciated very much!

[PhysXu]

I found a temporary solution:
After conda install -c conda-forge match-series, I run

pip uninstall hyperspy
pip install hyperspy==1.7.6

However, the dimensions of multi-frame SI is a bit strange after run loaded.apply_deformations_to_spectra()
This is a multi-frame data with 128x128 in real space, 10 in frames and 1028 channels in energy axis.
The size of energy axis changed from the initial 1028 to a strange 128.

What would cause this problem?

[PhysXu]

Fixed.

[din14970]

Heya, sorry for the delay. Indeed, since I have not worked on this software for a long time, it's very likely that HyperSpy has since evolved to be no longer compatible. I would need to invest significant time to update the package, and at this moment I don't have it. Feel free to make a pull request if you feel up to it. Indeed installing an older version of HyperSpy is the way to go at the moment. I'm glad you found a temporary solution to your problem.

[PhysXu]

Hi, @din14970 , thanks very much for your reply.

My further idea is to do the non-rigid registration for multi-frame 4D-STEM data by using pyMatchSeries.

I noticed that pyMatchSeries is support the dask dtype. So it may be a better way to load my data by dask.

However, I am a newcomer for coding. So there is a stupid question:
Need I change something in the matchseries.py file?
and how can I install/compile it after some changing?

[PhysXu]

Hi, @din14970 , I have another question:

How to avoid the error of ERROR: logBaseTwo: invalid parameter: 0

I run the codes with a dataset with the dimensions of 10 frame and 128 x 128 in real space, it is OK.

However, when I run another data with 5 frames and 128 x 128, the error above appeared.

This is the calculation.configuration which I used:

I suppose the requirement of the power of 2 is only for the real space dimensions, so I do not understand why the error appeared for 5 x 128 x 128.

[din14970]

Most of pyMatchSeries is currently simply a python wrapper over the match-series binary. The most interesting thing I do is indeed allow you to apply the calculated deformations to datasets, mainly STEM-EDX and STEM-EELS. Applying the deformations to 4D-STEM datasets may be very tricky as you will likely have to somehow blend diffraction patterns in different positions. This is currently not implemented in pyMatchSeries and you will be on your own.

To modify pyMatchSeries, you best create a fork first. Then you clone the project, set up a virtual environment, then install pyMatchSeries in there in editable mode (pip install -e .). To explain this in detail here would take me too far, have a look here for example: https://stackoverflow.com/questions/52248505/how-to-locally-develop-a-python-package. You can use all kinds of tools for creating your virtual environment, I would recommend conda. You can then modify the code and run it as you wish. Python does not need to compile, it is an interpreted language.

Indeed the power of 2 requirement only applies to the size of the real space images, not the number of images. Could it be you did not update the calculation.configuration["numTemplates"] line to equal 5? If all the other parameters are the same, I don't see why one would work and the other not.

[PhysXu]

Thanks, after updating the calculation.configuration["numTemplates"], this error disappeared.

I have a small question about this numTemplates param. :
I see, in the default_parameters.param file, it says that # maximum number of images to consider

Thus, this param. should be equal to the frame-number of my data, right?
However, in the example jupyter file, the data has 163 frames while the numTemplates is set as 50.
So I get a bit confused.

[PhysXu]

For applying the deformations to 4D-STEM datasets, I tried an idea today as follows:

for example, I have a dataset of 10 (in frames), 128x128 (in real space), 64x64 (in the diffraction space). I reshaped this dataset in the diffraction space to 4096 (=64x64), so that I got a new dataset (10x128x128x4096) with the similar structure to STEM-EDX-SI and the 4096 channels can be treated as the energy axis, which we usually do in your codes.

After applying the deformations, I reshaped it back to 10x128x128x64x64 and sum up the frame-axis. Here, I used the virtual_adf_images for calculating the deformations.

I am still working on this, and maybe will give some feedback later.

[PhysXu]

For the part of modifying pyMatchSeries, I have more things to discuss:

You mean that I can modify pyMatchSeries in the disk where I installed it?

I used conda to install pyMatchSeries by setting up a virtual environment of pyMatchSeries, like:

conda create -n pyMatchSeries python=3.10
conda activate pyMatchSeries
pip install --user pyMatchSeries
conda install -c conda-forge match-series
pip uninstall hyperspy
pip install hyperspy==1.7.6

I can see HyperSpy in the folder of ...\pyMatchSeries\Lib\site-packages, but I can not find something about pyMatchSeries or match-series.

So, where can I find the files which I want to modify?

Hope these questions will not bother you @din14970 too much and thanks again for your reply with such detailed information.

[PhysXu]

Hi, @din14970 , I got the method to modify Python packages locally by following your suggestions.

I am reading your codes line-by-line and trying my best to understand them.

Further progress and feedback will be post here.