How to use the analyse_and_plot_pulses.py python script

[ozel]

Josep posted two questions in #15 (comment), I will answer one here in order to keep the Issues tab clear from regular questions:

On the other hand, I have no idea how to use this python script: https://github.com/ozel/DIY_particle_detector/blob/master/data_analysis_and_reference_measurements/diode_detector/analyse_and_plot_pulses.py
I have downloaded the program annaconda. Then, it represents that I have to copy all the python script adding on it the file name and something else? I have to copy all the script or I have to delet the clarifications, the instructions of the beggining...? And where do I have to copy the script?

[ozel]

You do not have to delete anything in the python scripts, just download it as it is in a folder, then open a command line/terminal and go into this folder and move the measurement file in the same folder (the easiest way).
Inside the python script, you have to scroll down below # SELECT DATASET and if you used the web browser pulse recorder, set file_name = to the name of the .msgp file.
If you used pulse_recorder.py for the recording, then you have to state the .pkl file name a bit lower in df = pd.read_pickle(
You then run the script with python analyse_and_plot_pulses.py on the command line.
Make sure that you have all required python modules installed (some are included in python by befault), they are mentioned in the beginning of the script: import pandas etc.
You can install these modules with conda install pandas etc.
I am in the process of simplfying this python module setup and also the measurement file selection in the analyse_and_plot_pulses.py script, will update it soon to be just a command line parameter.

[josepanglada6]

Thanks Ozel,
How can I install pandas and the other modules?

[josepanglada6]

Hi Ozel,
What I have done is this: I have downloaded the Visual Studio Code and I have installed the python extension. Then I open a folder where I have the analyse_and_plot_pulses.py and the document .msgp of the oscilloscope web. Then I copy the and paste the script in a new file and I go to # SELECT DATASET and set the file name with the document .msgp. But finally, when I press "run python file in terminal", I see in the screen that there are the problems: unable to import pandas, unable to import the numpy, and all the other ones... What can I do? Will be better to use a program different of Visual Studio Code? What do you recommend me?
Lots of thanks!!

[ozel]

Hi Josep,
I am afraid, the source code of this project is not a good start when using python scripts for the first time.
You either need to look up what a "comand line" or "terminal window" is (there is one in vscode, too) or how to install packages with the Anaconda graphical installer if you got one. https://docs.anaconda.com/anaconda/user-guide/tasks/install-packages/

I plan to do an online version of my scripts that will work in the browser, but that will take me a few months more until I have time.
Please have a look at other graphical software that can be used for recording energy spectra completley without my scripts.
It is described in the Wiki: https://github.com/ozel/DIY_particle_detector/wiki/Software#graphical-software

Cheers,
Oliver

[josepanglada6]

Hi Ozel, sorry for writing to you in the wrong place.

When I copy the script analyse_and_plot_pulses.py and go to SELECT DATASHEET, I put where its requiered the name of the document .msgp. But when I run the terminal (execute) i have this error: FileNotFoundError: [Errno 2] No such file or directory: './data/KCL_9-44g_1x3x3cm_touchingdiodecase_pulses_2019-07-08_23-07-18___947___11-47.pkl'
What represents that I have to do? Do I have to download also the documents .pkl?
Thanks,
Josep

[ozel]

no, this is just a reference measurement that you may use to see how the script works.
if you have set the path to a .msgp file correctly to file_name= you have to disable the line that tries to load those .pkl files by putting '#' in the beginning of that line (line nr. 110 at the moment)
As I said, I will improve the loading of files soon and will give you a notification here once it is done.

[josepanglada6]

Ozel,
I have put a # in front of the line and the problem have solves. But now it appears this error:
line 151, in
for i,p in df.iterrows():
NameError: name 'df' is not defined
What do I have to do now?

Thanks,
Josep Anglada

[ozel]

this is not enough information to be sure what happened, but it is posible that your measurement file does not have pulses in it.

[SteveF161]

I found a couple of annoyances in the analysis script. As Oliver says, it doesn't really handle the electron detector. Secondly, there are some bugs in the handling of .msgp files which seem to be related to updates in panda (?? - I think). Finally, it needs a PROGRESS BAR! I've written a revised script which follows the science from the original script as closely as possible but adds some of the missing user interface. This is still an alpha version, but feel free to play with it. (My son and I are still building the alpha detector so I've not tested it on my own data. It works as well as can be expected on the electron data that we collected.)

Runs stand-alone or in Spyder. Stand-alone is best for the plots

@oliver - please add to the repository if you want, but this is still work in progress

Git.zip

[ozel]

Hi Steve,

nice work! I am glad you're having some fun with making your own software.
Personally I think a web version of the software is what's needed for beginners and the core pulse processing logic should be fiddled into a reusable library/python module, it's on my list but will take time. Far more people than I would have expected build the project and try to use the scripts while having never programmed before.

Regarding your problems with .msgp files I would be interested to understand what was happening. On a quick glance your .msgp parsing code looks like mine, minus some sanity checks.
Since you got now energy spectra from the electron detector, do you agree that they are kind of useless because they always look the same?

Cheers,
Oliver

[SteveF161]

Hi Oliver,

I agree that a web utility would be great, but I wanted to stay as close as possible to your original script, and I needed to brush up my Python skills for another project, so I decided to work with Python. I can see that other people were struggling even with simple Python coding so I thought that maybe a self-contained utility was needed.

The problem with the .msgp files was really odd. The following snippet seemed to be the problem:

            for i,row in df.iterrows():
                df_new['pulse'].loc[i] = np.asarray(row['pulse'], dtype='int16')
            df['pulse'] = df_new['pulse'] 
        # javascript's Date() saves timestamp formated in UTC and milliseconds
        df['ts'] = pd.to_datetime(df['ts']*1000000)

The call to np.asarray seemed to always create an empty DataFrame, which meant that the datetime conversion also failed and the data file always seemed to contain no pulses. I couldn't see any obvious problem and the code worked when I broke it down into discrete steps. When I created a version that worked, I stopped trying to fix it further :-)

The electron detector was a great exercise and I now understand it a lot better. It does work but, so far as I can figure out, the sampling rate is maybe too slow for the narrow pulses and the software cannot always detect them reliably. However, it works great as a pulse counter and individual pulses look good on the web oscilloscope. It was also good practice for building the alpha detector.

My son and I finally finished building our alpha detector this afternoon. We've not had much time to work on it for many weeks. Now, the board has passed all connectivity tests. I've powered it up and it has passed the "flick the case" test so everything looks to be working OK. The bad news is that I ordered some uranium glazed pottery shards off Amazon to use as a radiation source but they won't arrive until 14th Jan, so we can't do much with the detector until then.

Steve