qc environment setup

[apcraig]

@phil-blain had some problems setting up a working python+ environment for the qc test. We would like to understand that better and try to make things a bit more robust.

[phil-blain]

See the discussion on #325 for background.

I just created a conda environment with the same exact versions and builds as @mattdturner listed in #325 (comment) (System 2) :

conda create -n cice-qc -c defaults basemap=1.2.0=py27h705c2d8_0 libnetcdf=4.6.1=h11d0813_2 matplotlib=2.2.3=py27hb69df0a_0 netcdf4=1.4.2=py27h808af73_0 numpy=1.15.4=py27h1d66e8a_0 numpy-base=1.15.4=py27h81de0dd_0 python=2.7.16=h9bab390_0

I still have the error (#325 (comment)) This leads me to think that maybe something is wrong with my outputs, and not with the script. I have uploaded the output from the qc test to a tar archive here : https://1drv.ms/u/s!AqQgQ3VdlaGUgi7oi_SkB29or4xF?e=iyxarn : if someone can download that (5 GB) and run the QC test on it we could verify that hypothesis.

[eclare108213]

has anyone had a chance to look at @phil-blain 's output?

[mattdturner]

Not yet. I should be able to look at it tomorrow, though.

[mattdturner]

There was an issue in the QC script that wasn't encountered with the data that had been previously used for testing, but was encountered with @phil-blain 's output. If I'm remembering the QC process correctly, it looks like the script is taking each data point and looking it up in a critical T-value table. The problem arose when 1 of the data points fell exactly in the center between 2 T-values. The script uses
idx = np.where(np.abs(df[x]-df_table) == min_val)
to determine the index in the T-value table. Under normal circumstances, this idx variable will only have 1 value. However, using @phil-blain 's data there was an instance where idx had 2 values in it, causing the crash.

I have a working fix for this bug, and will try to create a pull request before the end of the day. When I tested the data provided by @phil-blain with the fixed QC script, the QC test passes:

INFO:__main__:Running QC test on the following directories:
INFO:__main__:  /p/work1/turner/tmp/brooks_intel_smoke_gx1_44x1_medium_qc.qc_base_20190626/history/
INFO:__main__:  /p/work1/turner/tmp/brooks_intel_smoke_gx1_44x1_medium_qc.qc_test_20190626/history/
INFO:__main__:Number of files: 1825
INFO:__main__:2 Stage Test Passed
INFO:__main__:Quadratic Skill Test Passed for Northern Hemisphere
INFO:__main__:Quadratic Skill Test Passed for Southern Hemisphere
INFO:__main__:
INFO:__main__:Quality Control Test PASSED

[mattdturner]

I'm going to have to hold off on the pull request until next week. I'm trying to reduce the amount of memory required by the script. Currently, I have a virtual memory limit of 8GB on one of the machines that I am using. I am unable to run the script on that machine. I have been successful running it on other machines with a larger virtual memory limit.

[phil-blain]

I've tried the new script with the input I uploaded here, and the test passes (same result as @mattdturner above.)
However, Amélie @abouchat tried the new script on her outputs for #331 and she still has the error :

./configuration/scripts/tests/QC/cice.t-test.py /fs/site2/dev/eccc/mrd/rpnenv/amb001/runs/brooks_intel_smoke_gx1_44x1_alt02_medium_qc.qc_base_original_alt02 /fs/site2/dev/eccc/mrd/rpnenv/amb001/runs/brooks_intel_smoke_gx1_44x1_alt02_medium_qc.qc_test_alt02
INFO:__main__:Running QC test on the following directories:
INFO:__main__:  /fs/site2/dev/eccc/mrd/rpnenv/amb001/runs/brooks_intel_smoke_gx1_44x1_alt02_medium_qc.qc_base_original_alt02
INFO:__main__:  /fs/site2/dev/eccc/mrd/rpnenv/amb001/runs/brooks_intel_smoke_gx1_44x1_alt02_medium_qc.qc_test_alt02
INFO:__main__:Number of files: 1825
Exception RuntimeError: 'generator ignored GeneratorExit' in <generator object maenumerate at 0x2acfe3e65690> ignored
Traceback (most recent call last):
  File "./configuration/scripts/tests/QC/cice.t-test.py", line 564, in <module>
    main()
  File "./configuration/scripts/tests/QC/cice.t-test.py", line 516, in main
    PASSED, H1_array = two_stage_test(data_base, nfiles, data_diff, files_base[0], dir_a)
  File "./configuration/scripts/tests/QC/cice.t-test.py", line 241, in two_stage_test
    t_crit[x] = t_crit_table[idx]
  File "/fs/home/fs1/ords/cmdd/cmde/phb001/logiciels/miniconda3/envs/cice-qc-builds2/lib/python2.7/site-packages/numpy/ma/core.py", line 3330, in __setitem__
    _data[indx] = dval
ValueError: setting an array element with a sequence.

Amélie's outputs are here :
base : https://1drv.ms/u/s!AqQgQ3VdlaGUgi97LkuvOd0y_oEi?e=FhDFOi
test : https://1drv.ms/u/s!AqQgQ3VdlaGUgjB2eboFCfLr6oVz?e=Ep9NAa

[mattdturner]

I'll look into this new case where the script crashes within the next day or 2.

[abouchat]

After update #346 , the QC test now also runs on my output!
And I get the same results as Matt:

INFO:__main__:Running QC test on the following directories:
INFO:__main__:  ../data/eccc-ppp2/runs/brooks_intel_smoke_gx1_44x1_alt02_medium_qc.qc_base_original_alt02
INFO:__main__:  ../data/eccc-ppp2/runs/brooks_intel_smoke_gx1_44x1_alt02_medium_qc.qc_test_alt02
INFO:__main__:Number of files: 1825
INFO:__main__:2 Stage Test Passed
INFO:__main__:Quadratic Skill Test Failed for Northern Hemisphere
INFO:__main__:Quadratic Skill Test Failed for Southern Hemisphere
INFO:__main__:
ERROR:__main__:Quality Control Test FAILED

DEBUG:__main__:Northern Hemisphere skill score = 0.979277
DEBUG:__main__:Southern Hemisphere skill score = 0.912965

[phil-blain]

This issue can now be closed