IMOS Toolbox v2.6.16 - 25 September 2023 - PRE-RELEASE

This is a pre-release only and could possibly include breaking changes.

The main change included is:

• adding new *_failed_test variable on QC'd variables to track which QC test(s) failed

cfff03e release:2.6.16
e7c9356 (Fix) Failed tests index deletion badly made
5192361 improve comment attribute; merge manualQc and historicalQc together
c43afae replace variable flag_tests with failed_test string
aa7be9f Add scenario for imosErrorVelocitySetQC
4777cd6 (Feat) imosQCTests function to read imos flags from config

Current bugs currently under investigation:

• seems to only happen when running the toolbox on a single file an not with the DDB. Let's say that on a single file, we do some manual QC by selecting the start of the data on the plot, and flag it as bad. It will generated a *.pqc file alongside the original file. Now if you re-rerun the QC, the code should delete the failed_test variable so that values don't get sum up with previous run. However the variable doesn't get deleted.

IMOS Toolbox v2.6.15 - 12 January 2023

The main changes included are:

• fixing the bottom tracking sub-parser of Signature files
• OceanContour parser to include bin mapping support

7ea6f15 (origin/FixBottomTracking) Fix(readAD2CPBinary): various issues with Bottom tracking parser
cd9c771 (origin/ocean_contour_updates) fix(Parser): handle non binmapped oceancontour NetCDFs and add OceanContour unittests
45e84a9 feat(Parser): add import option for binmapping
e17d935 fix(Util): Improve handling adcp dimensions
3c6d1f8 feat(IMOS): allow multiple attributes args
15a2616 Add instrument info to spikeWindow GUI. This nessicitated copying the meta sruct which contains make/model info. Use genSampleDataDesc to create instrument string.

IMOS Toolbox v2.6.14 - 10 May 2022

The main changes included are:

• a usability feature allowing to resize the depthPP window
• down-facing instrument bin mapping for 4-beams signature RDI. Previous code was only correct for up-facing ADCPs. Data reprocessing will be required

5512740 release: 2.6.14
4a14925 feat(tests): refactor and add test for adcpBinMapping
c81c93f fix(docs): remove old link and add link to wiki
351a41b fix(Parser): allowing down-facing instrument bin mapping for RDI 4 beams
534df35 [3901] Unify license file with md extension
61ca6f6 [3901] Add license info to LICENSE.md
5b0cc3f feat(graph): update ADCP thresholds plots
c9ee603 fix(Parser): Error velocity QC should be & not |
06fb1da fix(Parser): Replace references of old facility to DWM
0129086 fix(UI): Allow user to resize depthPP window
c89d69d fix(build): --matlab_path argument parsing

hotfix - IMOS Toolbox v2.6.13 - 11 June 2021

Release 2.6.13 is a hot-fix release.

The release contains three important bug fixes( #741,#742,#743), and releases 2.6.11 and 2.6.12 are now marked as do not use.

This hotfix corrects the workhorseParse function in two segments: 1. the wrong assignment of workhorse velocity components (#741) and 2. fix magnetic declination variable attributes assignment for wave data (#743). The release also fix the usability of the two new QC tests (imosEchoIntensitySetQC,imosSurfaceDetectionByDepthSetQC).

Finally, updates to the build script functionality, as well as test coverage fixes and docstrings are also included.

Bugfixes:

d1d760f fix(Parser): wave magnetic variable renaming
67e52fb fix(QC): EchoIntensitySet SurfaceDetectionByDepth
3806980 fix(Parser): workhorseParse U and V swap bug

Documentation/tests updates:
fbb58e6 fix(tests): fix docstrings for windows
ee2890d feat(tests): batchTesting extensions
bde2448 feat(tests): filtering interactive tests
13e094b feat(tests): filtering and rm randc function
283ed0e fix(tests): remove verbose print in docstring
9fde469 fix(tests): point to xunit test reference
4aaaddb fix(tests): conform to docstring format
29af5ed feat(Util): report test status in checkDocstrings
1ae8107 fix(Schema): isequal_ctype docstring
2b080c4 fix(tests): point to xunit test as reference
7ffda67 fix(tests): comment docstring to avoid double call
0eda60b fix(tests): wrong test call and missing case
ffbbf69 fix(tests): update missing docstrings tests
9ef5d56 fix(tests): fix docstrings
26ca6e5 fix(tests): imosEchoIntensitySetQC tmp file path
7d31b8f fix(tests): class name in testworkhorseParse
6777f65 fix(tests): fix wrong logic
1cfccd1 fix(tests): avoid picking tmp files as raw files
dd14010 fix(tests): miscelaneous docstring fixes

Build:
72db8ec feat(build): automatic detection of arch
8ff98cd feat(build): build improvements

Misc:
b13b8dc refact(PP): export find_beam_vars as function
13799b4 fix(Util): add missing close file id
7e934c8 fix(Util): improve error message with types
f093ea6 fix: binary updates for hotfix release
6785f31 Add Windows Binary
8da5816 fix: hot-fix release version bump
839dca2 fix(Util): comments -> comment typo
cd96774 fix(PP): msg adcpWorkhorseVelocityBeam2EnuPP
ad54242 fix(linting): testworkhorseParse
0274484 fix(linting): testadcpWorkhorseBeam2EarthPP
5ae1694 fix(linting): test_imosEchoIntensitySetQC.m
b8972c4 fix(test): gitignore, tmp folder for runtime tests

hotfix - IMOS Toolbox v2.6.12 - 28 April 2021

DO NOT USE THIS RELEASE - see release v.2.6.13 notes.

Release 2.6.12 is a hot-fix release.

The release contains a fix for adcpBinMappingPP. In release 2.6.11 we inadvertently removed the DIST_ALONG_BEAMS coordinate (see #737 for details). The bug created a non-compliant NetCDF file with missing coordinates, triggering a failed ingestion at the cc-imos-checker step in the AODN ANMN pipeline.

Affected files are all workhorse ADCP files processed with 2.6.11.

Commits in this release:

9b3bbff fix: hot-fix release version bump
ec119b5 fix: remove tabs in testadcpBinMappingPP
d612070 feat: hot-fix release binaries
403b3a0 fix: new dim removal testcase adcpBinMappingPP
638dd86 fix: indentation in testadcpBinMappingPP
9aeb159 fix: wrong removal of DIST_ALONG_BEAMS dimension

IMOS Toolbox v2.6.11 - 20 April 2021

DO NOT USE THIS RELEASE - see notes in v2.6.13.

Release 2.6.11 is the Q2 & Q3 release of the IMOS Toolbox.

The release contains an overhauled handling of RDI workhorse ADCP data, updates to ADCP quality control, new magnetic declination tables, some bug fixes, and internal improvements.

---

Changes to RDI data ingestion:

The RDI workhorse parser is now using more information included in the binary file and with a reduced peak memory usage. We also added support for Beam coordinates to ENU coordinates in the parser and through a new PreProcessing step. See the wiki page for more details and the +Teledyne and +Workhorse packages for some of the related changes.

071783b feat(PP): RDI TeledyneADCP rotation conversions
eca2e8c feat(PP): RDI support and refact BinMappingPP
b2db280 feat(parser): Workhorse parse overhaul
495c601 review(Parser): fix typo in help
5f67b98 feat(rename): rename workhorseParse testfile
aeb6523 fix(adcpBinMappingPP): allow downward adcp
2543330 fix:(adcpWorkhorseVelocityBeam2EnuPP): comments
dd47547 fix(readWorkhorseEnsembles): compatibility revert
7a9a1fd fix(readWorkhorseEnsembles): remove typo
41de5f8 fix(readWorkhorseEnsembles): turn on memory saving

---

New optional ADCP QC tests:

1. imosSurfaceDetectionByDepthSetQC - for simple surface detection in both upward/downward cases.
   02658c7 feat(QC): adcp surface detection by Depth

2. imosEchoIntensitySetQC - for customisable behaviour in echo intensity markings. The test may be configured to be equivalent to the fish detection algorithm.
   9bafc2a feat(QC): a customisable echoIntensity QC test

In particular, imosEchoIntensitySetQC intend to deprecate the various attempts of Echo Intensity filtering scattered in the codebase and provide an easy way to customise its behaviour in the User Interface. Both tests are optional and in the evaluation phase. Other tests are likely to be deprecated in the future.

---

Behaviour Changes:

1. imosErrorVelocitySetQC now uses deviations from the mean to classify ADCP error velocity estimates.
   7aa8513 BREAKING CHANGES(QC): ErrorVelocity adcp markings

2. imosEchoIntensityVelocitySetQC now inspect fewer bins and support downward-looking ADCPs.
   ff061aa BREAKING CHANGES(qc): EchoIntensityVelocity

---

Miscellaneous changes:

1. Draft support for [adcpThreshold] (https://github.com/aodn/imos-toolbox/blob/master/Graph/adcpThresholds.m) graphics.
   4f29ab8 feat(graph): improve timeSeries default axis plots
   c29c97c feat(graph): adcp Threshold visualisation diags
2. Draft support for RDI bin averaging.
   4cd1e31 feat(processing): rdi adcp bin averaging
3. Faster adcpBinMappingPP.
   eca2e8c feat(PP): RDI support and refact BinMappingPP
4. Update magnetic parameters/Geomag to IGRF13 to support data collected beyond 2020+.
   e7c7477 fix: geomag for 2020+ with new IGRF13 tables
5. Restore the ability to run the VerticalSpikeQC test.
   4a6f70b fix(QC): fix call signature matching
6. Allow overwriting DEPTH estimates in depthPP.
   15d5eb9 fix(PP): allow overwriting of depth variable

---

New internal features:

97d1146 feat(IMOS): schema evaluation for PP/QC
fc362f0 feat(IMOS): metadata extraction queries
d994c7f feat(IMOS): adcp utilities
cccffc8 feat(IMOS): detect variables with given dimensions
bad34c6 feat(Util): disp that shows where the call came
28baf98 feat(Util): allow update config files
5e28104 feat(genSampleDataDesc): support missing fields
14a0030 feat(IMOS): New IMOS utility functions
6a4fce9 feat(Util): mem usage and double/cell comparisons
a19ef3b feat(Util): allow file extension filtering
7428351 feat(Util): only persist values in deployed case
a27c18c feat(IMOS): upgrades to the IMOS package
6866b9d feat(Util): improvements to errormsg/swarning
9efcff9 feat(test): update sites/instruments tables
a80159a feat(test): adcp single ping test
245d526 feat(qc): include EchoRangeSetQC tests

---

Other miscelaneous fixes and cleanups:

c3f1e61 hotfix: include compatible binaries
b531528 feat(Util): remove wrapTo360 dependency
990e09e fix(build): rm test files from binaries
b00323a fix(QC): support skip of invalid datasets
ddb6841 fix(QC): fix warning msgs
f5c6102 fix(Util): simplify genSampleDataDesc for tests
7c26add fix(imosEchoRangeSetQC): enu detection & clean
21bb738 fix: rm surfacedetectionsetQC old code
4f11f09 fix(genSampleDataDesc): remove license body
0ff821e fix(readMappings): close file id before error
d78fc69 fix(errormsg): put line error in single line
10de06b fix(Util): isinside support to diff. cells lens
b2f8708 fix(Util): only move files if their name differs
f49a895 cleanup: remove geomag tmp files
d967874 cleanup: remove rdiBeam2Earth.m deadcode
a6f948d fix(IMOS): add missing cf standard name
94e3f26 fix: avoid loading non-binary files in tests
3a60b58 fix(Util): missing semicolon in readMappings.
8217ace fix(Util): workaround to failing docstring tests
9fb7e36 fix(TestUtils): file report in checkDocstrings

We acknowledge some early code efforts provided by CSIRO and others in this release.

IMOS Toolbox v2.6.10 - 27 Nov Oct 2020

Release 2.6.10 is the Q1/2020 release of the IMOS Toolbox.

This release contains a new parser (OceanContour), a more flexible way to register manual QC comments in the UI popup, new options to display instrument names in the UI (instrument aliasing and information detail level), new internal features, and small bugfixes.

Breaking Changes:

• This release may require conflict resolution within the toolboxProperties.txt given the new aliases and UI instrument name display changes.

Major Changes:

df7f53a feat(Parser): OceanContour netcdf & mat parser
6abcccd feat(QC): Predefined ManualQC comments
319a2d7 feat(UI): detailLevel and aliases for instruments

New Internal Features:
9830892 feat(StructUtils): filterfields function
80a2d1f feat(netcdf): nc_get_var & nc_flat metadata lookup
2fe17f7 feat(IMOS): IMOS package utilities
a57cdb3 feat(Util): allrepeats, inclusive.
52cfa2d feat(Util): remove singleton dims from size arrays
22a9eef feat(Util): getindex & wrappers for type validation
6ceac6c feat(Util): squashCells and col/row vec creation
0ef3e44 feat(Util): errormsg function
78f15cb feat(utility): random generation wrappers
723fa99 feat(utility): tests as docstrings
c3c903d feat(docstrings): fix docstrings test evaluations
25e10d3 feat(docstrings): fix docstring template
66cd805 feat(docstrings): eval exceptions on docstring
b6c449b feat(docstrings): reduce verbosity on docstring evaluation

Fixes, Bugs, and Refactorings:
60d3cc5 fix(UI): disable ax interactions when using rbbox
6874436 fix(graph): fallback for missing the image toolbox
19fcaa8 fix(build): toolbox package folders & cleanup
1959c8b feat(bugfix): test_docstring/check_docstrings
f019179 feat(bugfix): fix output size in struct2parameters
1e73dea feat(bugfix): missing CommentStyle in readMappings
a219777 feat(bugfix): optional argname in readUntilMatch
a07372f feat(bugfix): return more than one fieldname
982e100 feat(refactor): simplify checkDocstrings ok check
26a3a89 feat(refact): fix docstrings, rm plus dirs path
b5399d8 feat(refact): multi-fixes in isunique

Changes triggered by review:
96ae07c feat(review): rename funcs with snake_case names
9ef5736 feat(review): specific nargin in randomBetween

IMOS Toolbox v2.6.9 - hotfix release - 25 Aug 2020

Release 2.6.9 is a hotfix release.

This release fixes a regression introduced in 2.6.8 that blocks users from using the spikeQC functionality in non-burst data. More tests are now available, as well as updated tests for netcdfParser to match the recent changes.

Changes included in this release:

Fix spikeQC regression for non-Burst data 7294f73

Update/Refactoring netcdfParser tests 518a269

IMOS Toolbox v2.6.8 - hotfix release - 13 Aug 2020

Release 2.6.8 is a hotfix release.

This release contains important fixes:

1. Fix regression when handling ADCP tilt data in imosTiltVelocitySetQC introduced in 2.6.7. 9a0d83c
2. Avoid listing/processing of variables with more than one dimension within the imosTimeSeriesSpikeQC test/UI. eb1b86c
3. Allow burst/non-burst datasets imported with the netcdfParser to be Quality controlled with the imosTimeSeriesSpikeQC method. 307fb3b f
4. Fix typos and bugs regarding auto mode in imosTimeSeriesSpikeQC. 2acf9d5
5. Some speed improvements for non-standalone execution, particularly at startup. 6bd84d1

IMOS Toolbox v2.6.7 - 15 Jul 2020

Release 2.6.7 is the Q4/2019-20 release of the IMOS Toolbox.

This release contains a new IMOS acknowledgement, the second part of the SpikeQC functionality (burst spikeQC tests), a SpikeQC preview window for parameter exploration, improved selection of SpikeQC tests in a per variable basis, new table values for aquadopp current meter tilts (TiltQC), Improvements to the netcdfParser and SBEparsers (new variables), and better interaction in the DEPTH vs nominal depth
checker.

Two new SpikeQC tests for Burst data are included:

• BurstHampel - A Hampel classifier applied to the burst data. The filter can be configured to run over individual bursts of N-consecutive bursts.

• BurstRunningStats - A general statistical filter for burst data. This builds upon the BurstQC stats branch and allows the operator to construct different combinations of statistical filters to apply in every burst, by allowing the selection of scale/dispersion functions. The filter can easily mimic the one implemented in BurstQC stats by selecting the scale function to be nanmean, dispersion function to be nanstd, and a simple scale to be applied to the dispersion. Other configurations are also allowed (e.g. median/mad).

See the wiki for more details.

Major changes since v.2.6.6:

• Update acknowledgement 91dc376
• add missing global attributes imports in the netcdfParse 03ed354
• refactoring and new variables for SBE parsers 5bf562d,
  0024058, 18cd1d5,
• new QC values for tilts in aquadopp current meters 6ade4aa
• new popup window to confirm QC tilts 6ade4aa
• new utility functions to handle burst data 71416b1
• new functions for spikeQC in burst data 66d086f
• new handling of spikeQC tests and variable selection 0dccfb6
• bugfix for the DEPTH vs Nominal DEPTH interaction 517628a
• Version bump and binary updates 24e620a