Refactor process() testing & cleaning cube masking/filtering logic

[truth-quark]

Resolves #50 (also related to #14).

This is a fairly complicated PR around testing & refactoring the central process() workflow function in um2netcdf/um2nc.

This work is intended to provide initial testing & cleanup of process(), providing a framework to ensure the workflow is consistent over future refactoring steps.

For background:

• um2nc processes UM files, modifying variables & saving to NetCDF files
• The core um2nc driver is the process() workflow function (it's also the main API entry point)
• Commit 0 in this repo had no unit tests
• process() was initially 140 lines of processing logic & too difficult to unit test
• Recent work extracted blocks of functionality (e.g. variable renaming) to separate functions. This simplified & compressed process() to focus on the workflow logic
• As process() shrank, I identified tricky logic & inefficient data handling (see Clean & optimise masking logic/workflow #50 notes)
• The requirement to retrofit unit testing naturally encourages refactoring to simplify workflow logic

Known Problems:

• Two process() tests require substantial setup & multiple ugly nested mock.patch() calls. This is partly due to complexity in process(), touching separate I/O libraries.
• Both mule and iris fields file objects are complex
  • Multiple attrs are created dynamically when opened
  • Mocking these is tricky with the spec= option, since spec only detects init time attributes
• Some mocks may not represent real object config, however I've tweaked some code to skip some side issues
• process() doesn't return success or failure data for the cubes

Future steps/direction/thoughts:

• For upcoming steps, process() is likely to gain cube modification functionality as it is extracted from cubewrite()
• process() tests should be flexible to expand to cover the additional mod steps
• process() needs reordering to divide into 3 broad steps: input I/O, cube modification, output I/O
  • Idea: sub-steps could be refactored to independent sub-workflow funcs, which suggests process() tests could be split into smaller chunks. This ought to reduce setup dependencies, reducing test setup burden (e.g. less mock.patch() nesting)

For the review, any opinions on the restructure, gaps & correctness checks are welcome! It would be good to reduce the complex setup.

[CodeGat]

Added branch protections to this repo so it'll restrict merging without a review.

[CodeGat]

I don't think I'd be able to comment of the design as it's a bit above my level...but I do have some suggestions

[blimlim]

Changes are looking good to me! The masking logic looks like its doing what it should be, and I think is also easier to read.

From what I can understand, the tests look good too.

I just had a couple of small suggested changes to do with function names and docstrings which I think can now be updated to reflect changes in what the functions are doing.

[blimlim]

All tests passed on Gadi!

[truth-quark]

@blimlim I've made the changes, which definitely make the docs better.

Do you want to go over the fixes quickly & resolve any the sub-discussions you're happy with?

[blimlim]

Nice!

[CodeGat]

Had a re-review of the changes and it looks good. Note: I'm not a python expert so I might have missed some stuff.

[truth-quark]

Had a re-review of the changes and it looks good. Note: I'm not a python expert so I might have missed some stuff.

As it's an evolving project, omissions can be fixed :-). We're in a better position now with testing covering key code blocks.

[truth-quark]

The freak show is merged, with 100+ comments...