Skip to content

Commit

Permalink
Merge remote-tracking branch 'origin/develop'
Browse files Browse the repository at this point in the history 
Merging all changes from the develop branch to master branch in
preparation for the METplus Gamma Release which will  be cut from
the master branch.

This commit has been run through and passed all working unit and integration
tests.
  • Loading branch information
James Frimel committed May 4, 2018
2 parents 74645b0 + 97eec97 commit 860e997
Show file tree
Hide file tree
Showing 171 changed files with 29,732 additions and 3,418 deletions.
14 changes: 14 additions & 0 deletions .travis.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@

language: python
python:
- "2.7"

# Build only the 'develop' branch
# safelist
branches:
only:
- develop

script:
- pytest ./METplus/internal_tests/pytests/produtil/regress_test_produtil.py ./METplus/internal_tests/pytests/produtil/produtil_test.conf

201 changes: 48 additions & 153 deletions doc/README_install.md
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
Model Evaluation Tools Plus (METplus) (May 2017) {#METplus_install_guide}
Model Evaluation Tools Plus (METplus) {#METplus_install_guide}
======================================================

Welcome to the documentation for METplus. METplus is a set of Python wrapper scripts around the MET verification tools
(and eventually METViewer, a tool used for plotting MET output verification statistics).
and METViewer (a tool used for plotting MET output verification statistics).


Background and Future
Expand Down Expand Up @@ -37,27 +37,17 @@ METplus was developed using Python version 2.7.9. Python version 2.7 or greater
Version Control
---------------

METplus uses GIT for version control in a private NCAR repository.
METplus uses GIT for version control in a public GitHub repository:
https://github.com/NCAR/METplus.


Getting the Code and Test Data
------------------------------

Get the METplus package by running:
Get the METplus latest release from the NCAR/METplus GitHub repository, and
click on the 'releases' link to download the latest release, documentation,
and sample data.

wget http://www.dtcenter.org/met/users/downloads/METplus/METplus_vX.2017XXXX.tar

Get the METplus test data by running:

wget http://www.dtcenter.org/met/users/downloads/METplus//METplus_test_data.20170109.tar.gz

Decide where you would like to put the code and copy the METplus Package to that location. Unpack the gzipped tar file in that directory by running:

tar -xf METplus_vX.2017XXXX.tar

Decide where you would like to put the test data and copy the METplus test data to that location. Unpack the gzipped tar file in that directory by running:

tar -zxf METplus_test_data.20170109.tar.gz


Configuring the Environment
Expand All @@ -69,184 +59,88 @@ Open up your .cshrc (or similar file) using the editor of your choice and add th

To your PYTHONPATH, add:

(full path to METplus/ush):${PYTHONPATH} (replacing the text and () with the full path)
(full path to METplus/ush):${PYTHONPATH} (replacing the text and () with the full path)

If you do not currently have a PYTHONPATH, add:

setenv PYTHONPATH (full path to METplus/ush) (replacing the text and () with the full path)
setenv PYTHONPATH (full path to METplus/ush) (replacing the text and () with the full path)

To your PATH (path), add:
setenv PATH ${PATH}:(full path to METplus/ush) (replacing the text and () with the full path)

setenv PATH ${PATH}:(full path to METplus/ush) (replacing the text and () with the full path)

Optional: Add the METplus job log file, JLOGFILE

setenv JLOGFILE (full path/filename) (replacing the text in () with the desired full
setenv JLOGFILE (full path/filename) (replacing the text in () with the desired full
path and filename of your job log file.

Save the changes and source your .cshrc (or similar file) by running, for example:

source ~/.cshrc
source ~/.cshrc


Configuration Files
-------------------

There are two sets of configuration files - one for running METplus and one for running MET.
The MET configuration files are found in the parm/met_config directory.

- METplus

The main configuration file for METplus is metplus.conf, which is located in the "parm" subdirectory.

Users have the ability to override specific fields in metplus.conf in their own config file.
As a starting point, an example file with some typical fields to override is user.template.conf
There are three main configuration files for METplus:
+ METplus_data.conf
+ METplus_system.conf
+ METplus_runtime.conf

These are located in the parm/metplus_config subdirectory.

- MET

The configuration files for the MET tools are also located in the "parm" subdirectory. Currently, the applicable configuration files are
TCPairsETCConfig (for the extra tropical cyclone TCPairs run) and the SeriesAnalysisConfig_by_init and SeriesAnalysisConfig_by_lead.
Users have the ability to override specific fields in metplus_data.conf, metplus_system.conf, and/or
metplus_runtime.conf via use case configuration files and/or their own config file.
The three main METplus configuration files do not need to be specified on the
command line when running METplus. Any additional configuration file


Configuration Setup
-------------------

The user should look at metplus.conf to modify necessary and desired fields. The information below will cover the various variables:

NON-MET EXECUTABLES

Some of these fields may need to be modified based on the location of the executables on your system, but some may be standard. Note that
the WGRIB2 executable is not currently in a standard location and will need to be modified.


COMMONLY USED BASE VARIABLES

MET_BUILD_BASE is the base location for the MET release that you will be using. Please set that to an appropriate location.

OUTPUT_BASE is the base area for where the user would like to store their output data.

PARM_BASE is the parm subdirectory for the METplus configuration files.


MET EXECUTABLES

These fields rely on MET_BUILD_BASE and its "bin" subdirectory.


INPUT DATA DIRECTORIES

These fields indicate where your input data is located.

For example, the METplus_test_data.20170109.tar.gz, includes a "reduced_model_data" directory, which contains GFS data, and a "track_data"
directory, which contains extra tropical cyclone track data. If you wanted to put this data at "/d1/data/SBU", you would set the following:

PROJ_DIR = /d1/data/SBU
MODEL_DATA_DIR = GFS_DIR = {PROJ_DIR}/reduced_model_data
TRACK_DATA_DIR = {PROJ_DIR}/track_data


OUTPUT DIRECTORIES

These fields include a log directory and a tmp directory along with other output directories.
The TRACK_DATA_SUBDIR_MOD refers to the subdirectory where the track data will be written,
reformatted to be in true ATCF format, which the MET tools need for processing.


FILENAME TEMPLATES

These fields contain templates for filenames and filename prefixes and regular expressions.


CONFIGURATION FILES
The user should look at metplus_data.conf, metplus_system.conf, and metplus_runtime.conf
to modify necessary and desired fields.
Please refer to @ref confguide

These fields indicate which configuration files to use for MET.

How to Run?
-----------

LISTS AND SETTINGS

PROCESS_LIST is the list of processes that the user wants the master script to run.
For example, a full run from start to finish for running series analysis by lead, would be:

PROCESS_LIST = run_tc_pairs.py, extract_tiles.py, series_by_lead.py

STAT_LIST is the list of statistics to be computed (e.g. STAT_LIST = TOTAL, FBAR, OBAR ).
NOTE: Currently, "TOTAL" is a REQUIRED cnt statistic used by the series analysis scripts, so it must be in the STAT_LIST.

INIT_DATE_BEG is the beginning date in the format YYYYMMDD (e.g. 20141201 ) for the initialization time.
INIT_DATE_END is the ending date in the format YYYYMMDD (e.g. 20150331 ) for the initialization time.
INIT_HOUR_INC is the hour increment in the format H < 10 or HH >= 10 (e.g. 6)
INIT_HOUR_END is the last increment hour you'd like to process in the format (e.g. For the sample data provided, GFS has 00, 06, 12, and 18, so this value would be "18")

VAR_LIST OR EXTRACT_TILES_VAR_LIST is the list of variables of interest with their levels:
Values SHOULD NOT be present in both the VAR_LIST and the EXTRACT_TILES_VAR_LIST.
e.g. VAR_LIST = HGT/P500, PRMSL/Z0, TMP/Z2
EXTRACT_TILES_VAR_LIST =

The following are used for performing series analysis based on lead time:
FHR_BEG is the beginning forecast time.
FHR_END is the ending forecast time.
FHR_INC is the forecast hour increment.

NLAT and NLON are the dimensions of the tile.

DLAT and DLON is the resolution of the data in degrees.

LON_ADJ and LAT_ADJ are the degrees to subtract from the center lat and lon to calculate the lower left lat (lat_ll) and lower left lon (lon_ll) for a grid that is 2n X 2m,
where n = LAT_ADJ degrees and m = LON_ADJ degrees. For example, where n=15 and m=15, this results in a 30 deg X 30 deg grid.


TC PAIRS filtering options

These variables contains the options used for the call to MET's tc_pairs code.


TC-STAT filtering options

These variables contains the filtering options for the call to MET's tc_stat code.


OVERWRITE OPTIONS

These variables exist so that you can choose whether or not to overwrite already processed data sets.


PLOTTING

These variables contains the possible plotting options


REGRIDDING

Tese variables contain the possible regridding options.
REGRID_USING_MET_TOOL is currently set to FALSE, as METplus is currently using wgrib2,
as opposed to regrid_data_plane, for part of its processing.


TESTING

These options are currently used by the developers and shouldn't need to be modified.

Once you have set up your environment variables and necessary configuration
files, run the following from the command line:

LOGGING
+ for running a particular use case:

These variables contain the logging options.
A LOG_LEVEL of "DEBUG" will likely provide too much information for the general user, so the user
may wish to start off with "INFO" instead.
master_metplus.py -c parm/use_cases/<use_case_name>/<use_case_name>.conf

+ for running a specific example under a use case:

master_metplus.py -c parm/use_cases/<use_case_name>/<use_case_name>.conf
-c parm/use_cases/<use_case_name>/examples/<some_example_name>.conf

How to Run?
-----------
+ for running a particular use case with settings from your custom configuration
file (to over-ride existing settings set in one of the main METplus configuration files):

Once you have set up user.template.conf you can simply run:
master_metplus.py -c parm/use_cases/<use_case_name>/<use_case_name>.conf
-c /full/path/to/my_custom.conf

master_metplus.py -c user.template.conf



================================================================================
Release Notes
================================================================================
Beta Release Notes:
------------------
2017 October 20:
METplus Beta is available under the 'release' link of the GitHub repository. The
release contains the METplus training tutorial documentation and accompanying
sample data. The feature relative, QPE, track and intensity, and cyclone plotter
use cases are now available for users to use by simply replacing /path/to with
the appropriate full file paths.

Alpha Release Notes:
-------------------
Expand All @@ -257,3 +151,4 @@ and some file operations are implemented.

2017 Jan:
- Initial release of the code.

64 changes: 42 additions & 22 deletions doc/confguide.dox
View file
Original file line number Diff line number Diff line change
@@ -1,40 +1,60 @@
/** @page confguide METplus Configuration Guide

@todo edit me: confguide.dox, THIS PAGE is a STARTING PLACEHOLDER.
THIS PAGE NEEDS TO BE FULLY REVIEWED AND EDITED.


This page explains how to modify the *.conf files in the METplus parm
directory, and explains the purpose of the various conf files. There
are a number of subpages with more detailed information:

+ @subpage metplus-conf
+ @subpage available_configurations link to page that has examples of various configurations to run METplus.
+ @ref metplus_data_conf
+ @ref metplus_system_conf
+ @ref metplus_runtime_conf


@todo Add input source and METplus configuration pages.

The *.conf files configure various aspects of the METplus system.
The *.configure various aspects of the METplus system.
Users can also override these configuration settings manually on the
command line. In the future, programmatically in the metplus.prelaunch()
function.

These are the standard conf files read in by all workflows, in the
order they are read in:

File | Purpose
------------------------ | ----------------------------------------------------
parm/metplus.conf | Detailed configuration of most aspects of METplus.
parm/user.template.conf | Basic configuration settings.

The first file, metplus.conf, should not need to be modified unless
one is making extensive changes to METplus, or adding new functionality to
the underlying scripts. Instead, one can specify additional
configuration files that override the default settings. See @ref
available_configurations for a list of alternate METplus configurations,
and how to enable them.

The user.template.conf should be modified manually. In fact, it is meant
as a starting point for a user to override the values set in metplus.conf
in order to run METplus in the users environment.
File | Purpose
------------------------- | ----------------------------------------------------
parm/met_config | Directory for MET configuration files
parm/metplus_config/metplus_data.conf | Any data related settings (e.g. location of input data, filename templates, etc.)
parm/metplus_config/metplus_system.conf | User system/host machine settings (e.g. location of executables)
parm/metplus_config/metplus_runtime.conf | Runtime related settings (e.g. process list)


NOTE: All MET configuration files should be placed in the parm/met_config directory.

There are three mandatory METplus configuration files that reside in the parm/metplus_config
directory: metplus_data.conf, metplus_system.conf,
and metplus_runtime.conf. The METplus settings are defined by key-value pairs that are
grouped under the appropriate section heading:

[dir]
OUTPUT_DIR=/path/to/output-dir
INPUT_DIR=/path/to/input-dir
TMP_DIR=/path/to/tmp-dir

The metplus_data.conf file contains information
related to data, such as the locations of input data or any filename templates.
The metplus_system.conf configuration file contains locations of executables,
source code, output directories, and any other system-related settings.
The metplus_runtime.conf configuration file contains information pertinent at
run-time, such as a process list, any begin and end times and begin and end dates for
evaluations, the log level, etc. Once these three METplus configuration files are
established, they can be reused for subsequent evaluations. Customizations
are accomplished through additional configuration files, at the
command line. The additional configuration files can be used to define new
settings or over-ride existing settings in any of the three mandatory/default
METplus config files or other existing config files.

A master METplus configuration file is created (the name and location are set in
the parm/metplus_runtime.conf file) by collecting all the information from
the three METplus configuration files and any additionally specified
configuration files.

*/
Loading

0 comments on commit 860e997

Please sign in to comment.