Merge pull request #41 from NOAA-MDL/40-updates-for-iwxxm-2023-1-release

40 updates for iwxxm 2023 1 release

.github/workflows/python-package.yml

@@ -15,12 +15,12 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ['3.8']
+        python-version: ['3.11']
 
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
     - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v2
+      uses: actions/setup-python@v4
       with:
         python-version: ${{ matrix.python-version }}
     - name: Install dependencies

README.md

@@ -21,7 +21,7 @@ IWXXM became a WMO standard on 5 November 2020. Met Offices shall disseminate ME
 
 As XML--and creating XML documents--may be unfamiliar technology to those without an IT background, MDL is providing software to assist those in creating the new XML documents based on IWXXM schemas.
 
-It should be understood that the software provided here is a short-term solution as TAC forms of these products will cease to be a ICAO/WMO standard by 2029.
+It should be understood that the software provided here is a short-term solution as TAC forms of these products will eventually cease to be a ICAO/WMO standard.
 
 ## Prequisites
 This software is written entirely in the Python language. Python interpreter v3.9 or better is required.

demo/README.md

@@ -3,12 +3,12 @@
 To illustrate the use of GIFTs, this subdirectory contains two simple python programs.
 
 ### demo1.py
-demo1.py makes use of a small aerodrome database and sample files that we'll use to translate the TAC forms into IWXXM documents.  This demonstration program requires the Python Tk/Tcl package which is readily available with Python v3.8. 
+demo1.py makes use of a small aerodrome database and sample files that we'll use to translate the TAC forms into IWXXM documents.  This demonstration program requires the Python Tk/Tcl package which is readily available with Python v3.9+. 
 
     $ cd GIFTs/demo
     $ demo1.py
 
-If the GIFTs package installation was successful, you should see a small GUI appear on your screen<sup>1</sup>, the 'Generate IWXXM From TAC Demonstrator', like so,
+If the GIFTs package installation was successful, you should see a small GUI appear on your screen, the 'Generate IWXXM From TAC Demonstrator', like so,
 
 ![Initial state](images/demo1-1.png)
 
@@ -30,25 +30,23 @@ means at the point of the caret, '^', the decoder stopped because it was expecti
 
 The second observation with a decoding problem:
 
-    METAR LGKL 110120Z 00000KT 9999 SCTO3O 18/16 Q1012
+    METAR LGKL 110120Z 00000KT 9999 SCT03O 18/16 Q1012
                                     ^
     Expecting directional minimum visibility or runway visual range or precipitation or obstruction to vision or precipitation in the vicinity or NCD, NSC or vertical visibility or cloud layer
 
-The caret indicates that the decoder doesn't understand the scattered cloud layer at 3,000 ft. Do you see why? There are typos: a capital O were used in place of zeroes, 0.  For some fonts, the difference between the two characters are subtle. This illustrates that the decoder must understand everything in the TAC report in order to properly encode the data into XML. By fixing these typos, the IWXXM message for aerodrome LGKL can be created and the LGKL TAC can now be decoded cleanly by others as well.
+The caret indicates that the decoder doesn't understand the scattered cloud layer at 3,000 ft. Do you see why? There are typos: a capital O was used in place of a zero, 0.  For some fonts, the difference between the two characters are subtle. This illustrates that the decoder must understand everything in the TAC report in order to properly encode the data into XML. By fixing these typos, the IWXXM message for aerodrome LGKL can be created and the LGKL TAC can now be decoded cleanly by others as well.
 
 The remaining METAR reports were decoded without issues and their data encoded into IWXXM and packaged up in an Meteorological Bulletin.
 
 The Encoder class requires that the input file contain one WMO AHL line, appropriate for the TAC forms within it. Here are the regular expressions used to identify the WMO AHL line for particular TAC forms:
 
     S(A|P)[A-Z][A-Z]\d\d\s+[A-Z]{4}\s+\d{6}(\s+[ACR]{2}[A-Z])? # for METAR/SPECI
-    FN\w\w\d\d\s+[A-Z]{4}\s+\d{6}(\s+[ACR]{2}[A-Z])? # Space Weather Advisories
     FK\w\w\d\d\s+[A-Z]{4}\s+\d{6}(\s+[ACR]{2}[A-Z])? # Tropical Cyclone Advisory
     F(C|T)\w\w\d\d\s+[A-Z]{4}\s+\d{6}(\s+[ACR]{2}[A-Z])? # TAF
     FV\w\w\d\d\s+[A-Z]{4}\s+\d{6}(\s+[ACR]{2}[A-Z])? # Volcanic Ash Advisory
 And for capturing the individual TAC forms:
 
     ^(?:METAR|SPECI)\s+(?:COR\s+)?[A-Z][A-Z0-9]{3}\s.+?=
-    ^SWX ADVISORY.+
     ^TC ADVISORY.+
     ^TAF(?:\s+(?:AMD|COR|CC[A-Z]|RTD))?\s+[A-Z]{4}.+?=
     ^VA ADVISORY.+
@@ -63,11 +61,10 @@ This program runs as a UNIX/Linux [daemon](https://en.wikipedia.org/wiki/Daemon_
 
     $ pip install watchdog
 
-The associated configuration file, `iwxxmd.cfg`, is well documented internally and should suffice in getting the IWXXM daemon up and running. Once the configuration file is properly set up, to start the daemon, just issue the command:
+The associated configuration template file, `iwxxmd.cfg`, is well documented internally and should suffice in getting a IWXXM daemon up and running. Once the configuration file for a particular product is created, start the daemon. For instance,
 
-    $ iwxxmd.py iwxxmd.cfg
+    $ iwxxmd.py metar.cfg
 
-Any misconfiguration will result in an error message being written to the console and the daemon will not start. Like most UNIX/Linux daemons, the process can run indefinitely in the background. Should the daemon run into any difficulties, it will write messages to its log file.  The log file name format follows this format `<product>_iwxxmd_<DOW>` where `<product>` is one of `'metar'`, `'swa'`, `'taf'`, `'tca'`, or `'vaa'`, and `<DOW>` is the abbreviated day of the week, e.g. `'vaa_iwxxmd_Mon'`. The daemon will 'ping' to the log file every hour to indicate that it is 'alive'. While active, the daemon will report the files read in and the files written out. When midnight arrives, the daemon will switch to a different log file. Thus, a maximum of seven log files are created with each file being overwritten after 6 days.
+By copying the template file with new names as needed, several IWXXM daemons can run simultaneously.
 
--------------------
-<sup>1</sup>When running the demo1.py program for the first time, it may take several minutes before the GUI appears. This is due to the Skyfield module downloading a large ephemris file from the JPL website. To disable this activity, comment out (or remove) lines 104-105 in demo1.py which initializes the Space Weather Advisory Encoder.
+Any misconfiguration will result in an error message being written to the console and the daemon will not start. Like most UNIX/Linux daemons, the process can run indefinitely in the background. Should the daemon run into any difficulties, it will write messages to its log file.  The log file name format follows this format `<product>_iwxxmd_<DOW>` where `<product>` is one of `'metar'`, `'swa'`, `'taf'`, `'tca'`, or `'vaa'`, and `<DOW>` is the abbreviated day of the week, e.g. `'metar_iwxxmd_Mon'`. The daemon will 'ping' to the log file every hour to indicate that it is 'alive'. While active, the daemon will report the files read in and the files written out. When midnight arrives, the daemon will switch to a different log file. Thus, a maximum of seven log files are created with each file being overwritten after 6 days.

demo/demo1.py

@@ -101,8 +101,6 @@ def __init__(self):
         self.encoders = []
         self.encoders.append((re.compile(r'^S(A|P)[A-Z][A-Z]\d\d\s+[A-Z]{4}\s+\d{6}', re.MULTILINE),
                               gifts.METAR.Encoder(aerodromes)))
-        self.encoders.append((re.compile(r'^FN[A-Z][A-Z]\d\d\s+[A-Z]{4}\s+\d{6}', re.MULTILINE),
-                              gifts.SWA.Encoder()))
         self.encoders.append((re.compile(r'^F(C|T)[A-Z][A-Z]\d\d\s+[A-Z]{4}\s+\d{6}', re.MULTILINE),
                               gifts.TAF.Encoder(aerodromes)))
         self.encoders.append((re.compile(r'FK\w\w\d\d\s+[A-Z]{4}\s+\d{6}', re.MULTILINE), gifts.TCA.Encoder()))

demo/iwxxmd.py

@@ -119,7 +119,7 @@ def __init__(self, encoder, delete_flag, header, outputDirectory):
         else:
             self.ext = 'xml'
 
-    def on_modified(self, event):
+    def on_closed(self, event):
         """If a new file saved in monitored directory, read it."""
 
         if not event.is_directory:

gifts/common/xmlConfig.py

@@ -6,7 +6,7 @@
 # Author: Mark Oberfield
 # Organization: NOAA/NWS/OSTI/Meteorological Development Laboratory
 # Contact Info: [email protected]
-# Date: 16 January 2020
+# Date: 5 June 2023
 #
 import os
 # -----------------------------------------------------------------------------------
@@ -33,8 +33,8 @@
 # -----------------------------------------------------------------------------------
 #
 # IWXXM versioning
-_iwxxm = '2021-2'
-_release = '2021-2'
+_iwxxm = '2023-1'
+_release = '2023-1'
 #
 IWXXM_URI = 'http://icao.int/iwxxm/%s' % _iwxxm
 IWXXM_URL = 'https://schemas.wmo.int/iwxxm/%s/iwxxm.xsd' % _release

setup.cfg

@@ -4,7 +4,7 @@ description = IWXXM encoders for Annex3 TAC products
 author = Mark Oberfield - NOAA/NWS/OSTI/MDL/WIAD
 author_email = [email protected]
 maintainer = Mark Oberfield
-version = 1.4.3
+version = 1.5.0
 classifiers = Programming Language :: Python :: 3
     Operating System :: OS Independent
     Topic :: Text Processing :: Markup :: XML

validation/README.md

@@ -1,24 +1,18 @@
 # Introduction
 
-This software package can be used to check IWXXM documents for correctly-formed XML, schema and schematron
-('business rules') validation. The Java code, CRUX, performs the validation steps. Information on CRUX can
-be found at the following URL: https://github.com/NCAR/crux.
+This software package can be used to check IWXXM documents for correctly-formed XML, schema and schematron ('business rules') validation. The Java code, CRUX, performs the validation steps. Information on CRUX can be found at the following URL: https://github.com/NCAR/crux.
 
 
 Prequisites
 -----------
 
 The software consists of Python and Java code.
 
-Python code requires the 'lxml' and 'requests' modules to parse XML documents and retrieve IWXXM schemas and
-RDF files for local (and fast) validation. These modules can be obtained via the 'pip' command, if not already
-installed.
+Python code requires the 'lxml' and 'requests' modules to parse XML documents and retrieve IWXXM schemas and RDF files for local (and fast) validation. These modules can be obtained via the 'pip' command, if not already installed.
 
-The version of python must be at least 2.7 or better.
+The version of python must be at least 3.9 or better.
 
-This code has been developed, tested and works on a reasonably up-to-date Linux OS and Windows 10 machine.
-For old and/or other operating systems, this code may need some adjustments, but there are no guarantees.
-We are not obligated to make this software work on your computers.
+This code has been developed, tested and works on a reasonably up-to-date Linux OS and Windows 10 machine. For old and/or other operating systems, this code may need some adjustments, but there are no guarantees.
 
 
 Installation
@@ -117,8 +111,7 @@ The package has the following directory tree:
 IWXXM Validation
 ----------------
 
-The python script, 'iwxxmValidator.py' requires a single argument, the directory path to the IWXXM XML
-documents.
+The python script, 'iwxxmValidator.py' requires a single argument, the directory path to the IWXXM XML documents.
 
 Invoking the script for help with the '-h' or '--help' flag provides the following options:
 
@@ -139,51 +132,36 @@ Invoking the script for help with the '-h' or '--help' flag provides the followi
 	  -k, --keep            do not delete catalog file when validation finishes
 	  -v VERSION, --version VERSION
                         	IWXXM version major.minor number to validate against,
-                        	default '3.0'
+                        	default '2023-1''
 
 --version flag
 ------------------
-By default, the validation tool checks IWXXM 3.0 documents. If your IWXXM XML documents are based on a 
-different version of IWXXM, provide the appropriate combination using the '-v' or '--version'
+By default, the validation tool checks IWXXM 2023-1 documents. If your IWXXM XML documents are based on a different version of IWXXM, provide the appropriate combination using the '-v' or '--version'
 flag.
 
-The script will check for this version's local copy of the IWXXM schemas and schematron, and associated
-RDF files from the WMO Code Registry. If a copy is not found, the script will go to the canonical sources,
-'https://schemas.wmo.int/iwxxm' and 'http://codes.wmo.int', to download them. Therefore, your machine will
-need access to the Internet when running this script the first time, and when switching to new versions.
+The script will check for this version's local copy of the IWXXM schemas and schematron, and associated RDF files from the WMO Code Registry. If a copy is not found, the script will go to the canonical sources, 'https://schemas.wmo.int/iwxxm' and 'http://codes.wmo.int', to download them. Therefore, your machine will need access to the Internet when running this script the first time, and when switching to new versions.
 
 
 --fetch flag
 ----------------
-If circumstances require it, you can force the script to download and overwrite the local cache of the 
-IWXXM schemas and schematron files with the '-f' or '--fetch' flag. (Default: do not fetch)
+If circumstances require it, you can force the script to download and overwrite the local cache of the IWXXM schemas and schematron files with the '-f' or '--fetch' flag. (Default: do not fetch)
 
 
 --keep flag
 ---------------
-The validator creates an OASIS style Catalog file on-the-fly for local validation which speeds up the
-process up considerably. It is normally deleted when the script finishes. (Default: do not keep)
+The validator creates an OASIS style Catalog file on-the-fly for local validation which speeds up the process up considerably. It is normally deleted when the script finishes. (Default: do not keep) The catalog file can be used in "XML-aware" editors that can perform XML full validation. With this flag set, the OASIS Catalog file is kept in the top-level directory with the name, 'catalog-VERSION.xml' where VERSION is the IWXXM version string.
 
-The catalog file can be used in "XML-aware" editors that can perform XML full validation. With this flag
-set, the OASIS Catalog file is kept in the top-level directory with the name, 'catalog-VERSION.xml' where
-VERSION is the IWXXM version string.
-
-If the VERSION catalog file is already present in the directory, the script will NOT overwrite it, but use it as
-is.
+If the VERSION catalog file is already present in the directory, the script will NOT overwrite it, but use it as is.
 
 
 --noGMLChecks flag
 ----------------------
-After performing XML validation, a further examination of the internal and external references within each
-XML document is done. As a prerequisite, this step requires the XML document to be 'well-formed'. If this
-flag is given, this check is skipped. (Default: do GML reference checks)
+After performing XML validation, a further examination of the internal and external references within each XML document is done. As a prerequisite, this step requires the XML document to be 'well-formed'. If this flag is given, this check is skipped. (Default: do GML reference checks)
 
 
 --useInternet flag
 ----------------------
-If GML checks are enabled, the algorithm has the option to query code registries which requires
-Internet connectivity (and can be slow). Or the algorithm can refer to the local copy of the RDF files to
-determine valid references to code lists (fast).  (Default: use local copy of RDF files)
+If GML checks are enabled, the algorithm has the option to query code registries which requires Internet connectivity (and can be slow). Or the algorithm can refer to the local copy of the RDF files to determine valid references to code lists (fast).  (Default: use local copy of RDF files)
 
 To run:
 -------
@@ -196,13 +174,13 @@ is sufficient.
 
 Notes:
 ------
-This software is not meant to be a subtitute for more sophisticated XML-aware applications. This is a basic, relatively "unfriendly" tool if you are new to XML documents and the technology associated with them. It _can_ help you find errors in your documents but sometimes the error messages from CRUX are cryptic. We make no apologies for that.
+This software is not meant to be a subtitute for more sophisticated XML-aware applications. This is a basic, relatively "unfriendly" tool if you are new to XML documents and the technology associated with them. It _can_ help you find errors in your documents but sometimes the error messages from CRUX are cryptic.
 
 When using tool the first time on Windows machines, the user must have the ability to create a symbolic link to a file as it's created as part of downloading files from the WMO Code Registry.  After the initial downloading of files and setup, this particular privilege is no longer needed on subsequent invocations of `iwxxmValidator.py`.
 
 This script can be used to quickly validate IWXXM messages before dissemination in an operational environment.
 
-There are numerous examples of validated IWXXM documents on the Internet.  The canoncial IWXXM source has a few instances in the https://schemas.wmo.int/iwxxm/VERSION/examples folders.
+There are numerous examples of validated IWXXM documents on the Internet.  The canonical IWXXM source has a few instances in the https://schemas.wmo.int/iwxxm/VERSION/examples folders.
 
 Another repository of examples is the WMO-IM GitHub site: https://github.com/wmo-im/iwxxm-translation
 
@@ -216,9 +194,6 @@ Your e-mail will go directly to TT-AvData team members.
 
 To access our team's e-mail archive, visit https://groups.wmo.int to subscribe.
 
-An 'ignoredURLs.txt' file is provided for the case when your IWXXM documents have `<extension>` blocks that
-contain references to URLs that are not part of the WMO Code Registry. By adding the URLs in this file,
-this will suppress warning messages from the checkGMLReferences routine.
+An 'ignoredURLs.txt' file is provided for the case when your IWXXM documents have `<extension>` blocks that contain references to URLs that are not part of the WMO Code Registry. By adding the URLs in this file, this will suppress warning messages from the checkGMLReferences routine.
 
-The schematron portion of the CRUX utility will create a directory cache called 'cruxcache' sub-directory
-in the directory designated in java.io.tmpdir
+The schematron portion of the CRUX utility will create a directory cache called 'cruxcache' sub-directory in the directory designated in java.io.tmpdir

validation/iwxxmValidator.py

@@ -105,8 +105,8 @@ def validate_xml_files(cmdargs):
     parser.add_argument("--noGMLChecks", help="skip GML link checking", action="store_true", default=False)
     parser.add_argument("-k", "--keep", help="do not delete catalog file when validation finishes",
                         action="store_true", default=False)
-    parser.add_argument("-v", "--version", help="IWXXM version major.minor number to validate against, default '3.0'",
-                        type=str, default="3.0")
+    parser.add_argument("-v", "--version", help="IWXXM version major.minor number to validate against, default '2023-1'",
+                        type=str, default="2023-1")
     parser.add_argument("directory", help="directory path containing IWXXM XML documents for validation", type=str)
     cmdargs = parser.parse_args()