From 77f4771f748b181c645ec78dbe8c4fa7876917bc Mon Sep 17 00:00:00 2001 From: bikegeek Date: Mon, 24 Jul 2023 12:17:40 -0600 Subject: [PATCH] Completed instructions, first commit --- docs/Users_Guide/load_data.rst | 118 +++++++++++++++++++++++++++++++-- 1 file changed, 112 insertions(+), 6 deletions(-) diff --git a/docs/Users_Guide/load_data.rst b/docs/Users_Guide/load_data.rst index 71162a32..e0db90d2 100644 --- a/docs/Users_Guide/load_data.rst +++ b/docs/Users_Guide/load_data.rst @@ -1,8 +1,112 @@ -Load Data -========= +Background +========== -METdbload is used to insert MET output data into the database. The usage -statement: +The METdbLoad module provides support for inserting MET output data into the database. + +Before using the METdbLoad module, the database must exist and have the proper permissions +(i.e. grant privileges to insert, delete, update, and index). A schema file, *mv_mysql.sql* is available in the +METdataio/METdbLoad/sql/ directory for importing into the database. + +The METdbLoad script *met_db_load.py* performs the loading of data based on settings in an XML specification file. + +In the METdataio/METdbLoad/sql/scripts directory, there are two configuration files: + + * db_load_specification.xml + + * data_loading_config.yaml + +The *db_load_specification.xml* is a template XML specification file, and *data_loading_config.yaml* +is a template YAML configuration file. The *data_loading_config.yaml* file contains +information about the database (username, password, database name, etc.). This information is used by the +*generate_xml_spec.py* script to generate the XML specification file which is then used to load data into the database. + +Generate the XML specification file +----------------------------------- + +Copy the *data_loading_config.yaml* file to a secure location in your workspace, as this file will contain the username +and password to the database. **Do not put this file where it can be read by anyone who should not have access to this +information.** + +.. code-block:: ini + +cp data_loading_config.yaml /path-to-your-dir/ + +Replace the *path-to-your-dir* with the actual path to where this file is to be saved. + +Change directory to the location where the *data_loading_config.yaml* file was copied. + +Open the data_loading_config.yaml file: + +.. literalinclude:: ../../METdbLoad/sql/scripts/data_loading_config.yaml + +Update the database information with information relevant to the database you are using: + +* dbname + +* username + +* password + +* host + +* port + + +Update the path to the schema location, provide the full path to the *mv_sql_mysql.sql* schema file: + +* schema_location + +Provide the name and full path to the *db_load_specification.xml* template file, this will be updated +with the settings in this YAML configuration to create a new XML specification file using these settings: + +* xml_specification + +Provide the group and description. The databases in METviewer are grouped, provide the name of the appropriate +group and a brief description of the database in which the data is to be loaded: + +* group + +* description + +Provide the full path to the directory where the MET data to be loaded is saved: + +* data_dir + +Indicate which data types are to be loaded by setting the appropriate settings to True: + +* load_stat + +* load_mode + +* load_mtd + +* load_mpr + +* load_orank + +Generate the new XML specification file by running the following: + +.. code-block:: ini + + cd path-to-METdataio-source/METdataio/METdbLoad/sql/scripts + + *Replace path-to-METdataio-source to the location where the METdataio source code is saved. + + python generate_xml_spec.py path-to/data_loading_config.yaml + + *Replace the path-to with the path to the directory you created to store the copy of the data_loading_config.yaml + file as specified earlier. + +A new XML specification file *load_met.xml*, will be generated and saved in the +same directory where the YAML configuration file was copied. + +Load data +--------- + +Now the MET data can be loaded in the database using the *met_db_load.py* script in the path-to/METdataio/METdbLoad/ush +directory. The *path-to* is the directory where the METdataio source code is saved. + +The usage statement: .. code-block:: ini @@ -18,7 +122,7 @@ statement: -h, --help show this help message and exit -index Only process index, do not load data -The **xmlfile** passes information about the MET output files to load +The **xmlfile** is the XML specification file that passes information about the MET output files to load into the database to METdbload. It is an XML file whose top-level tag is and it contains the following elements, divided into functional sections: @@ -141,6 +245,9 @@ functional sections: Example _______ +The data to be loaded can be filtered by date and field names (i.e. by model, valid_time, vx_mask, etc.). +The load_met.xml specification file can be further modified to accomplish this. + Here is a simple example: .. code-block:: XML @@ -228,4 +335,3 @@ _______________ row already exists in the table with the insert information, then the existing record will be used instead of trying to insert a dupilcate. -