Merge branch 'main' of github.com:NOAA-GSL/idss-engine-commons into main

README.md

@@ -2,7 +2,7 @@
 [![Pytest](https://github.com/NOAA-GSL/idss-engine-commons/actions/workflows/run-tests.yml/badge.svg?branch=main)](https://github.com/NOAA-GSL/idss-engine-commons/actions/workflows/run-tests.yml)
 
 <!-- Pytest Coverage Comment:Begin -->
-<a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main/README.md"><img alt="Coverage" src="https://img.shields.io/badge/Coverage-54%25-orange.svg" /></a><details><summary>Coverage Report </summary><table><tr><th>File</th><th>Stmts</th><th>Miss</th><th>Cover</th><th>Missing</th></tr><tbody><tr><td colspan="5"><b>python/idsse_common/idsse/common</b></td></tr><tr><td>&nbsp; &nbsp;<a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main/python/idsse_common/idsse/common/__init__.py">__init__.py</a></td><td>0</td><td>0</td><td>100%</td><td>&nbsp;</td></tr><tr><td>&nbsp; &nbsp;<a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main/python/idsse_common/idsse/common/aws_utils.py">aws_utils.py</a></td><td>77</td><td>5</td><td>5</td><td><a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main//home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/aws_utils.py#L 94%"> 94%</a></td></tr><tr><td>&nbsp; &nbsp;<a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main/python/idsse_common/idsse/common/config.py">config.py</a></td><td>74</td><td>9</td><td>9</td><td><a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main//home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/config.py#L 88%"> 88%</a></td></tr><tr><td>&nbsp; &nbsp;<a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main/python/idsse_common/idsse/common/json_message.py">json_message.py</a></td><td>22</td><td>22</td><td>22</td><td><a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main//home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/json_message.py#L  0%">  0%</a></td></tr><tr><td>&nbsp; &nbsp;<a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main/python/idsse_common/idsse/common/log_util.py">log_util.py</a></td><td>38</td><td>38</td><td>38</td><td><a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main//home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/log_util.py#L  0%">  0%</a></td></tr><tr><td>&nbsp; &nbsp;<a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main/python/idsse_common/idsse/common/path_builder.py">path_builder.py</a></td><td>127</td><td>10</td><td>10</td><td><a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main//home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/path_builder.py#L 92%"> 92%</a></td></tr><tr><td>&nbsp; &nbsp;<a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main/python/idsse_common/idsse/common/publish_confirm.py">publish_confirm.py</a></td><td>141</td><td>141</td><td>141</td><td><a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main//home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/publish_confirm.py#L  0%">  0%</a></td></tr><tr><td>&nbsp; &nbsp;<a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main/python/idsse_common/idsse/common/utils.py">utils.py</a></td><td>78</td><td>33</td><td>33</td><td><a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main//home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/utils.py#L 58%"> 58%</a></td></tr><tr><td><b>TOTAL</b></td><td><b>557</b></td><td><b>258</b></td><td><b>54%</b></td><td>&nbsp;</td></tr></tbody></table></details>
+<a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main/README.md"><img alt="Coverage" src="https://img.shields.io/badge/Coverage-54%25-orange.svg" /></a><details><summary>Coverage Report </summary><table><tr><th>File</th><th>Stmts</th><th>Miss</th><th>Cover</th><th>Missing</th></tr><tbody><tr><td colspan="5"><b>python/idsse_common/idsse/common</b></td></tr><tr><td>&nbsp; &nbsp;<a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main/python/idsse_common/idsse/common/__init__.py">__init__.py</a></td><td>0</td><td>0</td><td>100%</td><td>&nbsp;</td></tr><tr><td>&nbsp; &nbsp;<a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main/python/idsse_common/idsse/common/aws_utils.py">aws_utils.py</a></td><td>78</td><td>6</td><td>6</td><td><a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main//home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/aws_utils.py#L 92%"> 92%</a></td></tr><tr><td>&nbsp; &nbsp;<a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main/python/idsse_common/idsse/common/config.py">config.py</a></td><td>74</td><td>9</td><td>9</td><td><a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main//home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/config.py#L 88%"> 88%</a></td></tr><tr><td>&nbsp; &nbsp;<a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main/python/idsse_common/idsse/common/json_message.py">json_message.py</a></td><td>22</td><td>22</td><td>22</td><td><a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main//home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/json_message.py#L  0%">  0%</a></td></tr><tr><td>&nbsp; &nbsp;<a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main/python/idsse_common/idsse/common/log_util.py">log_util.py</a></td><td>38</td><td>38</td><td>38</td><td><a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main//home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/log_util.py#L  0%">  0%</a></td></tr><tr><td>&nbsp; &nbsp;<a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main/python/idsse_common/idsse/common/path_builder.py">path_builder.py</a></td><td>127</td><td>10</td><td>10</td><td><a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main//home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/path_builder.py#L 92%"> 92%</a></td></tr><tr><td>&nbsp; &nbsp;<a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main/python/idsse_common/idsse/common/publish_confirm.py">publish_confirm.py</a></td><td>141</td><td>141</td><td>141</td><td><a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main//home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/publish_confirm.py#L  0%">  0%</a></td></tr><tr><td>&nbsp; &nbsp;<a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main/python/idsse_common/idsse/common/utils.py">utils.py</a></td><td>78</td><td>33</td><td>33</td><td><a href="https://github.com/NOAA-GSL/idss-engine-commons/blob/main//home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/utils.py#L 58%"> 58%</a></td></tr><tr><td><b>TOTAL</b></td><td><b>558</b></td><td><b>259</b></td><td><b>54%</b></td><td>&nbsp;</td></tr></tbody></table></details>
 <!-- Pytest Coverage Comment:End -->
 
 ## Overview

python/idsse_common/idsse/common/aws_utils.py

@@ -11,8 +11,8 @@
 
 import logging
 import os
-from datetime import datetime, timedelta, timezone
-from typing import Sequence, Tuple
+from datetime import datetime, timedelta
+from typing import Sequence, Tuple, Optional
 
 from .path_builder import PathBuilder
 from .utils import TimeDelta, datetime_gen, exec_cmd
@@ -85,23 +85,23 @@ def aws_cp(self, path: str, dest: str) -> bool:
                 commands = ['aws', 's3', '--no-sign-request',  'cp', path, dest]
                 exec_cmd(commands)
                 return True
-            except:
+            except Exception:  # pylint: disable=broad-exception-caught
                 return False
             finally:
                 pass
-
 
-    def check_for(self, issue: datetime, valid: datetime) -> Tuple[datetime, str]:
+
+    def check_for(self, issue: datetime, valid: datetime) -> Optional[Tuple[datetime, str]]:
         """Checks if an object passed issue/valid exists
 
         Args:
             issue (datetime): The issue date/time used to format the path to the object's location
             valid (datetime): The valid date/time used to format the path to the object's location
 
         Returns:
-            Tuple[datetime, str]: If object exists the valid date/time (indicated by object's
-                                  location) and the object's location (path) is returned as a tuple,
-                                  else the tuple(None, None) is returned
+            Optional[Tuple[datetime, str]]: A tuple of the valid date/time (indicated by object's 
+                                            location) and location (path) of a object, or None 
+                                            if object does not exist
         """
         lead = TimeDelta(valid-issue)
         filenames = self.aws_ls(self.get_path(issue, valid), prepend_path=False)
@@ -113,20 +113,20 @@ def check_for(self, issue: datetime, valid: datetime) -> Tuple[datetime, str]:
 
     def get_issues(self,
                    num_issues: int = 1,
-                   issue_start: datetime = None,
-                   issue_end: datetime = datetime.now(timezone.utc)
+                   issue_start: Optional[datetime] = None,
+                   issue_end: datetime = datetime.utcnow()
                    ) -> Sequence[datetime]:
         """Determine the available issue date/times
 
         Args:
-            num_issues (int, optional): Maximum number of issue to return. Defaults to 1.
+            num_issues (int): Maximum number of issue to return. Defaults to 1.
             issue_start (datetime, optional): The oldest date/time to look for. Defaults to None.
-            issue_end (datetime, optional): The newest date/time to look for. Defaults to None.
+            issue_end (datetime): The newest date/time to look for. Defaults to now (UTC).
 
         Returns:
-            Sequence[Tuple[datetime, str]]: A sequence of issue date/times
+            Sequence[datetime]: A sequence of issue date/times
         """
-        issues_found = []
+        issues_found: Sequence[datetime] = []
         if issue_start:
             datetimes = datetime_gen(issue_end, timedelta(hours=-1), issue_start, num_issues)
         else:
@@ -148,8 +148,8 @@ def get_issues(self,
 
     def get_valids(self,
                    issue: datetime,
-                   valid_start: datetime = None,
-                   valid_end: datetime = None) -> Sequence[Tuple[datetime, str]]:
+                   valid_start: Optional[datetime] = None,
+                   valid_end: Optional[datetime] = None) -> Sequence[Tuple[datetime, str]]:
         """Get all objects consistent with the passed issue date/time and filter by valid range
 
         Args:
@@ -161,10 +161,12 @@ def get_valids(self,
 
         Returns:
             Sequence[Tuple[datetime, str]]: A sequence of tuples with valid date/time (indicated by
-                                            object's location) and the object's location (path)
+                                            object's location) and the object's location (path). 
+                                            Empty Sequence if no valids found for given time range.
         """
         if valid_start and valid_start == valid_end:
-            return [self.check_for(issue, valid_start)]
+            valids_and_filenames = self.check_for(issue, valid_start)
+            return [valids_and_filenames] if valids_and_filenames is not None else []
 
         dir_path = self.path_builder.build_dir(issue=issue)
 

python/idsse_common/idsse/common/config.py

@@ -13,7 +13,7 @@
 import json
 import logging
 from inspect import signature
-from typing import Self, Union, List
+from typing import Self, Union, List, Optional
 
 logger = logging.getLogger(__name__)
 
@@ -23,7 +23,7 @@ class Config:
 
     def __init__(self,
                  config: Union[dict, List[dict], str],
-                 keys: Union[list, str] = None,
+                 keys: Optional[Union[list, str]] = None,
                  recursive: bool = False,
                  ignore_missing: bool = False) -> None:
 
@@ -49,7 +49,7 @@ def __init__(self,
             filepaths = glob.glob(config, recursive=recursive)
             if len(filepaths) == 0:
                 raise FileNotFoundError
-            elif len(filepaths) == 1:
+            if len(filepaths) == 1:
                 self._from_filepath(filepaths[0], keys)
             else:
                 self._from_filepaths(filepaths, keys)

python/idsse_common/idsse/common/log_util.py

@@ -31,7 +31,7 @@
 def set_corr_id_context_var(originator: str, key: uuid = None, issue_dt: Union[str, datetime] = None) -> None:
     if not key:
         key = uuid.uuid4()
-        
+
     if issue_dt:
         if not isinstance(issue_dt, str):
             issue_dt = to_iso(issue_dt)
@@ -72,7 +72,7 @@ def get_default_log_config(level, with_corr_id=True):
         format_str = '%(asctime)-15s %(name)-5s %(levelname)-8s %(corr_id)s %(module)s::%(funcName)s(line %(lineno)d) %(message)s'
     else:
         format_str = '%(asctime)-15s %(name)-5s %(levelname)-8s %(module)s::%(funcName)s(line %(lineno)d) %(message)s'
-        
+
     return {
     'version': 1,
     'disable_existing_loggers': False,

python/idsse_common/idsse/common/publish_confirm.py

@@ -6,9 +6,9 @@
 import logging.config
 import json
 import time
+from threading import Thread
 
 import pika
-from threading import Thread
 from pika.exchange_type import ExchangeType
 
 from idsse.common.log_util import get_default_log_config, set_corr_id_context_var

python/idsse_common/idsse/common/utils.py

@@ -13,7 +13,7 @@
 import logging
 from datetime import datetime, timedelta
 from subprocess import Popen, PIPE, TimeoutExpired
-from typing import Sequence
+from typing import Sequence, Optional, Generator, Any
 
 logger = logging.getLogger(__name__)
 
@@ -70,7 +70,7 @@ def __delitem__(self, key):
         del self.__dict__[key]
 
 
-def exec_cmd(commands: Sequence[str], timeout: int = None) -> Sequence[str]:
+def exec_cmd(commands: Sequence[str], timeout: Optional[int] = None) -> Sequence[str]:
     """Execute the passed commands via a Popen call
 
     Args:
@@ -96,8 +96,8 @@ def exec_cmd(commands: Sequence[str], timeout: int = None) -> Sequence[str]:
         raise OSError(process.returncode, errs.decode())
     try:
         ans = outs.decode().splitlines()
-    except Exception as e:  # pylint: disable=broad-exception-caught
-        raise RuntimeError(e) from e
+    except Exception as exc:  # pylint: disable=broad-exception-caught
+        raise RuntimeError(exc) from exc
     return ans
 
 
@@ -145,8 +145,8 @@ def dict_copy_with(old_dict: dict, **kwargs) -> dict:
 
 def datetime_gen(dt_: datetime,
                  time_delta: timedelta,
-                 end_dt: datetime = None,
-                 max_num: int = 100) -> datetime:
+                 end_dt: Optional[datetime] = None,
+                 max_num: int = 100) -> Generator[datetime, Any, None]:
     """Create a date/time sequence generator, given a starting date/time and a time stride
 
     Args:

python/idsse_common/test/coverage.txt

@@ -15,14 +15,14 @@ test_utils.py ..........                                                 [100%]
 Name                                                                                                            Stmts   Miss  Cover
 -----------------------------------------------------------------------------------------------------------------------------------
 /home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/__init__.py              0      0   100%
-/home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/aws_utils.py            77      5    94%
+/home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/aws_utils.py            78      6    92%
 /home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/config.py               74      9    88%
 /home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/json_message.py         22     22     0%
 /home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/log_util.py             38     38     0%
 /home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/path_builder.py        127     10    92%
 /home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/publish_confirm.py     141    141     0%
 /home/runner/work/idss-engine-commons/idss-engine-commons/python/idsse_common/idsse/common/utils.py                78     33    58%
 -----------------------------------------------------------------------------------------------------------------------------------
-TOTAL                                                                                                             557    258    54%
+TOTAL                                                                                                             558    259    54%
 
-============================== 58 passed in 0.46s ==============================
+============================== 58 passed in 0.38s ==============================