Code

Automate R script #25

Code

Merged

glitt13 merged 122 commits intoNOAA-OWP:mainNOAA-OWP/formulation-selector:mainfrom glitt13:hfabglitt13/formulation-selector:hfabCopy head branch name to clipboardOct 24, 2024

+392-49

Commits

Commits on Aug 22, 2024

• feat: developing algorithm training and evaluation module

  glitt13

  committed
  1c420db

• fix: minor bug fixes with paths and np array values retrieval

  glitt13

  committed
  7f3921e

• feat: create initial fs_algo package

  glitt13

  committed
  d9c8dfe

Commits on Aug 23, 2024

• feat: contain all training/eval into a single class

  glitt13

  committed
  1ebcc23

• feat: simplify evaluation file write and module import

  glitt13

  committed
  1a74ca6

• feat: add basic unit testing for AlgoTrainEval class

  glitt13

  committed
  a9019bd

• feat: convert save dir structure creation and fsds dataset reader into modular functions inside package

  glitt13

  committed
  3a7025e

• Update README.md

  Show description for 5e922c4
  glitt13

  authored
  5e922c4

• feat: simplify aspects of attribute organization and combining with metrics

  glitt13

  committed
  b6dffaf

• feat: beginning to convert attribute wrangling into a class

  glitt13

  committed
  84d9de6

• feat: add algorithm configuration file

  glitt13

  committed
  f78cb2c

• feat: established class for attribute configuration file, scripts functioning

  glitt13

  committed
  f1d15a8

• feat: add verbose option

  glitt13

  committed
  b75e3d5

• fix: update to warnings.warn()

  glitt13

  committed
  02325af

• feat: building out additional unit tests for AttrConfigAndVars class

  glitt13

  committed
  2c480d6

Commits on Aug 27, 2024

• chore: remove spaces

  glitt13

  committed
  2c2ae67

• chore: remove spaces

  glitt13

  committed
  dd8f796

Commits on Aug 28, 2024

• feat: add unit test for fs_read_attr_comid

  glitt13

  committed
  f96ca62

• feat: add UserWarnings and associated unit test

  glitt13

  committed
  96debe1

Commits on Aug 29, 2024

• feat: add unit tests for _find_feat_srce_id, fs_retr_nhdp_comids and fix associated functions when behavior didn't follow expected behavior

  glitt13

  committed
  c7f8e61

• feat: add unit test for fs_save_algo_dir_struct

  glitt13

  committed
  54a3a35

• feat: a basic unit test for _open_response_data_fsds

  glitt13

  committed
  619e1a5

• chore: simplify algo script based on functionality moved into fs_algo_train_eval module

  glitt13

  committed
  f28d90d

• doc: add sphinx documentation to _read_attr_config and fs_read_attr_comid

  glitt13

  committed
  48a1e6a

• doc: add sphinx-formatted documentation to the functions in the fs_algo_train_eval module; feat: move some hard-coded variables into the algorithm config file

  glitt13

  committed
  b9b034e

• fix: changes vars to attrs in AlgoTrainEval arg

  glitt13

  committed
  75cfb58

• fix: added the new parameters that were hard-coded (test_size & seed)

  glitt13

  committed
  901ff10

• fix: swapped the train/test fractions to appropriate printout order

  glitt13

  committed
  f4780dc

Commits on Aug 30, 2024

• feat: make sphinx documentation

  glitt13

  committed
  fa703c4

• fix: reinstall sphinx docs for fsds_proc

  glitt13

  committed
  43d88c0

• fix: remove unused path_camels

  glitt13

  committed
  342838d

• fix: remove unused references to path_camels

  glitt13

  committed
  283da30

• fix: update standard fsds_proc config files to create netcdf rather than csv; rename these files schema to config

  glitt13

  committed
  10cea4f

• doc: update config file documentation on preferred save_type

  glitt13

  committed
  591cbed

• doc: update description of yaml file's dataset

  glitt13

  committed
  6a4d645

• fix: update config files with featureID and featureSource entries

  glitt13

  committed
  8af7720

• fix: change vars to attrs based on package's object name change

  glitt13

  committed
  d4c842b

• fix: change logic to ensure config file read if dataset attribute read failed

  glitt13

  committed
  f179124

Commits on Aug 31, 2024

• feat: add a raw data input checker/corrector for cases when nwissite gage ids are missing the leading 0

  glitt13

  committed
  7275e8a

• fix: changed path_data to represent the raw input files containing corrected nwissite USGS gage ids (leading zeros)

  glitt13

  committed
  1f0a8be

• fix: added appropriate fillna for nwissite gage ids not needed to be corrected

  glitt13

  committed
  36414d5

• fix: adjust path check for attributes instead of algo

  glitt13

  committed
  728f1b4

• doc: add descriptive notes on algo pre-processing and suggest future improvements for datasets not processed with fsds_proc with TODO

  glitt13

  committed
  aef2f01

• doc: simplify attr_config, change dir_attrs to dir_db_attrs

  glitt13

  committed
  3b19a2a

• chore: add some additional hydroatlas and USGS NHD variables for consideration

  glitt13

  committed
  1dc0c6d

• chore: add updated attribute variables to config files, based on top 5 variables considered by Bolotin et al 2022 SI work

  glitt13

  committed
  f6ccc91

Commits on Sep 2, 2024

• fix: add error handling when hydrofabric could not be downloaded for a given comid

  glitt13

  committed
  5137b99

• fix: avoid index error generated from attr_ddf_sub.shape[0].compute() by simply performing attr_ddf_sub.compute() first, which is needed anyway

  glitt13

  committed
  c91e25f

• fix: change fs_read_attr_comid to return pd.DataFrame instead of dask df, and add checks ensuring 'value' data column being float type, check for no NA values present

  glitt13

  committed
  7d8b0f7

• feat: add NA drop prior to train/test split

  glitt13

  committed
  4743a47

• feat: create a separate function that standarizes the algorithm file save path

  glitt13

  committed
  15be337

• doc: add documentation to the std_algo_path func

  glitt13

  committed
  b4484aa

Commits on Sep 3, 2024

• Merge pull request #1 from glitt13/train_algo

  Show description for 49b878e
  glitt13

  authored
  49b878e

• feat: create script to generate algo prediction data for testing

  glitt13

  committed
  22835f5

• feat: generating predictions from trained algos under dev

  glitt13

  committed
  346f668

• feat: add processing of xssa locations, randomly selecting a subset to use for algo prediction

  glitt13

  committed
  b6723fd

Commits on Sep 4, 2024

• feat: develop algo prediction's config ingest, and determine paths to prediction locations and trained algos

  glitt13

  committed
  66f51dc

• feat: add config file path builder

  glitt13

  committed
  becfc45

• fix: resolve merge conflict

  glitt13

  committed
  10d2422

• feat: create metric prediction and write results to file

  glitt13

  committed
  8fe0a7c

Commits on Sep 5, 2024

• feat: build unit test for build_cfig_path()

  glitt13

  committed
  0fffc45

• feat: build unit test for build_cfig_path()

  glitt13

  committed
  62ff9aa

• feat: add unit testsfor std_pred_path and _read_pred_comid; test coverage now at 92%

  glitt13

  committed
  42edc50

• feat: add oob = True as default for RandomForestRegressor

  glitt13

  committed
  f07b9c9

Commits on Sep 6, 2024

• feat: add hyperparameterization capability using grid search and associated unit tests

  glitt13

  committed
  bae5175

• feat: add unit testing for train_eval()

  glitt13

  committed
  90c1443

• chore: change algo config for testing out hyperparameterization

  glitt13

  committed
  5adb43b

• chore: add UserWarning category specification to warnings.warn

  glitt13

  committed
  b0c3ef2

• fix: algo config assignment accidentally only looked at first line of params

  glitt13

  committed
  3abfb08

• fix: make sure that hyperparameter key:value pairings contained inside dict, not list

  glitt13

  committed
  c7de9ae

• fix: adjust unit test's algo_config formats to represent the issue of a dict of a list, which the list_to_dict() function then converts

  glitt13

  committed
  3e60519

• fix: _check_attributes_exist now appropriately reports missing attributes and comids

  glitt13

  committed
  b4034e8

• fix: ensure algo and pipeline keys contain algo and pipeline object types in the grid search case

  glitt13

  committed
  7e982d2

Commits on Sep 16, 2024

• Update pkg/fs_algo/fs_algo/fs_algo_train_eval.py

  Show description for 5d368c0
  

  glitt13

  and

  bolotinl

  authored
  5d368c0

• Update pkg/fs_algo/fs_algo/fs_algo_train_eval.py

  Show description for e34640a
  

  glitt13

  and

  bolotinl

  authored
  e34640a

• feat: merge the algo prediction features from the pred_alg branch

  glitt13

  committed
  f70f4d3

• chore: Update README.md

  Show description for bd42892
  glitt13

  authored
  bd42892

Commits on Sep 30, 2024

• merge upstream/main

  glitt13

  committed
  c7dddde

Commits on Oct 2, 2024

• fix: remove network hardcoding for lyrs in proc_attr_wrap call

  glitt13

  committed
  0080556

Commits on Oct 4, 2024

• fix: rename ext to fileext since ext is a pre-defined object

  glitt13

  committed
  1d5b4e2

• fix: change unit test use of ext to fileext

  glitt13

  committed
  88cd39a

• feat: experimenting with attribute grabbing

  glitt13

  committed
  292e949

Commits on Oct 13, 2024

• doc: revise function documentation for clarity

  glitt13

  committed
  3717fbc

• chore: rename fsds to fs in all python-related files and config files

  glitt13

  committed
  b607501

• chore: rename fsds_proc directory to fs_proc

  glitt13

  committed
  4df6fed

• chore: rename additional fsds to fs

  glitt13

  committed
  202e338

• chore: rename remaining fsds to fs

  glitt13

  committed
  f3c9e9c

• doc: minor change to install instructions of fs_proc

  glitt13

  committed
  3410c21

• feat: add requirements for fs_algo package

  glitt13

  committed
  41717b9

• feat: add requirements.yml for conda environment of fs_algo/fs_proc python packages

  glitt13

  committed
  8bc5bf1

• Merge pull request #2 from glitt13/fs_id

  Show description for 05b23f6
  glitt13

  authored
  05b23f6

Commits on Oct 14, 2024

• doc: add details on func for creating col_schema_df

  glitt13

  committed
  7b29342

• feat: add nwissite gage id leading zero checker as automated step

  glitt13

  committed
  20b5a6c

• fix: new line continuation in f-string messages related to nwis checker

  glitt13

  committed
  4dccd31

• fix: update local config path and example in script

  glitt13

  committed
  e80175a

• doc: change install description for this package

  glitt13

  committed
  2e64ae2

• fix: modify logical test on elif featureSource == nwissite

  glitt13

  committed
  9b048e6

• feat: update and add new unit testing that accommodates the check_fix_nwissite_gageids function

  glitt13

  committed
  3dbb9b9

• Merge pull request #3 from glitt13/fs_id

  Show description for a1e6acd
  glitt13

  authored
  a1e6acd

• fix: update temp directory assignment to work with non-Unix systems

  glitt13

  committed
  dfd661b

• doc: minor adjustment for instructional example on running unit tests

  glitt13

  committed
  70792db

• Merge pull request #4 from glitt13/fs_id

  Show description for 6cdfa4d
  glitt13

  authored
  6cdfa4d

Commits on Oct 16, 2024

• Make the change match the exact repo name

  bolotinl

  authored
  e757599

• Make changes match exact repo name

  bolotinl

  authored
  397fbcc

• doc: minor changes that will be removed: comid loc lookup

  glitt13

  committed
  c46ff89

• merge: upstream/main merge

  glitt13

  committed
  1f9718c

• fix: rename fsds to fs in files corresponding to proc.attr.hydfab R package

  glitt13

  committed
  71c0b14

• feat: update R package with name change of fsds to fs

  glitt13

  committed
  d916a03

• chore: update fsds to fs in config files and R unit tests

  glitt13

  committed
  8943877

• doc: update README from fsds to fs in non-url instances

  glitt13

  committed
  3522370

• doc: Update README.md

  Show description for 6427c1a
  glitt13

  authored
  6427c1a

• Update README.md

  Show description for 3a07b0b
  glitt13

  authored
  3a07b0b

• merge upstream/main

  glitt13

  committed
  702654e

• merge: upstream/main

  glitt13

  committed
  9afb17a

• merge origin/main

  glitt13

  committed
  217b53e

• merge: fsds to fs changes

  glitt13

  committed
  ecd1a7e

• chore: rename fsds_attrs_grab.R to fs_attrs_grab.R and add updated Rd documentation using fs instead of fsds

  glitt13

  committed
  61368a2

• merge: sync up with fsds to fs changes

  glitt13

  committed
  0eb76ee

Commits on Oct 17, 2024

• doc: update arg name change of ext to fileext

  glitt13

  committed
  97902d6

• doc: remove commented out code and create delineations on code sections

  glitt13

  committed
  07d00ca

Commits on Oct 23, 2024

• Merge branch 'main' into hfab

  bolotinl

  authored
  0fa4d55

Commits on Oct 24, 2024

• doc: correct mis-spellings

  glitt13

  committed
  0be6ab2