-
Notifications
You must be signed in to change notification settings - Fork 77
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add QAQC auto reporting tool #184
Comments
@bemcdonnell Good ideas, it might be also good to have the min and max of all elements. It also needs better example files, the bulk of the sample files are from Extran3 and SWMM4 converted to SWMM5. The sample files, IMHO, should have more than a dozen elements. It would be nice to have sample file with everything possible in SWMM5 as a hundred nodes, links and Subcatchments so that it at least looks like a decent sample model. |
@dickinsonre, for previous "official" releases of SWMM, how many benchmark models were used? And what was the process for cherry-picking results to present in the SWMM users manual? |
@bemcdonnell We did have about 700 test models for SWMM 5. You can see most of them on my Box account (I just made you and editor). However, most of the files on Box suffer from the same problems as these test files (the user example files do look like real networks as they came from real networks). Cherry picking, i don't know how Lew picked them but they seem to be most of the Extran3/4 files along with some PCSWMM files. I used the PCSWMM files for SWMM4 testing back in the day. I just checked and we about 200 test models for SWMM4, 130 hydraulics, 70 Hydrology and Water Quality. SWMM4 also had a nice Graph Block that compared the statistics of the model simulation for selected flows and depths to a user data file. Lew added may more calibration graphs but we lost the nice Statistical comparison possible in SWMM4. It is a good idea to use Python to add that back to the software for testing purposes. Example 1: pass |
Hello @bemcdonnell just to follow up on your cherry picking comment. I went to the QA report on the EPA Download site and converted it to a DOC file. The testing Lew Rossman did in the really nice STORM WATER MANAGEMENT MODEL QUALITY ASSURANCE REPORT: Dynamic Wave Flow Routing
|
@dickinsonre, excellent info! Thanks for diving into that! |
@bemcdonnell My pleasure, however, it looks like a lot of work done by Lew to make that document. It will be nice to have a complete automated testing with stats for every node, link, lid, gw, subcatchment etc etc. that can be saved as log files for future backwards and forwards testing. |
@DickensonRE cool idea! |
What if we develop this then morphed it into a feature. Like a calibration report that SWMM users could run as part of their QA. That would save Engineers tons of grunt work. Perhaps help them to be less fixated on changing model results. |
Link for Nash-Sutcliffe, Logarithmic Nash-Sutcliffe |
I have a conceptual question @bemcdonnell and @michaeltryby say you had a NS value for each node, link and subcatchment. If you have a large model this is still many numbers to view and absorb. How can one come up with one overall index of model comparison? A mean of the NS values even though the node depths and link flows while related are from two different sections of the code? Report the mean node, link and subcatchment NS value? |
@dickinsonre, could always report the top offenders above some threshold in a table as well as MEAN and STD? |
Okay, for some reason your @bemcdonnell cherry picking comment got stuck in my head! We have implemented comparison tests for all nodes, links and subcatchments. I will update my dickinsonre GitHub soon after I learn more about GIT. My son was complaining today I am using Git wrong. Thanks for your patience in this matter. |
As we work toward making releases, I believe it would be valuable to have a tool that automatically runs the benchmarks and plots/docs the results for the new version. It doesn’t need to be exhaustive for all objects / results - maybe the top differences.
What this tool should use:
PyLaTex
What we should show:
The text was updated successfully, but these errors were encountered: