📑 Feature Request: Extensions for updated test script

[kurysauce]

Describe your feature request.

After developing the updated test.py command for Ersilia #1203 , I believe having a detailed report for model contributors and users would be helpful in debugging and understanding the models in context.

The testing report summary, along with including the metrics measured in the test.py command should also include the following (not an exhaustive list):

• Inuputs that were tested, inputs that raised specific errors
• Software Dependencies: List of all libraries and tools used, including versions
  • Espeically helpful if users want to run and download the models locally
  • Warnings if models don't work on certain operating systems (MacOS, Windows, etc...)
• Performance Tests:
  • metrics such as speed, memory usage
• Output checks, If errors are raised or ouputs are invalid, describe them in context:
  • Missing Values
  • Outliers
  • Formatting Errors

[kurysauce]

• Support the method check_consistent_output with checking more Output Shape types (as mentioned on the Ersilia Model Template Book)
  • Current type check supported are Single,List. Needs support for Flexible List, Matrix, Serializable Object
  • From a meeting with @miquelduranfrigola , we need to incorporate models that support this output shape type

[miquelduranfrigola]

Thanks @kurysauce
@DhanshreeA to bring you up to speed, this is not something we need to tackle immediately but it is a nice feature request from @kurysauce . Let's leave it in out to-do list for now.

[kurysauce]

Update check_consistent_output and comparison between bash & Ersilia run in run_bash methods to allow comparisons for data structure containing different output types. Current method only supports comparisons between outputs that are all numerical values and all strings.

[miquelduranfrigola]

This makes sense. Thanks @kurysauce

[GemmaTuron]

Hi @DhanshreeA and @miquelduranfrigola

I am marking this as "on-hold" until we have a clear roadmap for Ersilia CLI commands

[GemmaTuron]

Hi @DhanshreeA

@Abellegese has been working on the test script right? so is this on hold still or undergoing? If so, which issues from the original list have been tackled? Thanks

[Abellegese]

Yes @GemmaTuron this addressed in depth in new test pipeline including the playground.

Currently it does not give list of packages with version. Thats only it, if its important will update to do that.

[GemmaTuron]

Hi @Abellegese

Thanks, please give more details and/or a link to the documentation before we can close this issue. I need more info to understand the pipeline.

[Abellegese]

Hi @GemmaTuron yes I have a doc for playground. For model tester, I will upload it and let you know.

[Abellegese]

I do have the docs, revise a bit and upload it to gitbook.

[Abellegese]

So this issue also has been addressed in the following cases:

1. Pytest for five ersilia CLI and all the associated function level testing using partial mocking.
   • Issue: 🐕 Batch: A more comprehensive PyTest pipeline #1368
   • PR: Pytest CLI test using partial mocking #1397
2. Playground test: to test model behaviour using custom rules
   • Issue: 🐕 Batch: A more comprehensive PyTest pipeline #1368
   • PR: CLI test integration #1383
3. Model Tester: To test and remote and local models
   • PR: Model tester and Inspect command functionality updates #1416

[Abellegese]

Hi @GemmaTuron the docs for model tester has been uploaded. You may check it.