Model testing

[GemmaTuron]

We need to better define what we mean by "Test a model" and provide example files etc, and a checklist so that contributors can follow step by step.
What testing would you like to see for each model?

[ZakiaYahya]

Hi @GemmaTuron

(1) Firstly, I think we should define a file like ersilia did for eml_canonical.csv, there should be a file named e.g. test_smiles.csv that contains atleast 10-20 smiles. So everyone use that smiles for testing.

(2) It's better to put one wrong smile string at the end of smiles to confirm the behaviour of model whether it is predicting right or wrong ( but i don't know how ersilia model behaves when a wrong input passes to it), May be for this we define some checks while reading smiles from the file test_smiles.csv that if wrong input pass, ignore it and put NAN in output.

[GemmaTuron]

yes, that is a good suggestion.
Perhaps we should have two csv files, one with all correct (s0 we first check that the modle works) and then one with some incorrect ones

[ZakiaYahya]

Yes, we can do this as well, testing two files but it will time taking then, i mean for one model testing, you have to run model two times on CLI, two times on Colab and two times on DockerHub at prediction time, Right @GemmaTuron ??

[GemmaTuron]

yes, you need to double the tests, but if you pass an incorrect smiles from the beginning it might be the error is somewhere else and you are not seeing it, so I'd go for two files

[ZakiaYahya]

Oh right, yeh for this we need to double test the files.

[GemmaTuron]

Hi @emmakodes, @samuelmaina @HellenNamulinda

Please give your views on this issue as well.

[HellenNamulinda]

This is late,
But from my experiences so far, Besides having separate files for correct and incorrect smiles,
I think the testing pipeline should be as follows;

1. Test on Colab. It's unlikely that one will make predictions for string inputs using Colab, but a file. Though the model will pass a list at run time. Since some models are big and slow at inference time, it would be better to have a file with a few molecules.
2. Test using Ersilia CLI. Here, the model should be tested for two types of inputs.
   - String inputs (like two molecules), and
   - file inputs. (for testing, 20 molecules per file should be okay)
3. For docker, it may take some time to pull the image. though should be tested with both inputs.

I'm insisting on two inputs for CLI because I have tested models that only work when given string inputs.

[emmakodes]

Hello @everyone, having two files to test a model is fair enough (one contains correct smiles and the other file include some incorrect smiles) but some models are pretty large and takes time. What I may suggest is we still have the two files but the number of smiles in each of the files can be reduced to say 5. These five smiles in each of the two files should be the standard and most common input to use test a model.

[miquelduranfrigola]

Thanks @emmakodes @emmakodes @HellenNamulinda @ZakiaYahya, this is very valuable, let me add a few extra thoughts:

Thoughts

I have written a placeholder at ersilia.publish.test, named LocalModelTester that should have multiple methods to test models before doing the PR. In particular, one could potentially evaluate the following:

• The metadata: we need to make sure that the information for the model is complete enough. This can be effectively done with a BaseInformation class, which can read the metadata.json file and will raise exceptions if something does not look good (for example, a wrong URL, or a description that is too short).
• Input-output consistency: this is basically what you all have been talking about. Note that we already have quite a few inputs to experiment with, especially in the ersilia-model-hub-maintained-inputs repositories. Also, please note that ersilia example is also a good command to get started with. In any case, I see many subcategories, here, for example:
  • Is the length of the input equal to the length of the output?
  • What happens if an input sample is incorrect (for example, a wrong SMILES)?
  • Does the model work with correct, but challenging, inputs, such as short or long SMILES?
  • What happens if the input is long? For example, >100 samples.
  • Does the model work with different input formats? (.json, .csv, .tsv)
• Output inspection: there is a lot of work that can be done on output serialization, and the earlier we do it the better. This would be a non-comprehensive list of ideas:
  • Are all outputs "labeled"? That is, do we have for example column/field names? If so, do these field names follow any convention or nomenclature? For example, I would like to see all field names in lowercase, using only alphanumeric characters and the - character. No ., _, $, etc.
• Is the output "shape" appropriate? For example, do I have an "array" of only one dimension? In such a case, shouldn't I be naming this "single" output?
• Are the output types correct? For example, if the output type is an integer, can I check that this is indeed the case in practice?
• Dockerization: this is related to the suggestion by @emmakodes and @leoank to test a priori whether a model work on linux/arm64
• Usage metrics: It would be nice to collect the following usage metrics. Perhaps this should be done in GitHub Actions instead of locally, but in any case:
  • Size of the model files and parameters: We can get this, eventually, from S3, I suppose, but we can also anticipate it locally.
  • Size of the docker image: We can get this, eventually, from DockerHub. So probably we won't be able to estimate it a priory.
  • Single point prediction time: measure how long it takes to do a single prediction
  • 10 predictions
  • 100 predictions (possibly, with a time cap)

Strategy

There is a lot of work to be done here, so let's go step by step. I would start with "Input-output consistency". @GemmaTuron , what do you think? In parallel, @emmakodes and @leoank can explore the AMD64/ARM64 point.

[samuelmaina]

Late to the party, sorry . Everyone has said pretty much what I would suggest. Thanks.