Guess / Not Guess optimization

[papipopapu]

I dont know if this is even possible, so i marked this as an idea.

I am working in a binary classification model, and I was wondering if there was a way to change the modelling process so that the model could sometimes just "not guess". I mean, it would be great that the model could fit perfectly the ground truth predictions, but since in my case that is not possible (a lot of noise), it would be much better if the model could distinguish the instances where its probably not going to get it right, and those where its more likely.
For the model to be optimized in this way I thought the way to go was to change the loss function so it depenalises to some degree "not guessing". Maybe it is neccessary to turn it somehow into a multiclass classification problem, where "not guessing" is an option, but I dont know how would I go about it since I alredy have the ground truth perfect binary predictions.

I tried doing some research, but I dont even know what to search for, so I came here looking for your input, since you have proven to be so helpful to us newbies. Thank you!

[mrdbourke]

Hey there,

You might want to look into a few different things:

• Finding the most wrong examples with a given output (e.g. the outputs which a model predicts as label y_1 with a high prediction probability but it should've been y_2) - https://dev.mrdbourke.com/tensorflow-deep-learning/08_introduction_to_nlp_in_tensorflow/#finding-the-most-wrong-examples
• Out of distribution detection (e.g. identify samples which aren't in a model's known distribution or the patterns its already learned) - https://medium.com/analytics-vidhya/out-of-distribution-detection-in-deep-neural-networks-450da9ed7044

The second one is a big research topic so I'd start by searching "out of distribution detection in machine learning" and go from there.

Though perhaps I've interpreted your question incorrectly.

If so, please let me know.

PS if you have an example of the ideal inputs and outputs of your model that would be great.

[papipopapu]

Yes, thats what I was looking for, somwhere to start searching. Great. Right now I dont have access to my project files, but ideally the input would be a matrix with different information in each column, and I'd like the model to classify each row ouptuting either 1,0 (binnary classification) or "not guess", which maybe could be -1, while it is being trained only with the ground truth 1's and 0's (I dont know when "not guessing" would be the right choice, it would be up to the model optimization or I would have to arbitrarily change the training output in some way).

I will look into all the resources you gave, thanks!