oops! synthetic data contains missing values

[lumbardhelshani]

I dont know why this error is happening, how can I avoid it?

[rayosclx]

Same here

[nickkunz]

Thank you for using SMOGN. Have you ensured that you have a valid pandas dataframe with no missing values?

[rayosclx]

Great idea implementing this, thanks.
Yes, absolutely, I explicitly dropna() all columns before hand. A call to df.isna().sum().sum() throws 0 (zero). Aren't t NaNs anyway dropped by default?
If I choose just the first 4000 rows from the df, as opposed to the full 15000, smogn runs fine.
I am yet to find what is exactly in the df it doesn't like.
Update: choosing samp_method to 'extreme' and rel_xtrm_type to 'high' makes the method run without errors over the same df. If I change the rel_thress from the default to 0.3 though, it sends the same error. It definitely isn't a dataset problem.

[philastrophist]

This only occurs for me in a specific circumstance:

import smogn
import pandas as pd
df = pd.DataFrame(pt.transform_x(X_train), columns=labels)
df['logl'] = pt.transform_y(y_train)
oversampled_train = smogn.smoter(data = df.iloc[:N], y = 'logl', samp_method = 'extreme')

When N is 1000, I get that error, when N=1001 it works....

This happens for different values of N dependent on the choice of parameters

[AcylSilane]

I'm still not sure what causes it (it seems to happen intermittently when I switch samp_method from 'extreme' to 'balanced'), but it's seemingly-random (in a dataset with ~300 columns and ~150 rows, no missing entries). I've had cases where I've run the exact same code twice, and one time it failed, the other time it didn't.

Here's a bit of an ugly workaround. Given the seemingly-random nature of it (I've never seen it happen more than twice in a row), you could wrap the smoter in a while block like this:

n_tries = 0
done = False
while not done:
    try:
        smogn.smoter(data=foo, y="bar")
        done = True

    except ValueError:
        if n_tries < 5:
            n_tries += 1
        else:
            raise

That said, keep in mind that this will still fail if there really is something wrong with the dataset you're feeding smogn. This only helps when things fail seemingly-randomly.

[thl456]

I encountered this error too but had verified that the dataframe fed to smogn.smoter did not have any missing values hence not sure why this error appears. Any solution to pass over the error? Moreover, when the data > 2000 rows, the distmatrix is very slow, any way to speed things up?