Flaky Test when seed is removed

[crawlingcub]

The test test_boptim in test/test_boptim.py is flaky when the seed is removed. The test failed 500 out of 500 times that I ran.

Interestingly, the test always only fails for the first parameter ("ei") and passes for the other two: "poi" and "cb". I observed that this is true even if I change the order of the parameters i.e., whatever comes first fails! Error message below.

Do you know why this happens or how can this be resolved? I will be happy to raise a PR if you have any suggestions!

Thanks!

Error message:

_________________________________________________________ test_boptim[ei-/home/saikat/projects/borntobeflaky/projects/GPim/test/test_data/test_ei.npy] _________________________________________________________

acqf = 'ei', result = '/home/saikat/projects/borntobeflaky/projects/GPim/test/test_data/test_ei.npy'

    @pytest.mark.parametrize(
        "acqf, result",
        [("ei", test_img_ei),
    #     ("poi", test_img_poi),
    #     ("cb", test_img_cb)
        ])
    def test_boptim(acqf, result):
        Z_sparse = initial_seed()
        X_full = gprutils.get_full_grid(Z_sparse)
        X_sparse = gprutils.get_sparse_grid(Z_sparse)
        expected_result = np.load(result)
        boptim = boptimizer(
            X_sparse, Z_sparse, X_full,
            trial_func, acquisition_function=acqf,
            exploration_steps=20,
            use_gpu=False, verbose=1)
        boptim.run()
        #print(expected_result)
>       assert_allclose(boptim.target_func_vals[-1], expected_result)

test/test_boptim.py:60:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _  
x = array([[3.69114071e-08,            nan,            nan,            nan,
                   nan,            nan,       ... nan,            nan,
                   nan,            nan,            nan,            nan,
        7.57299443e-34]])
y = array([[3.69114071e-08,            nan,            nan,            nan,
                   nan,            nan, 3.6477... nan,            nan,
                   nan,            nan,            nan,            nan,
        7.57299443e-34]])

func = <ufunc 'isnan'>, hasval = 'nan'

    def func_assert_same_pos(x, y, func=isnan, hasval='nan'):
        """Handling nan/inf.

        Combine results of running func on x and y, checking that they are True
        at the same locations.

        """
        x_id = func(x)
        y_id = func(y)
        # We include work-arounds here to handle three types of slightly
        # pathological ndarray subclasses:
        # (1) all() on `masked` array scalars can return masked arrays, so we
        #     use != True
        # (2) __eq__ on some ndarray subclasses returns Python booleans
        #     instead of element-wise comparisons, so we cast to bool_() and
        #     use isinstance(..., bool) checks
        # (3) subclasses with bare-bones __array_function__ implementations may
        #     not implement np.all(), so favor using the .all() method
        # We are not committed to supporting such subclasses, but it's nice to
        # support them if possible.
        if bool_(x_id == y_id).all() != True:                                                                                                                                                                                msg = build_err_msg([x, y],
                                err_msg + '\nx and y %s location mismatch:'
                                % (hasval), verbose=verbose, header=header,
                                names=('x', 'y'), precision=precision)
>           raise AssertionError(msg)
E           AssertionError:
E           Not equal to tolerance rtol=1e-07, atol=0
E
E           x and y nan location mismatch:
E            x: array([[3.691141e-08,          nan,          nan,          nan,
E                            nan,          nan,          nan,          nan,
E                   1.886172e-02,          nan,          nan, 2.844261e-02,...
E            y: array([[3.691141e-08,          nan,          nan,          nan,
E                            nan,          nan, 3.647780e-03,          nan,
E                            nan, 2.844261e-02,          nan,          nan,...

../../../../anaconda3/envs/GPim/lib/python3.6/site-packages/numpy/testing/_private/utils.py:740: AssertionError