-
Notifications
You must be signed in to change notification settings - Fork 140
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unknown Source of Variation in gp_next_points()
#452
Comments
Sorry I didn't respond sooner... just started a new job so I've been pretty busy. Plus I don't have a working linux installation at the moment to test on, lol. I believe Branin should work b/c we have tested with it before. Although while it is a very commonly considered case in academic papers, it never seemed particularly similar to our present MOE applications so we stopped focusing on it. So this should work. That said, a few things stand out to me...
I'm not entirely sure what domain it thinks it's getting. My only guess could be that the domain is coming out as [1.0, 1.0] for all hyperparameters. (Because the C++ accepts the domain in log10, so after exponentiating, 10 ** 0 = 1.) But I really don't know... that should have failed validation, sorry :( Also, I didn't build or interact with the easyinterface stuff like at all, so please bear with me. Lastly, you mean that doing somethign like this: |
Answering your q's:
I still tried adding a print statement to moe/views/rest/gp_hyper_opt to get the value of Nonetheless, I refactored the
As a side note, it seems like I can actually add to experiments from the easy interface, so it doesn't seem like that is actually an issue anymore.
|
Your code edit needs to happen on the server (like if you're running in docker or something, it needs to be in the code running there; so relaunch after edits or log in to docker). But that function is definitely being called. It's the only "view" that handles hyperparameter optimization.
|
I see things like:
Not linux kernel. iPython Kernel. I don't think this is important because this isn't even where the code is executing; it's executing on Docker, as you pointed out.
I spent hours trying to print the value of (as a side note, #401 turned out to be a blocker to do code edits easily. I commented there too)
When I said, "the overall values are still not converging to any of the global minima," I meant the overall values of the branin function. I changed the line in my
to:
This didn't seem to have any impact.
I'm not sure how to verify whether I'm doing this or not, but I haven't explicitly or intentionally done this I don't think...
This is 50 iterations. For reference, the global optima of the branin are at: (-pi, 12.275), (pi, 2.275) and (9.42478, 2.475). It doesn't look like MOE is getting close to finding any of them. It's hard to understand the sequence of points explored in a scatter plot, so I tried to color the points so the earlier ones would be more red and the later ones more blue. If you run |
Steps if interested: Copy the Anyway, sorry I don't have a working install yet but I'll get there. Should have looked into gcc default version before upgrading to ubuntu 15.10. |
Also does this guy in https://github.com/Yelp/MOE/blob/master/moe_examples/combined_example.py may want to disable the noise in |
There are multiple minima (all equal to 0.39) in the branin function see desc, so this is not entirely straightforward
These are "steps"... to what? I didn't understand what this was for, but I did it anyway. I couldn't add
Then I added this to my
I don't know how best to summarize this. It will probably be best to run it yourself and look at the contents of
I'm on OS X Yosemite (10.10.5 ) and Also, I got If I can't turn the corner on this problem soon, I think I'm gonna have to abandon MOE :(
|
I can understand having to abandon MOE... I'd be annoyed too if I were stuck for so long :( I was hoping to spend some time on this over the weekend but I ended up having to spend the weekend working on things for my job. So I have a couple more suggestions for you; I'll try to debug it myself this week as hopefully work has settled down. Anyway...
Quick things to try:
Hopefully I spelled all those fields correctly, lol. Edit:
|
Thank you for helping out @suntzu86 . Have you been able to work on this? I am having a similar problem. |
Nope, I haven't had a chance to dive deeper into this. A team at Cornell where some of the original research for MOE was done put a paper up on arxiv recently: They run a branin case successfully, code here: It's not going through the REST API though, but you can hopefully see how the python objects map into the api's json fields (like TensorProductDomain -> |
I have been using MOE to tune parameters of a black box function. One thing I haven't been able to understand is why -- holding constant the number of historical points in an experiment -- consecutive calls to
gp_next_points()
will return points that seem to vary randomly within the X domain.After reading the EGO paper, this doesn't seem to make sense. As I understand it, the idea behind EI is that the suggested points at any given stage are supposed to be deterministic. That is, EI returns the X point where the GP thinks there will be the greatest improvement in the underlying objective function.
To get to the bottom of this, I've tried running MOE on a known function (the Branin function), removing all the usual sources of variation. Yet, I still see the randomness mentioned above. What's more, benchmarked against GPyOpt, MOE doesn't seem close to finding the answer even after 50 iterations.
Here's what I did. First, the imports outside of MOE:
Now, define the Branin Function:
Random points are generated in the domain described in the Branin description
Next, I add these points to a
moe.easy_interface.experiment.Experiment
. I included no variance at these points, as I wanted to see whether I could reproduce the problematic randomness by removing all input sources of variance. Variance can be added to these points with the same end result.Next, I cycle MOE for 50 iterations using the function defined below. I included a covariance re-tuning every 5 iterations (as suggested in the MOE FAQ). This function returns -- among other things -- a dataframe
rdf
that summarizes the suggested points at each iteration, as well as the function value at each of these points (with no variance, again in an effort to isolate the problematic randomness). Additionally, the function also returns the experiment itself** and latest covariance_info, along with some metadata regarding time taken for each covariance tuning (tdf
)**Side Note: There is some weird bug where I wasn't able to add new points to an experiment after I send it to the REST server, so as a workaround, I would just add the points to a new experiment at each iteration.
This function had two helper functions:
and
After running
exp
throughcycle_moe()
for 50 iterations, when I make consecutive calls with the following line:I still see random variation in the next points suggested, which I don't understand. Additionally, when examining the results of each iteration in
rdf
, it doesn't look like MOE is getting close to finding one of the global optima in the Branin descriptionI ran the same number of iterations using GPyOpt on the same function, using the same method ('EI'), and it converged to one of the global optima.
Hopefully, with all my code and each step outlined above, somebody can help me figure out exactly what I'm doing wrong here. But up to now, this has been a serious head scratcher...
The text was updated successfully, but these errors were encountered: