How do I import moe.easy_interface.experiment into my python source code, if I install MOE by docker?

[ksxh]

I want to use MOE package in my python program, but I am not an expert in docker. I tried doing a full installation of MOE on my ubuntu, but it fails.

I can install MOE by docker and the webserver and REST interface can run on localhost.

But how do I import moe.easy_interface.experiment into my python source code? I am trying to replicate the MOE examples given.

Thanks for your help

[suntzu86]

Re local build: what version of gcc, python, ubuntu? What error msgs do you get? On newer versions of ubuntu, sometimes our python-finding logic doesn't work... commenting out lines 191 and 192 in setup.py might help.

Re easy_interface: not sure what you mean. Do you get a code error? Does it just not import? Try importing (put a script or run python interactive) from the MOE root directory; does that work? Probably MOE isn't on your PYTHONPATH.

[ksxh]

Thanks for your tips, I managed to use MOE in my python codes. The steps are pretty messy, and I have to change a line of MOE codes to get it work. I don't know if the necessity to change this line is due to some bug in my codes or in MOE's. It is a weird bug. I am listing out my steps here, hopefully you can explain what happen. =)

1. Before doing a full installation of MOE on ubuntu, I updated the packages versions in requirements,txt, with respect to the newer versions that are installed in my ubuntu. If I don't do this step, executing full installation of MOE will mess up my python, by downgrading my packages to older versions.

2. Follow the full installation instructions

3. Add to ~/.bashrc MOE’s path to export PYTHONPATH

4. To run MOE in my python code, I need to change MOE domain.py

   class TensorProductDomain(DomainInterface):
       def __init__(self, domain_bounds):
   
           self._domain_bounds = [ClosedInterval(bound[0], bound[1]) for bound in domain_bounds]
           # doesn't work, have to change to as above
           #self._domain_bounds = copy.deepcopy(domain_bounds)
   

Deepcopy fails to copy the functions of ClosedInterval, which baffles me.

[suntzu86]

A few comments below. Glad you got it working!

1: This is why we suggest using virtual environments. If you aren't familiar w/them, it's a little bit of learning up front but it is SO convenient. Otherwise it's virtually impossible to keep multiple different python projects going simultaneously b/c of what you observed.

That said MOE's requirements could be bumped.

3: Yeah this is hard to avoid unless easy_interface were to be broken out into its own package, I think? I'm honestly not sure what best practices are here although I was hesitant to automatically change peoples' PYTHONPATHs.

4: I have no idea on that one... I've never seen that happen (and I've definitely run/tested that code). What version of Python are you on?
Do you still have the failure message from this?

[ksxh]

@suntzu86 I tried using virtual environment for MOE, and my system is using Anaconda python 2.7. But I do not know how to import MOE into my system, since virtual environment is supposed to be separated from the main environment?

Here is the error message

File "/home/ksxh/.eclipse/org.eclipse.platform_4.4.2_1473617060_linux_gtk_x8
6_64/plugins/org.python.pydev_3.9.2.201502050007/pysrc/pydevd_exec.py", line 3,
in Exec
exec exp in global_vars, local_vars

File "", line 1, in

File "/media/ksxh/DATA/application_1/source_code/src/FFM/FFM.py", line 149, in test
exp = Experiment([[0, 2], [0, 4]])

File "/home/ksxh/anaconda/lib/python2.7/site-packages/moe/easy_interface/exp
eriment.py", line 29, in init
self.domain = TensorProductDomain(_domain_bounds)

File "/home/ksxh/anaconda/lib/python2.7/site-packages/moe/optimal_learning/p
ython/python_version/domain.py", line 45, in init
if interval.is_empty():
AttributeError: 'ClosedInterval' object has no attribute 'is_empty'

[suntzu86]

re virtual env: So the steps would be:

1. create new virtual env (put it in MOE folder or name it something MOE-related)
2. launch virtual env (source your_venv/bin/activate or use a wrapper tool like virtualenvwrapper)
3. install requirements.txt for moe
4. run moe's setup.py
   (You may need to set PYTHONPATH here.)
   Now your venv is set up.

In the future, when you want to do MOE stuff, you first launch the virtualenv, then you can use python interactive mode or run your scripts that import MOE.

Re error message:
I see. I'm pretty sure what happened here is that the version of gcc used to build boost and the version of gcc you're building MOE with are different. The most likely case for this is you're on ubuntu 15.10 or later, and trying to use gcc4.9 or earlier. This would mean the boost installed by apt-get is built with gcc5 (default on 15.10 and later). gcc is not abi compatible across major versions, so you cannot mix them. I've seen the exact error you have before under these circumstances, so let's check:

What version of gcc are you using to build MOE?
What is the default version of gcc for your system? (Or what version of Ubuntu are you on?)
What version of Boost are you using? (in /usr/lib/libboost*, what version do you see)
Where does libboost_python.so link to? (We also will have problems if it isn't a *-py27 version)

extra detail: ClosedInterval is a C++ object that is made accessible to Python by libboost_python. When you mix & match compilers & libraries, apparently this function doesn't get imported correctly, leaving ClosedInterval in a broken state in Python (the C++ version is just fine).

[ksxh]

@suntzu86 Thank you for your help.

I don't think I can use MOE in virtual environment, as I am using a separate python interpreter to run my system, and I am importing MOE as a module in my system.

Alternatively, I think I can setup MOE in virtual environment and use the REST interface, but I am not familiar with it. Will you guys provide examples on using REST interface?

Here are the details of my settings

• gcc (Ubuntu 4.9.2-10ubuntu13) 4.9.2. I think I used this to build MOE
• Ubuntu 15.04
• boost Version: 1.55.0.2
• I do not know how to check where libboost_python.so is linking to

[suntzu86]

Re virtualenv: Not sure what you mean by using a separate python interpreter. You should be building MOE against the libpython that backs your python interpreter (this could also be the source of the next error) if you aren't.

But again, virtualenv does not prevent you from using whatever python interpreter that you want.

Re REST: <moe_root_dir>/moe_examples shows using MOE through moe/easy_interface which in turn just sends POST/GET requests to a MOE server. The intro docs go through how to set up MOE server locally in a docker container.

Re errors: Interesting. I don't have 15.04 installed anymore, but when I did, I never ran into this problem (with the system installed gcc and boost). I tried the system boost against gcc 4.7, 4.8, and 4.9 with no issues so I'm really not sure.

ls -al in the directory where that file is will show the destination of soft-links. ls -al <target> to a specific expression will show just those file(s).

You might try installing gcc 4.7 or gcc4.8 to and build MOE with those to see if that helps (both worked for me as well). One way is to use update-alternatives:
http://askubuntu.com/a/313313

You can also building boost manually. The MOE docs go over how to do this (it's necessary on OSX). I'd go with 1.51 or 1.53... never had problems with gcc47 or 48 and those boost versions before.

[ksxh]

@suntzu86 Thank you for the advice. Ubuntu 16.04 is releasing soon, I will try setting up MOE in it and tell you if it works.

I also realise cmake version 2 doesn't work for me, I need to use cmake version 3.0.2

Can I ask, if I setup MOE in docker, how do I use it in my python program? For example, can I run http://yelp.github.io/MOE/_modules/moe_examples/combined_example.html#run_example? If it is possible, how do I 'import' this docker MOE into my python program?

[suntzu86]

I was able to compile and run in 15.10 with gcc5. Some tests were failing it seems (everything looked to still work, but those tests had too tight constraints before and gcc must have substantially changed the order of floating point operations)

cmake v3 is fine. MOE needs one of the later versions of cmake 2; I don't remember exactly which but it is documented in the CMakeLists.txt file.

MOE has a web server providing a REST API. You can launch it via pserve (see docs). Alternatively, building the docker container fires up the web server automatically (it's configurable what host/port it's on). I personally use it almost exclusively through the python (or C++) interface but some people find the API easier.

easy_interface is just a little wrapper for writing HTTP POST and GET requests that are sent to the web server. You may have to set the default host/port values to the ones your docker is using. Not all of MOE's functionality is available through easy_interface, so you may need to write additional wrappers yourself.

So you don't need the docker to run combined_example, but you do need the web server running. The docker does this for you.

[cancan101]

Related: #461