[BibTeX] cite as:
Nikolaus Hansen, Dimo Brockhoff, Olaf Mersmann, Tea Tusar, Dejan Tusar, Ouassim Ait ElHara, Phillipe R. Sampaio, Asma Atamna, Konstantinos Varelas, Umut Batu, Duc Manh Nguyen, Filip Matzner, Anne Auger. COmparing Continuous Optimizers: numbbo/COCO on Github. Zenodo, DOI:10.5281/zenodo.2594848, March 2019.
This code reimplements the original Comparing
Continous Optimizer platform, now rewritten fully in ANSI C
and Python
with
other languages calling the C
code. As the name suggests, the code provides a
platform to benchmark and compare continuous optimizers, AKA non-linear solvers
for numerical optimization. Languages currently available are
C/C++
Java
MATLAB/Octave
Python
Contributions to link further languages (including a better
example in C++
) are more than welcome.
The general project structure is shown in the following figure where the black color indicates code or data provided by the platform and the red color indicates either user code or data and graphical output from using the platform:
For more general information:
- read our benchmarking guidelines introduction
- read the COCO experimental setup description
- see the
bbob-biobj
andbbob-biobj-ext
COCO multi-objective functions testbed documentation and the specificities of the performance assessment for the bi-objective testbeds. - consult the BBOB workshops series,
- consider to register here for news,
- see the previous COCO home page here and
- see the links below to learn more about the ideas behind CoCO.
- For a machine running experiments
- A
C
compiler, such as gcc - Python >=2.6 with
setuptools
installed - optional:
git
- For a machine displaying data by running the post-processing
- Python 3 with
numpy
,scipy
,matplotlib
, andsix
installed. We recommend to install the Anaconda Python library
For Ubuntu 16.04+, all the requirements can be installed using the following command:
apt-get install build-essential python-dev python-numpy python-matplotlib \
python-scipy python-six python-setuptools
Under Windows, two alternative compile toolchains can be used:
- Cygwin which comes with gcc and make, available in 32- and 64-bit versions.
- MinGW's gcc (http://www.mingw.org/ for 32-bit or https://mingw-w64.org for 64-bit machines). Make sure to update the Windows path to MinGW's make.exe and rename/link the gcc.exe to cc.exe.
For using git
under Windows (optional), we recommend installing TortoiseGit.
Additional requirements for running an algorithm in a specific language.
- C:
make
, such as GNU make (when using GNU make for Windows, make sure that yourCC
environment variable is set togcc
by potentially typingset CC=gcc
if you see an error). - Java:
gcc
and any Java Development Kit (JDK), such thatjavac
andjavah
are accessible (i.e. in the system path). - MATLAB: at least MATLAB 2008, for details, see here
- Python on Windows with MinGW: Python 2.7 and the Microsoft compiler package for Python 2.7
containing VC9, available here.
These are necessary to build the C extensions for the Python
cocoex
module for Windows. The package contains 32-bit and 64-bit compilers and the Windows SDK headers. - Python on Linux:
python-dev
must be installed to compile/install thecocoex
module. - Octave: Octave 4.0.0 or later. On operating systems other than Windows, earlier versions might work.
Under Linux the package
liboctave-dev
might be necessary.
We tested the framework on Mac OSX, Ubuntu linux, Fedora linux, and Windows (XP, 7, 10) in various combinations of 32-bit and 64-bit compilers, python versions etc. Naturally, we cannot guarantee that the framework runs on any combination of operating system and software installed. In case you experience some incompatibilies, check out the Known Issues / Trouble Shooting Section below. Otherwise we will be happy if you can document them in detail on the issue tracker.
-
Check out the Requirements above.
-
Install the post-processing for displaying data (using Python):
pip install cocopp
As long as no experiments are meant to be run, the next points 2.-6. can be skipped and continue with points 7. and 8. below.
-
Download the COCO framework code from github,
- either by clicking the Download ZIP button
and unzip the
zip
file, - or by typing
git clone https://github.com/numbbo/coco.git
. This way allows to remain up-to-date easily (but needsgit
to be installed). After cloning,git pull
keeps the code up-to-date with the latest release.
The record of official releases can be found here. The latest release corresponds to the master branch as linked above.
- either by clicking the Download ZIP button
and unzip the
-
In a system shell,
cd
into thecoco
orcoco-<version>
folder (framework root), where the filedo.py
can be found. Type, i.e. execute, one of the following commands oncepython do.py run-c python do.py run-java python do.py run-matlab python do.py run-octave python do.py run-python
depending on which language shall be used to run the experiments.
run-*
will build the respective code and run the example experiment once. The build result and the example experiment code can be found undercode-experiments/build/<language>
(<language>=matlab
for Octave).python do.py
lists all available commands. -
Copy the folder
code-experiments/build/YOUR-FAVORITE-LANGUAGE
and its content to another location. In Python it is sufficient to copy the fileexample_experiment_for_beginners.py
orexample_experiment2.py
. Run the example experiment (it already is compiled). As the details vary, see the respective read-me's and/or example experiment files:C
read me and example experimentJava
read me and example experimentMatlab/Octave
read me and example experimentPython
read me and example experiment2
If the example experiment runs, connect your favorite algorithm to Coco: replace the call to the random search optimizer in the example experiment file by a call to your algorithm (see above). Update the output
result_folder
, thealgorithm_name
andalgorithm_info
of the observer options in the example experiment file.Another entry point for your own experiments can be the
code-experiments/examples
folder. -
Now you can run your favorite algorithm on the
bbob
andbbob-largescale
suites (for single-objective algorithms), on thebbob-biobj
suite (for multi-objective algorithms), or on the mixed-integer suites (bbob-mixint
andbbob-biobj-mixint
respectively). Output is automatically generated in the specified dataresult_folder
. By now, more suites might be available, see below. -
Postprocess the data from the results folder by typing
python -m cocopp [-o OUTPUT_FOLDERNAME] YOURDATAFOLDER [MORE_DATAFOLDERS]
Any subfolder in the folder arguments will be searched for logged data. That is, experiments from different batches can be in different folders collected under a single "root"
YOURDATAFOLDER
folder. We can also compare more than one algorithm by specifying several data result folders generated by different algorithms. -
We also provide many archived algorithm data sets. For example
python -m cocopp 'bbob/2009/BFGS_ros' 'bbob/2010/IPOP-ACTCMA'
processes the referenced archived BFGS data set and an IPOP-CMA data set. The given substring must have a unique match in the archive or must end with
!
or*
or must be a regular expression containing a*
and not ending with!
or*
. Otherwise, all matches are listed but none is processed with this call. For more information in how to obtain and display specific archived data, seehelp(cocopp)
orhelp(cocopp.archives)
or the classCOCODataArchive
.Data descriptions can be found for the
bbob
test suite at coco-algorithms and for thebbob-biobj
test suite at coco-algorithms-biobj.Local and archived data can be freely mixed like
python -m cocopp YOURDATAFOLDER 'bbob/2010/IPOP-ACT'
which processes the data from
YOURDATAFOLDER
and the archived IPOP-ACT data set in comparison.The output folder,
ppdata
by default, contains all output from the post-processing. Theindex.html
file is the main entry point to explore the result with a browser. Data from the same foldername as previously processed may be overwritten. If this is not desired, a different output folder name can be chosen with the-o OUTPUT_FOLDERNAME
option.A summary pdf can be produced via LaTeX. The corresponding templates can be found in the
code-postprocessing/latex-templates
folder. Basic html output is also available in the result folder of the postprocessing (filetemplateBBOBarticle.html
). -
In order to exploit more features of the post-processing module, it is advisable to use the module within a Python or IPython shell or a Jupyter notebook or JupyterLab, where
import cocopp help(cocopp)
provides the documentation entry pointer.
-
Once your algorithm runs well, increase the budget in your experiment script, if necessary implement randomized independent restarts, and follow the above steps successively until you are happy.
-
The experiments can be parallelized with any re-distribution of single problem instances to batches (see
example_experiment2.py
for an example). Each batch must write in a different target folder (this should happen automatically). Results of each batch must be kept under their separate folder as is. These folders then must be moved/copied into a single folder which becomes the input folder to the post-processing. (The post-processing searches in all subfolders and subsub... for.info
files to begin with. The folder structure of a single sub-experiment must not be changed, as the.info
file relies on it to find the data files.)
If you detect bugs or other issues, please let us know by opening an issue in our issue tracker at https://github.com/numbbo/coco/issues.
-
the
do.py
file in the root folder is a tool to build the entire distribution.do.py
is a neat and simplifying replacement for make. It has switches for just building some languages etc, typepython do.py
to see a list of all available commandes.
-
the code-experiments/build folder is to a large extend the output folder of the
./do.py build
command.- the
exampleexperiment.???
files in the build folder are the entry points to understand the usage of the code (as end-user). They are supposed to actually be executable (in case, after compilation, which should be taken care of bydo.py
and/ormake
) and run typically random search on (some of) the provided benchmark suites.
- the
-
documentation and examples might not be too meaningful for the time being, even though code-experiments/documentation/onion.py describes a (heavily) used design pattern (namely: inheritance) in a comparatively understandable way (though the implementation in C naturally looks somewhat different). Section Links and Documentation provides a list of pointers.
-
the code-experiments/src folder is where most of the important/interesting things happen. Many files provide comparatively decent documentation at the moment which are translated via doxygen into a more readable web page at http://numbbo.github.io/coco-doc/C/. Generally:
- coco.h is the public interface, in particular as used in the example_experiment.c file
- coco_internal.h provides the type definition of coco_problem_t
- coco_suite.c is code that deals with an entire benchmark suite (i.e. a set of functions, eg. sweeping through them etc...)
- coco_generics.c is somewhat generic code, e.g. defining a function call via coco_evaluate_function etc
- coco_problem.c is the implementation of the coco_problem_t type/object (allocation etc).
- observer / logger files implement data logging (as wrappers around a coco problem inheriting thereby all properties of a coco problem)
- most other files implement more or less what they say, e.g. the actual benchmark functions, transformations, benchmark suites, etc.
- currently, the following benchmark suites and corresponding logging facilities are
supported:
bbob
: standard single-objective BBOB benchmark suite with 24 noiseless, scalable test functionsbbob-biobj
: a bi-objective benchmark suite, combining 10 selected functions from the bbob suite, resulting in 55 noiseless functionsbbob-largescale
: a version of thebbob
benchmark suite with dimensions 20 to 640, employing permuted block-diagonal matrices to reduce the execution time for function evaluations in higher dimension.bbob-mixint
: a mixed-integer version of the originalbbob
andbbob-largescale
suites in which 80% of the variables have been discretizedbbob-biobj-mixint
: a version of the (so far not supported)bbob-biobj-ext
test suite with 92 functions with 80% discretized variablestoy
: a simple, probably easier-to-understand example for reading and testing
-
code-experiments/tools are a few meta-tools, mainly the amalgamate.py to merge all the C code into one file
-
code-experiments/test contains unit- and integration-tests, mainly for internal use
-
code-postprocessing/cocopp contains the postprocessing code, written in python, with which algorithm data sets can be read in and the performance of the algorithms can be displayed in terms of data profiles, aRT vs. dimension plots, or simple tables.
-
code-postprocessing/helper-scripts contains additional, independent python scripts that are not part of the cocopp module but that might use it.
-
code-postprocessing/latex-templates contains LaTeX templates for displaying algorithm performances in publisher-conform PDFs for the GECCO conference.
-
code-preprocessing/archive-update/ contains internal code for combining the archives of algorithms to create/update the hypervolume reference values for the
bbob-biobj
test suite -
code-preprocessing/log-reconstruction/ contains internal code for reconstructing output of the
bbob-biobj
logger from archive files (needed when the hypervolume reference values are updated) -
howtos contains a few text files with generic howtos.
If you see something like this when running python do.py run-java
or build-java
under Linux
COPY code-experiments/src/coco.h -> code-experiments/build/java/coco.h
WRITE code-experiments/build/java/REVISION
WRITE code-experiments/build/java/VERSION
RUN javac CocoJNI.java in code-experiments/build/java
RUN javah CocoJNI in code-experiments/build/java
Traceback (most recent call last):
File "do.py", line 590, in <module>
main(sys.argv[1:])
File "do.py", line 563, in main
elif cmd == 'build-java': build_java()
File "do.py", line 437, in build_java
env = os.environ, universal_newlines = True)
File "/..../code-experiments/tools/cocoutils.py", line 34, in check_output
raise error
subprocess.CalledProcessError: Command '['locate', 'jni.h']' returned non-zero exit status 1
it means javah
is either not installed (see above) or cannot be found in the system
path, see this
and possibly this for a solution.
If you see something like this when running python do.py build-matlab
AML ['code-experiments/src/coco_generics.c', 'code-experiments/src/coco_random.c', 'code-experiments/src/coco_suite.c', 'code-experiments/src/coco_suites.c', 'code-experiments/src/coco_observer.c', 'code-experiments/src/coco_runtime_c.c'] -> code-experiments/build/matlab/coco.c
COPY code-experiments/src/coco.h -> code-experiments/build/matlab/coco.h
COPY code-experiments/src/best_values_hyp.txt -> code-experiments/build/matlab/best_values_hyp.txt
WRITE code-experiments/build/matlab/REVISION
WRITE code-experiments/build/matlab/VERSION
RUN matlab -nodisplay -nosplash -r setup, exit in code-experiments/build/matlab
Traceback (most recent call last):
File "do.py", line 447, in <module>
main(sys.argv[1:])
File "do.py", line 429, in main
elif cmd == 'build-matlab': build_matlab()
File "do.py", line 278, in build_matlab
run('code-experiments/build/matlab', ['matlab', '-nodisplay', '-nosplash', '-r', 'setup, exit'])
File "/Users/auger/workviasvn/newcoco/numbbo/code-experiments/tools/cocoutils.py", line 68, in run
universal_newlines=True)
File "//anaconda/lib/python2.7/subprocess.py", line 566, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "//anaconda/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "//anaconda/lib/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
it might be because your system does not know the matlab
command. To fix this,
you should edit the file /etc/paths
and add the path to the matlab
bin file
(Linux/Mac) or add the path to the folder where the matlab.exe
lies to your
Windows path. For instance, the etc/paths
should look like something like this
/usr/local/bin
/usr/bin
/bin
/usr/sbin
/sbin
/Applications/MATLAB_R2012a.app/bin/
With the more complex SMS-EMOA example, the problem is related to the compilation
of the external C++ hypervolume calculation in hv.cpp
.
A fix for this issue consists in adding to the files hv.cpp
and paretofront.c
#define char16_t UINT16_T
just before the line:
#include "mex.h"
If it happens that you get some Access is denied
errors during
python do.py build-matlab
or python do.py run-matlab
like this one
C:\Users\dimo\Desktop\numbbo-brockho>python do.py run-matlab
Traceback (most recent call last):
File "do.py", line 649, in <module>
main(sys.argv[1:])
File "do.py", line 630, in main
elif cmd == 'run-matlab': run_matlab()
File "do.py", line 312, in run_matlab
os.remove( filename )
WindowsError: [Error 5] Access is denied: 'code-experiments/build/matlab\\cocoEv
aluateFunction.mexw32'
a reason can be that a previously opened Matlab window still has some
file handles open. Simply close all Matlab windows (and all running Matlab
processes if there is any) before to run the do.py
command again.
When running
python do.py run-octave
or
python do.py build-octave
and seeing something like
[...]
compiling cocoCall.c...error: mkoctfile: please install the Debian package "liboctave-dev" to get the mkoctfile command
then, unsurprisingly, installing liboctave-dev
like
sudo apt-get install liboctave-dev
should do the job.
If you see something like this
$ python do.py run-python # or build-python
[...]
PYTHON setup.py install --user in code-experiments/build/python
ERROR: return value=1
Traceback (most recent call last):
File "setup.py", line 8, in <module>
import setuptools
ImportError: No module named setuptools
Traceback (most recent call last):
File "do.py", line 562, in <module>
main(sys.argv[1:])
File "do.py", line 539, in main
elif cmd == 'build-python': build_python()
File "do.py", line 203, in build_python
python('code-experiments/build/python', ['setup.py', 'install', '--user'])
File "/vol2/twagner/numbbo/code-experiments/tools/cocoutils.py", line 92, in p ython
universal_newlines=True)
File "/usr/local/lib/python2.7/subprocess.py", line 575, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['/usr/local/bin/python', 'setup.py', 'i nstall', '--user']' returned non-zero exit status 1
then setuptools
needs to be installed:
pip install setuptools
or easy_install setuptools
should do the job.
If you see something like this:
$ python do.py run-python # or build-python
[...]
cython/interface.c -o build/temp.linux-i686-2.6/cython/interface.o
cython/interface.c:4:20: error: Python.h: file not found
cython/interface.c:6:6: error: #error Python headers needed to compile C extensions, please install development version of Python.
error: command 'gcc' failed with exit status 1
or
$ python do.py run-python # or build-python
[...]
cython/interface.c -o build/temp.linux-x86_64-2.7/cython/interface.o
cython/interface.c:4:20: fatal error: Python.h: No such file or directory
#include "Python.h"
^
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Under Linux
sudo apt-get install python-dev
should do the trick.
We have observed a case where the update of the cocoex
Python module seemed to have no
effect. In this case it has been successful to remove all previously installed versions,
see here for a few more details.
It can happen that the postprocessing fails due to too long paths to the algorithm data. Unfortunately, the error you get in this case does not indicate directly to the problem but only tells that a certain file could not be read. Please try to shorten the folder names in such a case.
We have occasionally observed some font issues in the pdfs, produced by the postprocessing
of COCO (see also issue #1335). Changing to
another matplotlib
version solved the issue at least temporarily.
Under the Mac operating system, bibtex
seems to be messed up a bit with respect to
absolute and relative paths which causes problems with the test of the postprocessing
via python do.py test-postprocessing
. Note that there is typically nothing to fix if
you compile the LaTeX templates "by hand" or via your LaTeX IDE. But to make the
python do.py test-postprocessing
work, you will have to add a line with
openout_any = a
to your texmf.cnf
file in the local TeX path. Type
kpsewhich texmf.cnf
to find out where this file actually is.
If for some reason, your python installation is corrupted and running
python do.py install-postprocessing
crashes with an error message like
[...]
safe = scan_module(egg_dir, base, name, stubs) and safe
File "C:\Users\dimo\Anaconda2\lib\site-packages\setuptools\command\bdist_egg.py", line 392, in sca
n_module
code = marshal.load(f)
EOFError: EOF read where object expected
[...]
try adding zip_safe=False
to the setup.py.in
file in the code-postprocessing
folder. More details can be found in the issue #1373.
Earlier versions of cocopp
have written extracted data to a folder named _extracted_...
.
If the post-processing is invoked with a *
argument, these folders become an argument and
are displayed (most likely additionally to the original algorithm data folder). Solution:
remove the _extracted_...
folders and use the latest version of the post-processing
module cocopp
(since release 2.1.1).
-
The C code features an object oriented implementation, where the
coco_problem_t
is the most central data structure / object.coco.h
,example_experiment.c
andcoco_internal.h
are probably the best pointers to start to investigate the code (but see also here).coco_problem_t
defines a benchmark function instance (in a given dimension), and is called viacoco_evaluate_function
. -
Building, running, and testing of the code is done by merging/amalgamation of all C-code into a single C file,
coco.c
, andcoco.h
. (by callingdo.py
, see above). Like this it becomes very simple to include/use the code in different projects. -
Cython is used to compile the C to Python interface in
build/python/interface.pyx
. The Python module installation filesetup.py
uses the compiledinterface.c
, ifinterface.pyx
has not changed. For this reason, Cython is not a requirement for the end-user. -
We continuously test the code through the open source automation server Jenkins on one ubuntu 12.04 machine, one OSX 10.9 machine, and two 32-bit Windows 7 machines (one with and one without cygwin).
You may cite this work in a scientific context as
N. Hansen, A. Auger, R. Ros, O. Mersmann, T. Tušar, D. Brockhoff. COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting, Optimization Methods and Software, 2020. [arXiv version]
@ARTICLE{hansen2020cocoplat,
author = {Hansen, N. and Auger, A. and Ros, R. and Mersmann, O. and
Tu{\v s}ar, T. and Brockhoff, D.},
title = {{COCO}: A Platform for Comparing Continuous Optimizers
in a Black-Box Setting},
journal = {Optimization Methods and Software},
doi = {https://doi.org/10.1080/10556788.2020.1808977},
year = 2020
}
- The BBOB workshop series, which uses the NumBBO/Coco framework extensively, can be tracked here
- Data sets from previous experiments for many algorithms are available at
- http://coco.gforge.inria.fr/doku.php?id=algorithms-bbob for the
bbob
test suite - http://coco.gforge.inria.fr/doku.php?id=algorithms-bbob-noisy for the
bbob-noisy
test suite - http://coco.gforge.inria.fr/doku.php?id=algorithms-bbob-biobj for the
bbob-biobj
test suite, and at - http://coco.gforge.inria.fr/doku.php?id=algorithms-bbob-largescale for the
bbob-largescale
test suite.
- http://coco.gforge.inria.fr/doku.php?id=algorithms-bbob for the
- Postprocessed data for each year in which a BBOB workshop was taking place can be found at http://coco.gforge.inria.fr/ppdata-archive
- Stay informed about the BBOB workshop series and releases of the NumBBO/Coco software by registering at http://coco.gforge.inria.fr/register
- Read about the basic principles behind the Coco platform in COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting.
- For details on the experimental set-up to carry out the benchmarking please refer to http://numbbo.github.io/coco-doc/experimental-setup/.
- More detailed documentation of the existing benchmark suites can be found here:
- for the
bbob
problem suite at http://coco.gforge.inria.fr/downloads/download16.00/bbobdocfunctions.pdf with the experimental setup at http://coco.lri.fr/downloads/download15.03/bbobdocexperiment.pdf - for the
bbob-biobj
andbbob-biobj-ext
problem suites at http://numbbo.github.io/coco-doc/bbob-biobj/functions - for the
bbob-largescale
problem suite at http://numbbo.github.io/coco-doc/bbob-largescale/functions - for the
bbob-mixint
andbbob-biobj-mixint
problem suites, we refer to https://hal.inria.fr/hal-02067932/document and to http://coco.gforge.inria.fr/preliminary-bbob-mixint-documentation/bbob-mixint-doc.pdf
- for the
- Online documentation of the NumBBO/Coco API (i.e. for the ANSI C code) is available at http://numbbo.github.io/coco-doc/C
- More detailed documentation on how the performance assessement is done can be found at http://numbbo.github.io/coco-doc/perf-assessment/.
- Downloading this repository
- via the above green "Clone or Download" button or
- by typing
git clone https://github.com/numbbo/coco.git
or - via https://github.com/numbbo/coco/archive/master.zip in your browser
-
General introduction: COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting
-
Experimental setup: http://numbbo.github.io/coco-doc/experimental-setup/
-
Testbeds
- bbob: http://coco.gforge.inria.fr/downloads/download16.00/bbobdocfunctions.pdf
- bbob-biobj: http://numbbo.github.io/coco-doc/bbob-biobj/functions/
- bbob-biobj-ext: http://numbbo.github.io/coco-doc/bbob-biobj/functions/
- bbob-noisy (only in old code basis): http://coco.lri.fr/downloads/download15.03/bbobdocnoisyfunctions.pdf
- bbob-largescale: http://numbbo.github.io/coco-doc/bbob-largescale/functions/
- bbob-mixint and bbob-biobj-mixint: https://hal.inria.fr/hal-02067932/document and http://coco.gforge.inria.fr/preliminary-bbob-mixint-documentation/bbob-mixint-doc.pdf
- bbob-constrained (in progress): http://numbbo.github.io/coco-doc/bbob-constrained/functions/
-
Performance assessment: http://numbbo.github.io/coco-doc/perf-assessment/
-
Performance assessment for biobjective testbeds: http://numbbo.github.io/coco-doc/bbob-biobj/perf-assessment/
-
APIs
C
experiments code: http://numbbo.github.io/coco-doc/C- Python experiments code (module
cocoex
): http://coco.gforge.inria.fr/apidocs-cocoex/cocoex.html - Python short beginners example experiment
- Python
example_experiment2.py
: http://coco.gforge.inria.fr/apidocs-example_experiment/example_experiment2.html - Postprocessing code (module
cocopp
): http://coco.gforge.inria.fr/apidocs-cocopp/cocopp.html
-
Somewhat outdated documents:
- Full description of the platform: http://coco.lri.fr/COCOdoc/
- Experimental setup before 2016: http://coco.lri.fr/downloads/download15.03/bbobdocexperiment.pdf
- Old framework software documentation: http://coco.lri.fr/downloads/download15.03/bbobdocsoftware.pdf
-
Some examples of results.