[bug] Initial setup for Conan test suite is not automated or documented

[valgur]

Describe the bug

The initial setup for running tests is currently quite substantial due to the number of tools and versions of them required.
I did not manage to find any docs for the setup besides https://github.com/conan-io/conan/blob/develop2/test/README.md#installation-of-tools

I assume this is mostly unintentional and simply due to the natural growth in the scope and complexity of the client? I hope I don't have to go into more detail why having a considerable overhead to running the test suite is far from ideal for the quality of external contributions to this repo.

A decent portion of the setup could be automated relatively easily and almost entirely so for Linux. Some dependencies, like Visual Studio, will inevitably have to be installed manually.

A version of this was proposed in #16548, which was closed. Not wanting to make the setup a part of the test suite itself is quite understandable. Would a separate setup script that handles the automatic setup of most of the tools be an acceptable solution? Meson, Scons and Ninja can be easily installed as Python dev dependencies as well.

How to reproduce it

No response

[valgur]

For reference, the current list of tools used by the test suite is:

• android_ndk
• apt_get
• autotools
• bazel 6.3.2
• bazel 7.1.2
• brew
• clang 12
• clang 13
• clang 16
• cmake 3.15
• cmake 3.16
• cmake 3.17
• cmake 3.19
• cmake 3.23
• cmake 3.28
• cygwin
• gcc
• git
• icc
• intel_oneapi
• make
• meson
• mingw32
• mingw64
• msbuild
• msys2
• msys2_clang64
• msys2_mingw64_clang64
• ninja
• pkg_config
• qbs
• scons
• ucrt64
• visual_studio 15
• visual_studio 16
• visual_studio 17
• xcodebuild
• xcodegen

[memsharded]

Thanks for your feedback.

This wouldn't be necessarily a bug. It is not intended that every developer have in their machines all tools installed to run the test suite, rather the opposite. Not even the maintainers have all of those tools installed locally, not even the majority of those tools, just a relatively small subset of them.

The idea is that local development never needs many tools. Most of the time, integration tests are good for developing, we use them like 90% of the time, and they don't need any tools. When needing to debug something very specific of a build system a specific tool might be needed, but just with having that tool it is enough for development. Then CI takes care of the rest.

The proposed approach is to have the configuration defined of the current tools, what we have in the conftest.py and conftest_user.py. I totally agree that this test feature of using the conftest needs an overhaul and improving it, maybe changing some defaults. We will try to improve it when possible, but focused first on the "definition" of available tools, but not on the automated installation of those tools yet. After that overhaul we might consider automating the installation of some tools.

[valgur]

It is not intended that every developer have in their machines all tools installed to run the test suite, rather the opposite.

Ok, that's good to know. A sentence or two in CONTRIBUTING.md and/or test/README.md mentioning this would be very helpful.

After that overhaul we might consider automating the installation of some tools.

Although it can be improved, I don't really mind the current structure of conftest.py. I think the automated installation of tools can be handled in isolation from conftest.py for the most part with a separate script.

Here's a potential solution you could consider (and I would be more than happy to help implement):

• Create a separate script, e.g. test/setup_tools.py, which installs as many of the tools for the current platform as possible or reasonable under test/tools. Could make the set of tools to be installed configurable via command-line args as well.
• Move the tools_locations = { ... } setting to a separate conf file and add a pytest command-line option such as pytest -t <conf>.py.
• Create a couple of different tool configs with different coverages:
  • minimal.py – no tools enabled. Would be the default when running just pytest.
  • auto.py – auto-installed tools.
  • standard.py – the baseline configuration, which is run on CI. Excludes currently disabled more obscure or unmaintained tools such as Qbs. Would expect the user to copy and adjust this as necessary.
  • full.py – everything enabled.

Ideally, I would expect a dev unfamiliar with the internals of Conan to be able to run pytest out of the box or ./test/setup_tools.py && pytest -t auto.py and see all tests pass.