Optimize library installer conan calls

[partouf]

Now that the Cloudfront caching in front of the Conan server is disabled, we need to figure out a way to manually optimize the process.

A large IO bottleneck is the checks that we do to prevent from building the same thing over and over again:

• has_failed_before https://github.com/compiler-explorer/infra/blob/main/bin/lib/library_builder.py#L783
• is_already_uploaded https://github.com/compiler-explorer/infra/blob/main/bin/lib/library_builder.py#L796
  • get_build_annotations https://github.com/compiler-explorer/infra/blob/main/bin/lib/library_builder.py#L752 is part of this

These are called for every compiler that will be built with. There might be a way to request the information for all compilers at once and then only do the lookup per compiler.

Related endpoint: https://github.com/compiler-explorer/conanproxy/blob/main/index.js#L454

• See also: https://github.com/compiler-explorer/conanproxy/blob/main/build-annotations.js#L18
• Unfortunately the conan_hash will be unique per compiler, but if you iterate over all the directories in the package directory (https://github.com/compiler-explorer/conanproxy/blob/main/build-annotations.js#L11C102-L11C111) you can match the hashes on the builder side

Lesser influence, but still IO:

• get_commit_hash

  infra/bin/lib/library_builder.py

  Line 769 in 8564738

  def get_commit_hash(self) -> str:

  • This really only has to be called once, so either the first time its called or just in the constructor

[AbrilRBS]

is_already_uploaded main/bin/lib/library_builder.py#L796

I'm thinking that Conan is meant to lift these concerns out from us. Maybe it's time to implement a more standard Conan workflow to build all libraries? I think the end result could be super cool :)

[partouf]

is_already_uploaded main/bin/lib/library_builder.py#L796

I'm thinking that Conan is meant to lift these concerns out from us. Maybe it's time to implement a more standard Conan workflow to build all libraries? I think the end result could be super cool :)

I invite you to try, but I can pretty much promise you you'll be wasting your time. I have tried all kinds of standard things, they just don't work for the library builder.