Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add TOPCOM 0.17.8 builder #3900

Merged
merged 5 commits into from
Nov 17, 2021
Merged

Conversation

lkastner
Copy link
Contributor

Add builder for TOPCOM

@giordano giordano changed the title Lk/topcom Add TOPCOM 0.17.8 builder Nov 16, 2021
@giordano
Copy link
Member

Thanks a lot for your contribution! I've pushed a change to avoid the symlinks, which I didn't really like (although I can see why you had to do that...). Tell me what you think 🙂 I think my change is less maintainable (it's a large patch), but at least the build script is more straightforward. I'm happy to revert it if you don't like it, I did it mostly as an exercise 😄

@giordano giordano force-pushed the lk/TOPCOM branch 2 times, most recently from ab33eae to f0a91b1 Compare November 16, 2021 22:57
@benlorenz
Copy link
Contributor

benlorenz commented Nov 16, 2021

@giordano @lkastner what do you think of this one instead: benlorenz@770ddb7

@giordano
Copy link
Member

I have also added a commit to build the shared libraries. They can't be used directly from Julia as they are since it's C++, but may be useful in the future nevertheless. Happy to get feedback on this as well!

@giordano
Copy link
Member

@benlorenz that looks great and also makes my last commit useless (but it's along the same lines) 😁

@giordano
Copy link
Member

Yeah, I think I like your change (much) better than mines 😄

@benlorenz
Copy link
Contributor

unfortunately my version doesn't seem to work on windows yet ...

@giordano
Copy link
Member

unfortunately my version doesn't seem to work on windows yet ...

Neither does mine 😂 Building the shared library seems to hit undefined symbols (only on the platforms where the linker actually check them by default). And also dlopening the libs fails

@benlorenz
Copy link
Contributor

I disabled windows for now, lets see if the other platforms work. The libraries libTOPCOM and libREGCHECK seem to depend on each other which is not really feasible like that on windows, and might cause the dlopen problems as well.

I can't really test all the platforms locally as I constantly get those annoying download failures:

ERROR: LoadError: Unable to automatically install 'GCCBootstrap-armv7l-linux-gnueabihf.v4.8.5.x86_64-linux-musl.squashfs'

@giordano
Copy link
Member

@staticfloat we're getting these download errors very frequently now, see also JuliaPackaging/BinaryBuilderBase.jl#178 (comment)

@giordano
Copy link
Member

@benlorenz do you have PkgServer disabled by any chance?

and enable libraryproducts
libCHECKREG needs to link against libTOPCOM
@benlorenz
Copy link
Contributor

benlorenz commented Nov 17, 2021

I have added the LibraryProducts now, dlopen does seem to work during the audit with the added libs, I will try to have a look at mingw tomorrow.

@benlorenz do you have PkgServer disabled by any chance?

I did add JULIA_PKG_SERVER="" at some point, probably to avoid registry delays and maybe because some other download failures a while ago. I will remove that and see if it helps.

If I try to download such a file manually with wget I get the following:

$ wget -O /dev/null https://github.com/JuliaPackaging/Yggdrasil/releases/download/GCCBootstrap-v5.2.0+2/GCCBootstrap-x86_64-w64-mingw32.v5.2.0.x86_64-linux-musl.squashfs.tar.gz
--2021-11-17 00:54:49--  https://github.com/JuliaPackaging/Yggdrasil/releases/download/GCCBootstrap-v5.2.0+2/GCCBootstrap-x86_64-w64-mingw32.v5.2.0.x86_64-linux-musl.squashfs.tar.gz
Resolving github.com... 140.82.121.3
Connecting to github.com|140.82.121.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/148698815/352c2180-e42c-11ea-8237-84d4759ee392?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211116%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211116T235450Z&X-Amz-Expires=300&X-Amz-Signature=b13e63e31de63e1214d408622e526098494af73c99666b8316635870c2e13a14&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=148698815&response-content-disposition=attachment%3B%20filename%3DGCCBootstrap-x86_64-w64-mingw32.v5.2.0.x86_64-linux-musl.squashfs.tar.gz&response-content-type=application%2Foctet-stream [following]
--2021-11-17 00:54:50--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/148698815/352c2180-e42c-11ea-8237-84d4759ee392?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211116%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211116T235450Z&X-Amz-Expires=300&X-Amz-Signature=b13e63e31de63e1214d408622e526098494af73c99666b8316635870c2e13a14&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=148698815&response-content-disposition=attachment%3B%20filename%3DGCCBootstrap-x86_64-w64-mingw32.v5.2.0.x86_64-linux-musl.squashfs.tar.gz&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com... 185.199.109.133, 185.199.111.133, 185.199.108.133, ...
Connecting to objects.githubusercontent.com|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 223849535 (213M) [application/octet-stream]
Saving to: ‘/dev/null’

/dev/null                                            100%[======================================================================================================================>] 213.48M  31.9MB/s    in 6.8s

2021-11-17 00:56:52 (31.5 MB/s) - ‘/dev/null’ saved [223849535/223849535]

Note that the total duration is slightly more than 2 minutes and I have seen even about 5 minutes and in some cases wget also failed after a while. Most of the time it is just stuck at HTTP request sent, awaiting response... before the 200 OK, the download itself took only a few seconds.
During the Yggdrasil builds the downloads abort a lot faster than this seems to be taking here.

@giordano
Copy link
Member

During the Yggdrasil builds the downloads abort a lot faster than this seems to be taking here.

Exactly, I wonder if Pkg has a rather low timeout, while GitHub is clearly taking a lot to serve the file. Response time from PkgServer is way faster for me instead: https://pkg.julialang.org/artifact/daf4fe73a24fc28599072795093e51dbfa6397c1

circular library-dependencies are fun

and try splitting up the build_tarballs to remove the LibraryProduct for windows
@benlorenz
Copy link
Contributor

I have set mingw to do static linking and removed the LibraryProduct for the mingw builds. I don't think anyone will miss them.

From my point of view this is ready now.

@lkastner lkastner marked this pull request as ready for review November 17, 2021 11:26
@lkastner
Copy link
Contributor Author

@giordano @benlorenz In the end almost everything got replaced, so thank you for doing all the work. ;)

Comment on lines +79 to +83
products_unix = [
LibraryProduct("libTOPCOM", :libTOPCOM),
LibraryProduct("libCHECKREG", :libCHECKREG),
products_win...
]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm slightly unhappy about having different products on different platforms, this can be confusing for users/package developers who'd expect to find libTOPCOM everywhere if TOPCOM_jll.is_available() is true. I thought about including products also in JuliaPackaging/BinaryBuilderBase.jl#174, but it looked error-prone to me. What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could leave out these products for all platforms or just remove the mingw platforms altogether, both would be fine for me.
But I really don't want to reorganize the source for these libraries, and currently both need symbols from the other one.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might also split this up into a libTOPCOM_jll which has just the shared libraries for unix platforms and TOPCOM_jll for all platforms with the binaries (which then might use platform-dependent dependencies ....). But this seems to be a little bit too much effort.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this seems to be a little bit too much effort.

Agreed. Ok, well, let's merge this as it is and let's see how it goes! Thanks everybody!

@giordano
Copy link
Member

In the end almost everything got replaced

If it wasn't for the weird requirement about having gmp and cddlib as static libraries in a very specific location, your initial PR would have been perfect!

@giordano giordano merged commit 375cab5 into JuliaPackaging:master Nov 17, 2021
simeonschaub pushed a commit to simeonschaub/Yggdrasil that referenced this pull request Feb 23, 2022
* Add TOPCOM 0.17.8 builder

* T/TOPCOM: Expand to remaining platforms

* [TOPCOM] less static linking and some minor improvements

* [TOPCOM] link libraries against gmp and cdd

and enable libraryproducts
libCHECKREG needs to link against libTOPCOM

* [TOPCOM] link statically on windows

circular library-dependencies are fun

and try splitting up the build_tarballs to remove the LibraryProduct for windows

Co-authored-by: Benjamin Lorenz <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants