Automated p2 update site

[vorburger]

It would be cool if we could make a p2 update site automatically available.

No "releases", just "continous delivery", from latest master.

@sylvainlaurent any thoughts how we could achieve this?

[vorburger]

Via #2 I realized that in this project first incarnation you had http://sylvainlaurent.github.io/eclipse-external-annotations/p2/ from https://github.com/sylvainlaurent/eclipse-external-annotations.github.io, right?

That was probably manual though. It would be cool if we could somehow fully automate it..

[sylvainlaurent]

I'll have a look at what we can do with travis CI. The thing is that we need to push the artifacts to an http server which is not a maven repo.

[vorburger]

Cool, thanks! I just found https://docs.travis-ci.com/user/deployment, but that's not what we want..

[ctron]

We could use the Eclipse Package Drone instance for this [1]

This way we could simply upload with mvn deploy and have a P2 repository out of the box.

[1] https://packagedrone.eclipse.org/

[ctron]

I am sorry, I have to revise that last idea. Actually we can't deploy this to the Eclipse Package Drone instance .. since this is not (yet) an Eclipse project.

But we could take a different Package Drone instance for doing this instead.

[ctron]

We could take my older version of Package Drone running on OpenShift [1]

[1] https://repository-ctron.rhcloud.com/

[ctron]

I did create a test channel for this: [1]

[1] https://repository-ctron.rhcloud.com/p2/lastnpe-ci/

[victornoel]

Why not do the following in travis:

• compile the update site with mvn clean package
• clone the gh-pages branch in another folder (which would contain a folder named p2/snapshots)
• go to the update site and execute:

mvn -Dtycho.mode=maven -Dfinal-update-site=/gh-pages/p2/snapshots tycho-p2-extras:publish-features-and-bundles tycho-p2:category-p2-metadata

This will add the new release to the already existing update site so previous one will be kept (but you could also discard previous version and just remove the old one before)

• push to gh-pages (see https://github.com/steveklabnik/automatically_update_github_pages_with_travis_example, http://awestruct.org/auto-deploy-to-github-pages and one of the many other tutorial on the internet)

You would get CD like this, and you could do the same for releases by the way on tagging

[kwin]

Why not leverage the approach also taken for kwin/cppcheclipse#72 (i.e. bintray)?

[vorburger]

@victornoel and/or @kwin would either one of you be willing to set this up as a contribution to the project? I'd be happy to make you project contributors on GitHub, and grant you any permissions you may require etc. (I haven't had the time recently to look after this.)

Either approach is fine with me. If I had to choose myself, I'd go for the approach @victornoel suggested with hosting a p2 repo on Git Hub (pages); that's how I've done the https://github.com/lastnpe/lastnpe.github.io. For a project with a community like what I hoped this would become, that seems a bit easier than relying on Bintray and dealing with secrets to deploy there? I wouldn't use a gh-pages branch though, because that blows up the size of the source git repo, but would just create another git repo, named like eclipse-external-annotations-m2e-plugin-p2-site; that works great (I've done this in other projects). What I've never looked into myself is how to get Travis to clone another repo (simple bash script?), and commit and push into that (there's probably some way to give it the secret?)...

[victornoel]

@vorburger I can take a look during the week.
I also don't know exactly how to work with multiple repo from travis, I think there will be some need to have some secrets to deal with too for it to work… but I will explore and tell you what I find!
@kwin feel free to give some arguments for using bintray instead of github pages, I'm not exactly sure which is best either :)

[vorburger]

I've just created https://github.com/lastnpe/eclipse-external-annotations-m2e-plugin-p2-site if you want to use that (I don't mind removing it again if we end up going for another solution here). Invited you as a collaborator to that, if you have the time.

http://www.lastnpe.org/eclipse-external-annotations-m2e-plugin-p2-site/ is HTTP serving what's in that repo now. NB GitHub pages don't do the directory index listing thing, which can initially be confusing, that's why http://www.lastnpe.org/eclipse-external-annotations-m2e-plugin-p2-site/ 404s, but e.g. www.lastnpe.org/eclipse-external-annotations-m2e-plugin-p2-site/README.md works. This does work for Install from Eclipse, because it will just look for p2 metadata, not directory listing.

Perhaps you could even have folders for branches it was built from in that - like master/ for continuous delivery from latest master, and possible future stable/, if this projects ever decided that is needed. (There is some git command you can use to obtain the current branch the build was from; I remember having done that in another project.)

[victornoel]

@vorburger great, I will see what is doable and get back to you :)

[kwin]

@vorburger Using bintray has the advantage that this repo is really thought for binaries while git clearly is not. You can still trigger the deployment via Travis (although this should only be done for SNAPSHOTs, not for releases). With Travis you can even maintain a secret which is not exposed to everyone (https://docs.travis-ci.com/user/encryption-keys/).

[victornoel]

Sorry, took me some time to start looking.

So to be able to publish something to another repository from travis, we could use something like this:
https://github.com/angular/material2/blob/master/scripts/release/publish-build-artifacts.sh
This means a token/key should still be registered in travis (I suppose that we can generate one from https://github.com/lastnpe/eclipse-external-annotations-m2e-plugin-p2-site, most certainly with delpoy keys or machine users), in the same way than if using bintray (but without the need to use bintray on top of github and travis).

I don't see a problem deploying release artefact to it (simply trigger a travis job on tags, and the same should be true for bintray), we just need to find the right way to do it :)

[vorburger]

sorry for super long silence on this.. finally picking this up now:

I've meanwhile learnt out how to use Bintray in another project of mine.

On https://github.com/lastnpe/eclipse-external-annotations-m2e-plugin/compare/deploy-bintray I'm trying to deploy to https://bintray.com/vorburger/lastnpe.org/eclipse-external-annotations-m2e-plugin/master ... but I've no idea yet what's causing this weird Ruby error.

@kaikreuzer FYI ^^^

[victornoel]

@vorburger it means gsub was called on a null object, and it should have been a string.
Looking at dpl's code, I suspect it is because uploadPattern in your configuration is not set as you expect it to be!

Also should you put the deploy key in clear? Shouldn't you use something like some variable that you configure in travis so that nobody can't see it?

[vorburger]

@victornoel as far as I understand that deploy key is actually not in clear, I've used "travis encrypt", as per doc.

@victornoel tx for the Ruby de-coding ;) you're right and I had figured, something deployed now; I'm checking if it's right and if it works (i.e. if it really can be installed from Eclipse).

[vorburger]

Test goal is to use https://dl.bintray.com/vorburger/lastnpe.org/eclipse-external-annotations-m2e-plugin/master/ but the p2 site doesn't quite work yet because somehow the features/ and a plugins/ ended up at the root instead of inside eclipse-external-annotations-m2e-plugin/master/ - anyone reading this got any clue what could possibly be wrong in the bintray.json ?

An error occurred while collecting items to be installed
session context was:(profile=_home_vorburger_dev_ODL_eclipses_oxygen_eclipse, phase=org.eclipse.equinox.internal.p2.engine.phases.Collect, operand=, action=).
Artifact not found: https://dl.bintray.com/vorburger/lastnpe.org/eclipse-external-annotations-m2e-plugin/master/plugins/eclipse-external-annotations-m2e-plugin.core_1.0.0.201709150202.jar.
https://dl.bintray.com/vorburger/lastnpe.org/eclipse-external-annotations-m2e-plugin/master/plugins/eclipse-external-annotations-m2e-plugin.core_1.0.0.201709150202.jar
Artifact not found: https://dl.bintray.com/vorburger/lastnpe.org/eclipse-external-annotations-m2e-plugin/master/features/eclipse_external_annotations_m2e_plugin.feature_1.0.0.201709150202.jar.
https://dl.bintray.com/vorburger/lastnpe.org/eclipse-external-annotations-m2e-plugin/master/features/eclipse_external_annotations_m2e_plugin.feature_1.0.0.201709150202.jar

Once that's sorted out, next steps here before closing will be:

1. raise a PR for community review from test branch
2. update README.md with link to final URL

PS: Here are some references to more complicated steps I found but don't seem to have to use:

• http://www.lorenzobettini.it/2016/02/publish-an-eclipse-p2-composite-repository-on-bintray/
• https://github.com/LorenzoBettini/p2composite-bintray-example
• https://stackoverflow.com/questions/42445278/bintray-support-for-eclipse-p2-repositories
• Provide p2 update site kwin/cppcheclipse#72
• https://github.com/vogellacompany/bintray-publish-p2-updatesite/blob/master/pushToBintray.sh
• https://github.com/openhab/openhab-core/blob/master/features/repo/src/main/filtered-resources/publish-bintray.sh

[victornoel]

maybe some problem with relative versus abolute urls in the upload pattern that is computed?

[vorburger]

@victornoel yeah I probably got the RegEx wrong.

BUT I think the TravisCI -> Bintray integration may be broken ?!

I'll wait for some time to see if any replies on travis-ci/issues/8441 .. else could try doing some custom script such as openHAB's publish-bintray.sh to upload like via curl, but that seems a shame (and I've already sunk too much time into this!).

[vorburger]

OK enough time sunk into this... ;-) Got it working easily using GitHub pages instead of Travis-Bintray, so p2 site now up on http://www.lastnpe.org/eclipse-external-annotations-m2e-plugin-p2-site/ - tested installation into Eclipse, all good - DONE! ;-) Phew.

[victornoel]

Excellent, thanks a lot!

That looks quite easy in the end, but the way to get here was no so easy :)

[victornoel]

@vorburger you forgot to update the README with the new info :)

[vorburger]

@victornoel see #22 re. README!

[askpythia]

Please update the P2 repos with new version working with M2E 2.x