Add a build_source module

[ekohl]

Currently we have a tito extension in foreman-packaging. The idea is expand this module to replace that and support building just by pointing at a git checkout. This is easy for development but also makes it easy to write a Jenkins job to build a scratch build of a PR.

[ehelms]

A few things I'm curious about:

1. This is quite specific to a Rakefile.dist existing and pkg:generate source rake task. We tend to have generic tarballs and gem sources as well.
2. This would just require ruby and rake to work? It'd be good to add module docs and include that requirement.

[ekohl]

This is very draft level. As you can see there's an if/else. I intend to expand that so it can also build gemspecs into gems. I imagine we can also detect setup.py to build python tarballs and/or wheels. We may need configuration since some packages have multiple gemspecs.

I haven't thought about dependencies.

Some thing I do wonder about is the Rakefile building method (which would come after the gemspec method) is that we need a bundle install. This module feels like the wrong place but on the other hand this is the only place that actually determines the method.

[ekohl]

Gemspec building still needs some stderr handling.

ekohl@donald ~/dev/obal (build-source) $ echo '{"ANSIBLE_MODULE_ARGS": {"directory": "/home/ekohl/dev/foreman"}}' | python3 obal/data/modules/build_source.py | jq --raw-output .path
/home/ekohl/dev/foreman/pkg/foreman-1.22.0-develop.tar.bz2
ekohl@donald ~/dev/obal (build-source) $ echo '{"ANSIBLE_MODULE_ARGS": {"directory": "/home/ekohl/dev/foreman-installer"}}' | python3 obal/data/modules/build_source.py | jq --raw-output .path
/home/ekohl/dev/foreman-installer/pkg/foreman-installer-1.22.0-develop.tar.bz2
ekohl@donald ~/dev/obal (build-source) $ echo '{"ANSIBLE_MODULE_ARGS": {"directory": "/home/ekohl/dev/katello"}}' | python3 obal/data/modules/build_source.py | jq --raw-output .path
WARNING:  licenses is empty, but is recommended.  Use a license identifier from
http://spdx.org/licenses or 'Nonstandard' for a nonstandard license.
WARNING:  no summary specified
WARNING:  open-ended dependency on rails (>= 0) is not recommended
  if rails is semantically versioned, use:
    add_runtime_dependency 'rails', '~> 0'
WARNING:  open-ended dependency on json (>= 0) is not recommended
  if json is semantically versioned, use:
    add_runtime_dependency 'json', '~> 0'
WARNING:  open-ended dependency on oauth (>= 0) is not recommended
  if oauth is semantically versioned, use:
    add_runtime_dependency 'oauth', '~> 0'
WARNING:  open-ended dependency on rest-client (>= 0) is not recommended
  if rest-client is semantically versioned, use:
    add_runtime_dependency 'rest-client', '~> 0'
WARNING:  open-ended dependency on rabl (>= 0) is not recommended
  if rabl is semantically versioned, use:
    add_runtime_dependency 'rabl', '~> 0'
WARNING:  open-ended dependency on dynflow (>= 1.2.0) is not recommended
  if dynflow is semantically versioned, use:
    add_runtime_dependency 'dynflow', '~> 1.2', '>= 1.2.0'
WARNING:  open-ended dependency on foreman_docker (>= 0.2.0) is not recommended
  if foreman_docker is semantically versioned, use:
    add_runtime_dependency 'foreman_docker', '~> 0.2', '>= 0.2.0'
WARNING:  open-ended dependency on activerecord-import (>= 0) is not recommended
  if activerecord-import is semantically versioned, use:
    add_runtime_dependency 'activerecord-import', '~> 0'
WARNING:  open-ended dependency on qpid_messaging (>= 0) is not recommended
  if qpid_messaging is semantically versioned, use:
    add_runtime_dependency 'qpid_messaging', '~> 0'
WARNING:  open-ended dependency on gettext_i18n_rails (>= 0) is not recommended
  if gettext_i18n_rails is semantically versioned, use:
    add_runtime_dependency 'gettext_i18n_rails', '~> 0'
WARNING:  open-ended dependency on apipie-rails (>= 0.5.14) is not recommended
  if apipie-rails is semantically versioned, use:
    add_runtime_dependency 'apipie-rails', '~> 0.5', '>= 0.5.14'
WARNING:  open-ended dependency on anemone (>= 0) is not recommended
  if anemone is semantically versioned, use:
    add_runtime_dependency 'anemone', '~> 0'
WARNING:  open-ended dependency on minitest-tags (>= 0, development) is not recommended
  if minitest-tags is semantically versioned, use:
    add_development_dependency 'minitest-tags', '~> 0'
WARNING:  open-ended dependency on mocha (>= 0, development) is not recommended
  if mocha is semantically versioned, use:
    add_development_dependency 'mocha', '~> 0'
WARNING:  open-ended dependency on webmock (>= 0, development) is not recommended
  if webmock is semantically versioned, use:
    add_development_dependency 'webmock', '~> 0'
WARNING:  open-ended dependency on rubocop-checkstyle_formatter (>= 0, development) is not recommended
  if rubocop-checkstyle_formatter is semantically versioned, use:
    add_development_dependency 'rubocop-checkstyle_formatter', '~> 0'
WARNING:  open-ended dependency on simplecov (>= 0, development) is not recommended
  if simplecov is semantically versioned, use:
    add_development_dependency 'simplecov', '~> 0'
WARNING:  open-ended dependency on simplecov-rcov (>= 0, development) is not recommended
  if simplecov-rcov is semantically versioned, use:
    add_development_dependency 'simplecov-rcov', '~> 0'
WARNING:  open-ended dependency on robottelo_reporter (>= 0, development) is not recommended
  if robottelo_reporter is semantically versioned, use:
    add_development_dependency 'robottelo_reporter', '~> 0'
WARNING:  See http://guides.rubygems.org/specification-reference/ for help
/home/ekohl/dev/katello/katello-3.12.0.pre.master.gem
ekohl@donald ~/dev/obal (build-source) $ echo '{"ANSIBLE_MODULE_ARGS": {"directory": "/home/ekohl/dev/smart-proxy"}}' | python3 obal/data/modules/build_source.py | jq --raw-output .path
/home/ekohl/dev/smart-proxy/pkg/foreman-proxy-1.22.0-develop.tar.bz2

This requires theforeman/foreman-installer#332 and theforeman/foreman#6576.

As you can see I've handled the multiple gemspecs by using the directory name. This means we can clone and create a symlink for the core package.

[komidore64]

As you can see I've handled the multiple gemspecs by using the directory name. This means we can clone and create a symlink for the core package.

What does this mean for us going forward? How are you imagining this would be used?

[ekohl]

I'm not entirely sure, but I'm thinking about expanding the package manifest. One example is https://github.com/theforeman/foreman_ansible which has two gemspecs. If no directory is given it's the basename of the git repository (as git clone normally also does), possibly stripping .git at the end.

packages:
  hosts:
    rubygem-foreman_ansible:
      source:
        git: https://github.com/theforeman/foreman_ansible
    rubygem-foreman_ansible_core:
      source:
        git: https://github.com/theforeman/foreman_ansible
        directory: foreman_ansible_core

Then we need some directive to enter build-from-source mode. I thought about passing in --git-checkout-path. If a git checkout with the right name is already present, it doesn't touch it. This allows us to use obal scratch --git-checkout-path $WORKDIR <packages> in Jenkins (when implementing [build rpm] on foreman.git together with GHPRB.