Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make jobs unique #60

Open
edolstra opened this issue Feb 15, 2013 · 4 comments
Open

Make jobs unique #60

edolstra opened this issue Feb 15, 2013 · 4 comments
Assignees

Comments

@edolstra
Copy link
Member

Currently a jobset evaluation can produce multiple builds with the same job name. This is bad because it means jobs don't really have a distinct identity.

This means many job-based comparisons in Hydra are very fuzzy: for instance, finding the last job status change is not well defined if there are multiple jobs with the same name. (E.g. in the previous eval, one "foo" build could have failed while the other succeeded. Then if in the current eval one fails and the other succeeds, we cannot really say if their status changed: they may have both flipped, or both stayed the same. We do use the system type to distinguish between builds, but that's very ad hoc.)

So we should make jobs unique in a jobset eval, i.e., within an eval there should be at most one build with a given job name.

Implementation / consequences:

  • We should add a "job" field to JobsetEvalMembers table. The primary key of that table will change from (eval, build) to (eval, job). So we have a functional dependency (eval, job) -> build.
  • We can get rid of the Jobs table.
  • Really the only reason why we have multiple builds per job is to have multiple system types per job (e.g. patchelf:trunk:build is done on several system types). But we can just use job names for this, e.g. build.x86_64-linux and build.i686-linux. This can be expressed explicitly in job expressions, i.e. you write build.x86_64-linux = buildMyPackage { system = "x86_64-linux"; }; and so on. Or you abstract over this pattern with a helper function that returns an attribute set containing builds for multiple system types. Compared to the current "input alternatives" mechanism, this style has the advantage that you can just do "nix-build release.nix -A build.x86_64-linux". In fact it removes the need for inputs like "system" (you just express the desired system types in your release.nix, which is more declarative). In other words, this unifies job names and attribute paths.
  • For backwards compatibility, hydra-eval-jobs can rename jobs automatically. So if it sees multiple jobs with the same name, it will add unique suffixes like ".x86_64-linux".
  • We can probably get rid of the BuildInputs table, since the JobsetEvalInputs contains everything needed to reproduce a build. We can probably also get rid of having multiple alternatives per jobset input, since the only place we use that (AFAIK) is for "system" inputs.
  • We can get rid of all special handling of "system", i.e. everything that now works on "distinct job, system" can just become "distinct job".
  • The /job controller should show all jobs with a given attribute path prefix, so going to /job/patchelf/trunk/build would show the jobs build.x86_64-linux, build.i686-linux and so on.
  • Upgrading the old schema will be a bit tricky since it needs to generate unique job names for all existing builds. So for every (jobset evaluation, job name), if there are multiple builds, it should add suffixes to make the job names unique. If the builds all differ in system type, it can just use the system type as a suffix. Otherwise it should probably just number them ("build.0", "build.1", ...). Fortunately this seems to be pretty rare.
@edolstra
Copy link
Member Author

Some stats on the non-uniqueness of (job, system): there are only 431 jobset evaluations (out of 261549) that contain multiple builds with the same (job, system). Mostly gnu:libidn-master, ltsmin:ltsmin-trunk and strategoxt:stratego-libraries-trunk.

Generated using:

select project, jobset, id from jobsetevals e where hasnewbuilds = 1 and exists 
  (select 1 from builds b join jobsetevalmembers on b.id = build and eval = e.id 
   group by job, system having count(*) > 1);

@ghost ghost assigned edolstra Feb 15, 2013
@shlevy
Copy link
Member

shlevy commented Apr 1, 2013

Maybe you explained this already, but how will Build output (same system) work? Require adding a separate jobset input for each supported system?

edolstra added a commit that referenced this issue Aug 20, 2013
Having different builds within a job is obsolete (issue #60), one
should have different job per platform (e.g. build.x86_64-linux).
edolstra added a commit that referenced this issue Aug 20, 2013
The per-system presentation doesn't make much sense any more given
issue #60.  It should be replaced by (say) a grid showing each job per
evaluation.
chexxor pushed a commit to chexxor/nixos-configurations that referenced this issue Nov 6, 2013
That is, there are now distinct jobs like ‘coreutils.x86_64-linux’ and
‘coreutils.x86_64-darwin’, rather than a single job ‘coreutils’ with
multiple builds.  This means that testing a job is simpler:

  $ nix-build pkgs/top-level/release.nix -A coreutils.x86_64-linux

See NixOS/hydra#60 for the motivation.
edolstra added a commit that referenced this issue Sep 24, 2014
This has been deprecated since a8db329.

Issue #60.
codyopel pushed a commit to triton/patchelf that referenced this issue Jun 2, 2015
E.g. "build.x86_64-linux" instead of "build".  Also, by not using a
Hydra input "system" anymore, the set of platforms to build on becomes
declarative.

See NixOS/hydra#60
@gilligan
Copy link
Contributor

gilligan commented May 8, 2020

@edolstra is this still relevant? You made various changes that contribute to this. Maybe you want to update the description with a list of items that would still have to be done to get this closed.

@edolstra
Copy link
Member Author

edolstra commented May 9, 2020

Yes, it's still relevant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants