New derivation-making builtin

[Ericson2314]

builtins.derivation (and builtins.derivationStrict) have a number of issues:

• drvPath has a funky "DrvDeep" string context that needs to be stripped away with the scary-sounding builtins.unsafeDiscardOutputDependency.

• Mixing together environment variable and magic flags is icky (this also applies to the on-disk .drv format)

• Returning the arguments to the user provided encourages memory leaks (derivationStrict already improves on this over derivation)

• Returning the boiled-down inputs (e.g. inputSrcs, inputDrvs) is sometimes useful, and we could just return it here.

• The outputs are mixed in with the other attributes (separation of dicts and records; see JSON guideline)

• Remembers meta attributes, including such attributes as

  • meta.timeout, tracked by EvalState and/or Worker

  • meta.credentials (as in Pluggable authentication #9857 (comment))

  • meta.position (as discussed at Zurich ZHF 2024.05)

• Possibly: arbitrary other meta attributes

TODO other things

Priorities

Add 👍 to issues you find important.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-01-15-nix-team-meeting-minutes-115/38297/1

[roberth]

Outdated - possible pkg.derivation attr in Nixpkgs

We should align this with

• lib.extendDerivation: Add derivation attribute nixpkgs#281161

Currently that means that the return type should have a path field containing the .drv, but other than that, I've proposed to keep it lean.

[roberth]

We should also consider whether we even want to return such attributes as inputSrcs, inputDrvs eagerly, as this will probably result in plenty of unused-but-referenced Values that we can't GC.
Technically we could work around that by adding them as thunks that call an (internal or not) primop for retrieving those from the .drv file. This is only a partial solution to the space leak though, because the attrs and thunks still take a bit of space, even if not evaluated.

[Ericson2314]

@roberth I was thinking the user would just drop them if they don't want them.

Outdated - possible pkg.derivation attr in Nixpkgs

But yes, this conflicts with derivation just being the result of the new primop.

[roberth]

Nix prints the position of the name attribute, but this is almost always useless:

whose name attribute is located at nixpkgs/pkgs/stdenv/generic/make-derivation.nix:331:7

Ideally, Nix would accept a __pos parameter with a user-provided position, such as mkDerivation's pos argument, which is already used for some Nixpkgs-specific error messages. Nix should use this too, but __pos must not be written to the actual derivation in the store, as that would affect the hash every time the position changes slightly.
Changes to derivation attribute hashing need to be forward compatible, and we have no good forward compatibility in the current derivation/derivationStrict primops.

In summary:

• Implement measures for forward compatibility
• Use __pos argument for error positions, but don't write it to the derivation.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-03-11-nix-team-meeting-132/42960/1

[roberth]

Added two examples of meta attributes that the store layer must be aware of.