Reorganise include files #7847

andreabedini · 2023-02-16T00:28:45Z

Motivation

Currently the pkg-config files says to use the following include path

❯ pkg-config --cflags nix-main
-I/nix/store/6c14rkp59v85n9c29x69bllv7ijp9wya-nix-2.13.2-dev/include/nix -std=c++17

I believe it would be more appropriate if this said

❯ pkg-config --cflags nix-main
-I/nix/store/6c14rkp59v85n9c29x69bllv7ijp9wya-nix-2.13.2-dev/include -std=c++17  # proposed

In this way consumers could do #include <nix/config.h>, while at the moment this does not work.

❯ g++ $(pkg-config --cflags nix-main) main.cc 
main.cc:1:10: fatal error: nix/config.h: No such file or directory
    1 | #include <nix/config.h>
      |          ^~~~~~~~~~~~~~
compilation terminated.

From what I can understand, in a stdenv pkg-config files are redundant because adding nix to your build dependencies automatically causes the compiler wrapper to pass -I${nix-dev}/include to the compiler, which is the same thing I believe we should do with pkg-config.

One example consumer of the nix libraries is hercules-ci-cnix-store, which cannot be compiled outside of a nix evironment (I should say without a nix wrapped compiler) because it includes the nix/config.h path.

~/code/hercules-ci-agent/hercules-ci-cnix-store
❯ rg nix/config
src/Hercules/CNix.hs
42:C.include "<nix/config.h>"

src/Hercules/CNix/Verbosity.hs
21:C.include "<nix/config.h>"

src/Hercules/CNix/Util.hs
28:C.include "<nix/config.h>"

src/Hercules/CNix/Store.hs
61:C.include "<nix/config.h>"

src/Hercules/CNix/Settings.hs
36:C.include "<nix/config.h>"

src/Hercules/CNix/Exception.hs
19:C.include "<nix/config.h>"

Context

I am making (so far very experimental) tools that link to nix code from outside of a nix stdenv. I have installed nix-dev in my profile and manually adjusted PKG_CONFIG_PATH.

❯ nix profile list | grep ^21
21 flake:nixpkgs#legacyPackages.x86_64-linux.nix github:NixOS/nixpkgs/8c619a1f3cedd16ea172146e30645e703d21bfc1#legacyPackages.x86_64-linux.nix /nix/store/6c14rkp59v85n9c29x69bllv7ijp9wya-nix-2.13.2-dev /nix/store/jifwi51d9sal3zql0n84xmabs3bk3v37-nix-2.13.2 /nix/store/vb7p3xsl06kks5zlkmwl5x9a73z7ncxz-nix-2.13.2-man

Checklist for maintainers

Maintainers: tick if completed or explain if not relevant

agreed on idea
agreed on implementation strategy
tests, as appropriate
- functional tests - tests/**.sh
- unit tests - src/*/tests
- integration tests - tests/nixos/*
documentation in the manual
code and comments are self-explanatory
commit message explains why the change was made
new feature or incompatible change: updated release notes

andreabedini · 2023-02-16T08:18:16Z

I just noticed this is not enough because nix headers also don't have the nix/ in front so they are still not useable outside of the nix codebase. I'll have a go at changing this too (hopefully I won't get lost).

edolstra · 2023-02-16T16:20:10Z

I just noticed this is not enough because nix headers also don't have the nix/ in front so they are still not useable outside of the nix codebase.

Yeah, this is the main reason we're not doing this already. I think it would require moving all the headers to a nix/ directory which would be quite annoying.

Ericson2314 · 2023-02-16T19:29:48Z

Thanks for doing this @andreabedini. It is also necessary to work on Hydra and Nix at the same time incrementally, which is critical for being able to address NixOS/hydra#1164 productively.

Yeah, this is the main reason we're not doing this already. I think it would require moving all the headers to a nix/ directory which would be quite annoying.

Git handles file moves very well. We should move the headers to /src/libnix*/nix subdirs and rip off the band-aid.

andreabedini · 2023-02-17T01:45:25Z

Git handles file moves very well. We should move the headers to /src/libnix*/nix subdirs and rip off the band-aid.

I started working on this, although I opted for src/libutil/include/nix/util. The advantage being you can tell right away what component you are including. E.g.

-#include "serialise.hh"
+#include "nix/util/serialise.hh"

Let me know if you have strong opinions about this.

Ericson2314 · 2023-02-17T05:06:41Z

I personally like that (I am all about good layering!), and it shouldn't matter since this stuff is unstable anyways, but I will note that this is a breaking change as to where the files are installed whereas the other thing isn't.

andreabedini · 2023-02-17T06:06:56Z

I personally like that (I am all about good layering!), and it shouldn't matter since this stuff is unstable anyways, but I will note that this is a breaking change as to where the files are installed whereas the other thing isn't.

You are right, this was a step past the original intention which was only changing the pc files are readjusting the include paths. That would have only required a small change in the consumer build setup (e.g. a change in a Makefile) while this brings in (possibly annoying) a change in the source code. Thanks for pointing this out @Ericson2314!

I could add a bunch of symlinks to ease up the transition pain? (edit: or shims with a deprecation warning?)

andreabedini · 2023-02-20T09:02:15Z

Added compatibility shims as well.

~/code/nix-tests $ make
g++ -I/nix/store/wmsdakkg2a6ndi8hy6faxdi191cvm7a9-nix-2.14.0pre20230220_a205add-dev/include -std=c++2a   -L/nix/store/ni1nk8dhd11f5zw4rb40kkb53pb62vyj-nix-2.14.0pre20230220_a205add/lib -lnixexpr -lnixstore -lnixutil -latomic_ops -lgc -lpthread -ldl   tmp.cpp   -o tmp
In file included from tmp.cpp:2:
/nix/store/wmsdakkg2a6ndi8hy6faxdi191cvm7a9-nix-2.14.0pre20230220_a205add-dev/include/nix/eval.hh:1:2: warning: #warning "Including nix/eval.hh is deprecated. Please include nix/expr/eval.hh instead." [-Wcpp]
    1 | #warning "Including nix/eval.hh is deprecated. Please include nix/expr/eval.hh instead."
      |  ^~~~~~~
In file included from tmp.cpp:3:
/nix/store/wmsdakkg2a6ndi8hy6faxdi191cvm7a9-nix-2.14.0pre20230220_a205add-dev/include/nix/util.hh:1:2: warning: #warning "Including nix/util.hh is deprecated. Please include nix/util/util.hh instead." [-Wcpp]
    1 | #warning "Including nix/util.hh is deprecated. Please include nix/util/util.hh instead."
      |  ^~~~~~~

thufschmitt

I really like the overall change (modulo a couple of small things). Given the amount of code churn, I'd like to leave @NixOS/nix-team a chance to voice their opinion, but it's a 👍 from me.

configure.ac

thufschmitt · 2023-02-20T10:39:59Z

src/libcmd/include/nix/command.hh

@@ -0,0 +1,2 @@
+#warning "Including nix/command.hh is deprecated. Include nix/cmd/command.hh instead."


Would it make sense to generate these files on-the-fly rather than checking them out?

(they aren't big nor anything, but I imagine them being a bit confusing and very annoying when browsing the codebase with a fuzzy-finder)

I personally don't think these stub files are worth the trouble. It's not too hard to do the bulk rename with a small script to see which library it was moved to. This stuff is explicitly unstable/internal, so breakage is not so much of an issue.

As a maintainer of a package that uses the headers headers, I will have to add a version check conditional around my imports anyway.

Automating this guidance is probably not worth the effort for our limited ecosystem of library consumers.

If that's the consensus, I am happy to remove the shims. @roberth do you see any way we can make the transition easier for downstream consumers?

I think the worst part is figuring out where each include file went, and the shims addressed this by mentioning the location of the new file. I could make a awk script to migrate to the new locations but it won't help if you need to support both.

This reverts commit 746f4d5.

This reverts commit a205add.

edolstra · 2023-02-21T15:54:11Z

I'm not really in favor of this, not just because of the massive amount of code churn, but also because I rather like having the header files next to their corresponding implementation files (e.g. foo.{cc,hh}).

Also, the resulting paths seem superfluous/repetitive (e.g. nix/src/libcmd/include/nix/cmd).

Ericson2314 · 2023-02-21T16:32:44Z

I suggest that this PR #7847, PR #7870, and my old #5589 both touch on very similar issues, and projects like #7735 (more bindings) somewhat force the issue.

These PRs are hard to rebase (thought actually easy to rebase other PRs over! The disruption is not bad!) so I guest we figure out what we want in an issue first.

I'll get started on writing that issue.

Ericson2314 · 2023-02-21T17:40:42Z

#7876 This is the issue. Please comment there!

fricklerhandwerk · 2023-02-24T12:29:53Z

Triaged in the Nix team meeting:

@edolstra: this PR changes essentially every source file to effectively better interop with pkg-config
- don't see how the cost justifies the advantage
- @Ericson2314: this unlocks some work down the line, would like to discuss this
@thufschmitt: should keep this low priority unless something important depends on this
to discuss

nixos-discourse · 2023-02-24T13:02:03Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-02-24-nix-team-meeting-minutes-35/25757/1

andreabedini · 2023-02-27T03:18:00Z

I am aware this change is quite invasive. pkg-config by itself is not the goal, the goal is to make it easier to link to nix C++ API from other projects (e.g. outside nix). Language bindings come to mind but the sky is the limit.

Ericson2314 · 2023-02-27T12:01:15Z

@andreabedini I will say the notes reflect but a very brief triaging conversation moving this to the "to discuss column".

fricklerhandwerk · 2023-03-27T12:38:33Z

Discussed in Nix team meeting:

@andreabedini we decided to postpone this for now, as the change is fairly costly in terms of moving around lots of files. We will reconsider it in light of the more general issue when we make more progress there: #7876

nixos-discourse · 2023-03-27T13:44:52Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-03-27-nix-team-meeting-minutes-44/26759/1

nixos-discourse · 2023-04-17T09:45:25Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-04-14-nix-team-meeting-minutes-48/27358/1

fricklerhandwerk · 2023-07-21T12:19:19Z

Revisited in the Nix maintainers meeting:

@roberth: counter-proposal: keep headers next to source files, and almost in their existing locations:
- instead of include/nix/store/*.hh (this PR) or src/libstore/*.hh (status quo) use src/libnixstore/*.hh (note the nix)
- then a source build would be -I./src (new) and for consumers -I${nix}/include (like this PR already)
- In both cases, #include "libnixexpr/*.hh" works
- the alternative of putting in place symlinks as a simulation of the old structure is probably tech debt, as it will complicate dealing with renames in Git
  - @edolstra: also we break the API all the time and never made guarantees about it
@thufschmitt: another hacky solution would be patching header files on installation
- @Ericson2314: this would preclude the "Cross-project incremental development" motivation of Reorganize this repo #7876, however
we agree on @roberth's proposal

nixos-discourse · 2023-07-21T12:25:33Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-07-21-nix-team-meeting-minutes-73/30768/1

Ericson2314 · 2023-07-23T15:10:02Z

Oh one downside to -I./src is that it would expose all headers to all libraries, allowing mistakes where upstream libraries could use downstream libraries' headers.

I think @thufschmitt had started to think of symlinking so we could use the layout from the issue while still keeping the headers next to sources as @edolstra wanted? That would also be a good option.

roberth · 2023-07-23T23:08:14Z

Oh one downside to -I./src is that it would expose all headers to all libraries, allowing mistakes where upstream libraries could use downstream libraries' headers.

Another way to check that is to have a derivation per library. That would save some rebuilds as well.

andreabedini · 2023-10-09T07:15:29Z

At this point I would say the PR is useful only for the discussion it generated. Given this PR is also mentioned in the discussion in #7876, I can be confident it will not get lost and close it ☺️

Better value for pkg-config cflags

d223504

andreabedini requested review from edolstra and thufschmitt as code owners February 16, 2023 00:28

Fix perl bindings

85db118

Fix NixOS#5832

746f4d5

andreabedini added 2 commits February 17, 2023 11:03

Move libutil include files

bb99593

Move libcmd include files

4fd6b62

Move libexpr include files

c6bc00c

Move libfetchers include files

c3c9cf5

andreabedini changed the title ~~Better value for pkg-config cflags~~ Reorganise include files Feb 17, 2023

Move libmain include files

fecba41

andreabedini marked this pull request as draft February 17, 2023 06:46

andreabedini added 11 commits February 17, 2023 17:37

Move libstore include files

76ad005

Fix resolve-system-dependencies

7c5cec2

Fix perl bindings

95db452

Fix header files installation

1dbc9d3

Fix plugins tests

7cf977b

Merge remote-tracking branch 'upstream/master' into fix-pkg-config

921078f

Remove extranous files

70891ac

Remove extranous files

6f85dde

Perl bindings use pkg-config

5df4c10

Add result-* to gitignore

a73a110

Add include shims with deprecation warning

a205add

Whitespace

69f0244

andreabedini marked this pull request as ready for review February 20, 2023 09:06

thufschmitt suggested changes Feb 20, 2023

View reviewed changes

andreabedini added 3 commits February 21, 2023 09:16

Revert "Fix NixOS#5832"

1ce1aee

This reverts commit 746f4d5.

Merge remote-tracking branch 'upstream/master' into fix-pkg-config

298f9d9

Revert "Add include shims with deprecation warning"

878ad8a

This reverts commit a205add.

thufschmitt added the contributor-experience Developer experience for Nix contributors label Feb 27, 2023

fricklerhandwerk mentioned this pull request Mar 27, 2023

Reorganize this repo #7876

Open

Ericson2314 mentioned this pull request Mar 29, 2023

Source filtering and disabling tests (to later run them externally) #7870

Open

7 tasks

andreabedini closed this Oct 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reorganise include files #7847

Reorganise include files #7847

andreabedini commented Feb 16, 2023

andreabedini commented Feb 16, 2023

edolstra commented Feb 16, 2023

Ericson2314 commented Feb 16, 2023

andreabedini commented Feb 17, 2023

Ericson2314 commented Feb 17, 2023

andreabedini commented Feb 17, 2023 •

edited

Loading

andreabedini commented Feb 20, 2023

thufschmitt left a comment

thufschmitt Feb 20, 2023

Ericson2314 Feb 20, 2023 •

edited

Loading

roberth Feb 20, 2023

andreabedini Feb 21, 2023

edolstra commented Feb 21, 2023 •

edited

Loading

Ericson2314 commented Feb 21, 2023 •

edited

Loading

Ericson2314 commented Feb 21, 2023

fricklerhandwerk commented Feb 24, 2023 •

edited

Loading

nixos-discourse commented Feb 24, 2023

andreabedini commented Feb 27, 2023

Ericson2314 commented Feb 27, 2023

fricklerhandwerk commented Mar 27, 2023

nixos-discourse commented Mar 27, 2023

nixos-discourse commented Apr 17, 2023

fricklerhandwerk commented Jul 21, 2023

nixos-discourse commented Jul 21, 2023

Ericson2314 commented Jul 23, 2023 •

edited

Loading

roberth commented Jul 23, 2023

andreabedini commented Oct 9, 2023

		@@ -0,0 +1,2 @@
		#warning "Including nix/command.hh is deprecated. Include nix/cmd/command.hh instead."

Reorganise include files #7847

Reorganise include files #7847

Conversation

andreabedini commented Feb 16, 2023

Motivation

Context

Checklist for maintainers

andreabedini commented Feb 16, 2023

edolstra commented Feb 16, 2023

Ericson2314 commented Feb 16, 2023

andreabedini commented Feb 17, 2023

Ericson2314 commented Feb 17, 2023

andreabedini commented Feb 17, 2023 • edited Loading

andreabedini commented Feb 20, 2023

thufschmitt left a comment

Choose a reason for hiding this comment

thufschmitt Feb 20, 2023

Choose a reason for hiding this comment

Ericson2314 Feb 20, 2023 • edited Loading

Choose a reason for hiding this comment

roberth Feb 20, 2023

Choose a reason for hiding this comment

andreabedini Feb 21, 2023

Choose a reason for hiding this comment

edolstra commented Feb 21, 2023 • edited Loading

Ericson2314 commented Feb 21, 2023 • edited Loading

Ericson2314 commented Feb 21, 2023

fricklerhandwerk commented Feb 24, 2023 • edited Loading

nixos-discourse commented Feb 24, 2023

andreabedini commented Feb 27, 2023

Ericson2314 commented Feb 27, 2023

fricklerhandwerk commented Mar 27, 2023

nixos-discourse commented Mar 27, 2023

nixos-discourse commented Apr 17, 2023

fricklerhandwerk commented Jul 21, 2023

nixos-discourse commented Jul 21, 2023

Ericson2314 commented Jul 23, 2023 • edited Loading

roberth commented Jul 23, 2023

andreabedini commented Oct 9, 2023

andreabedini commented Feb 17, 2023 •

edited

Loading

Ericson2314 Feb 20, 2023 •

edited

Loading

edolstra commented Feb 21, 2023 •

edited

Loading

Ericson2314 commented Feb 21, 2023 •

edited

Loading

fricklerhandwerk commented Feb 24, 2023 •

edited

Loading

Ericson2314 commented Jul 23, 2023 •

edited

Loading