Optionally generate "dep-info for external build system integration" for test and example binaries

[gsrushton]

Problem

Currently only artefacts destined for uplift to the top-level target directory have extended dep-info generated for them.

In order for external build tools to be able to trivially ascertain whether it's necessary to invoke cargo to rebuild (and potentially re-run) test and example executables it would be ideal if cargo could, optionally, generate the extended dep-info for them as well.

Proposed Solution

Generate extended dep-info for test and binary executables.

• Generation enabled by environment variable
• Generate a new file alongside the binary and rustc dep-info in the target/.../deps folder, perhaps named something of the form <artefact-name>-<hash>-extended.d.

Notes

I'm happy to do the work once the proposal is accepted and an implementation agreed.

[gsrushton]

@rustbot label: +A-dep-info

[epage]

Could you provide a reproduction case of "on this project, I run this command, and I'm reading dep info files in this way"?

[grushton-lidya]

Could you provide a reproduction case of "on this project, I run this command, and I'm reading dep info files in this way"?

Absolutely, I think the most trivial example would be a project that uses make to invoke cargo.

Imagine a project called foo created via cargo new foo. To the root of the project I can then add the following Makefile:

target/debug/foo:
	CARGO_BUILD_DEP_INFO_BASEDIR=$(CURDIR) \
	  cargo build --package foo

include $(wildcard target/debug/foo.d)

Which allows make to do out-of-date checks for foo's binary:

$ make
CARGO_BUILD_DEP_INFO_BASEDIR=/a/path/foo \
  cargo build --package foo
   Compiling foo v0.1.0 (/a/path/foo)
    Finished dev [unoptimized + debuginfo] target(s) in 0.29s
$ make
make: 'target/debug/foo' is up to date.
$ touch src/main.rs
$ make
CARGO_BUILD_DEP_INFO_BASEDIR=/a/path/foo \
  cargo build --package foo
   Compiling foo v0.1.0 (/a/path/foo)
    Finished dev [unoptimized + debuginfo] target(s) in 0.23s

This works even if foo depends on other local crates and it is their source that changes.

At a high-level, what I want to be able to do is something similar for tests. I want to author a Makefile that looks like this:

target/debug/deps/foo-somehash:
	CARGO_BUILD_DEP_INFO_BASEDIR=$(CURDIR) \
	  cargo test --package foo

include $(wildcard target/debug/deps/foo-somehash.d)

And have make only attempt to rebuild and re-run tests if any source files upon which the tests depend have changed.

Ignoring the somehash above, which I can already account for using --message-format=json; the above doesn't work reliably because target/debug/deps/foo-somehash.d only includes source files from within the foo crate, not from other local crates on which it might depend.

[weihanglo]

In order for external build tools to be able to trivially ascertain whether it's necessary to invoke cargo to rebuild (and potentially re-run) test and example executables it would be ideal if cargo could, optionally, generate the extended dep-info for them as well.

To make sure we're on the same page, the "extended dep-info" you are talking about is the third kind of dep-info in the doc, right? I believe "example binaries" already got their own extended dep-info files. They are kinda uplifted to target/debug/examples/ directory.

The only thing left is dep-info for test executables.

[weihanglo]

Generate a new file alongside the binary and rustc dep-info in the target/.../deps folder, perhaps named something of the form <artefact-name>-<hash>-extended.d.

This could work, but not ideal. People then have one more implicit detail to depend on. I wonder if we should move tests to a separate directory like target/debug/tests/ so it won't hit hard on duplicate artifact names again.

[weihanglo]

There is a discussion on Zulip about a query interafce that is a bit related: https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/Query.20Interface.20for.20Cargo.

[grushton-lidya]

To make sure we're on the same page, the "extended dep-info" you are talking about is the third kind of dep-info in the doc, right?

Yes, correct

I believe "example binaries" already got their own extended dep-info files. They are kinda uplifted to target/debug/examples/ directory.

I missed that, thanks for highlighting it :)

I wonder if we should move tests to a separate directory like target/debug/tests/ so it won't hit hard on #6313 again.

Would this include removing the fingerprint hash from the file-name as well, similar to the other binary artefacts that get uplifted? If so that would be ideal.