Opt out of precompilation

[schillic]

Is there a way to deactivate precompilation of all test dependencies?
I have a use case where a package has lots of optional dependencies that are pretty heavy to precompile (and PkgEval does not seem to use a cache). I am already defining cheaper tests via the environment variable JULIA_PKGEVAL such that those optional packages are not used in the tests. But the timeout hits even before running the tests because all those optional packages get precompiled first.

[maleadt]

There is currently no way to opt out of that at a package level.

PkgEval does not seem to use a cache

It does:

PkgEval.jl/src/evaluate.jl

Lines 507 to 536 in ff138cf

	# (cache and) clean-up output created by this package
	if use_cache
	depot_dir = joinpath(workdir, "upper", "home", "pkgeval", ".julia")
	local_compilecache = joinpath(depot_dir, "compiled")
	local_packages = joinpath(depot_dir, "packages")
	local_artifacts = joinpath(depot_dir, "artifacts")
	# verify local resources
	registry_dir = get_registry(config)
	isdir(local_packages) &&
	remove_uncacheable_packages(registry_dir, local_packages; show_status=false)
	isdir(local_artifacts) &&
	verify_artifacts(local_artifacts; show_status=false)
	isdir(local_compilecache) &&
	verify_compilecache(local_compilecache; show_status=false)
	# copy new local resources (packages, artifacts, ...) to shared storage
	lock(storage_lock) do
	for (src, dst) in [(local_packages, shared_packages),
	(local_artifacts, shared_artifacts),
	(local_compilecache, shared_compilecache)]
	if isdir(src)
	# NOTE: removals (whiteouts) are represented as char devices
	run(`$(rsync()) --no-specials --no-devices --archive --quiet $(src)/ $(dst)/`)
	end
	end
	end
	end
	chmod_recursive(workdir, 0o777) # JuliaLang/julia#47650
	rm(workdir; recursive=true)

Every build of Julia invalidates this cache though, as there's no guarantee compilation caches work across Julia commits.

[schillic]

There is currently no way to opt out of that at a package level.

Okay, thanks. Do you see an alternative solution?

If not, is this something that would be interesting to add? And how difficult would it be?
I do not know how PkgEval works, but since each PkgEval run is per-package, it may in principle be possible to deactivate precompilation conditionally. A way to signal skipping could be to add a file test/.PkgEval_no_precompile in one's repository.

PkgEval does not seem to use a cache

It does
Every build of Julia invalidates this cache though, as there's no guarantee compilation caches work across Julia commits.

Okay, sorry, I did not check this. Makes sense.
Do you mean that multiple new nightly versions come out during one PkgEval cycle on all tested packages?
If so, maybe fixing the Julia version for one complete cycle could speed things up (because each package has to be precompiled only once)? But maybe this is not so easy to implement (just typing nightly sounds easier at least).

I do not want to push, just trying to find a solution :)

[maleadt]

A way to signal skipping could be to add a file test/.PkgEval_no_precompile in one's repository.

That feels like a rather ugly way to accomplish this.

Do you mean that multiple new nightly versions come out during one PkgEval cycle on all tested packages?

No, we test against a single version, so every version of every package only has to be precompiled once. I only meant to say that the precompilation cache doesn't persist across nightly evaluations.

---

Although this seems like a potentially useful feature, or at least the more general mechanism of packages providing information (such as test flags, or whether to skip the package altogether), I'm not sure it's so valuable to be able to disable precompilation. It will result in collected run times to be incorrect, and we use that data to spot regressions, so I don't think I want to make it possible to do so.

Also, how much of a problem is this with your package? Does it time out right now?

[schillic]

It will result in collected run times to be incorrect, and we use that data to spot regressions, so I don't think I want to make it possible to do so.

Makes sense. Thanks for explaining.

Also, how much of a problem is this with your package? Does it time out right now?

Yes, typically it does (and I see this happening only recently). At the time of writing, the latest run actually finished.

Precompilation completed after 2560.34s
[...]
Testing completed after 44.37s

PkgEval succeeded after 2651.18s

It is not a problem per se, technically only a red PkgEval badge. But of course it would be nice to learn when something breaks on the nightly build.

We still use Requires-based optional dependencies. I think I read that weakdeps are not precompiled. So that could be the way to resolve things for us. Do you happen to know?

[maleadt]

I think I read that weakdeps are not precompiled. So that could be the way to resolve things for us.

I'm not sure. Maybe @KristofferC has some thoughts (on this, or on the general question of skipping precompilation for test deps PkgEval doesn't need).