Multi Module Builds & Scoverage

[hilcode]

[For multi module builds]

I have noticed that tests and their coverage data are not kept (entirely) in sync. If a Scala file disappears (by going back to an older commit, for example), the coverage still refers to that file, and the build breaks. Similarly, you will not necessarily get the correct coverage numbers (scoverage.consoleReportAll) when (re)running tests (especially after test failures, I have noticed).

I "solve" this by running a script that removes all scoverage directories under out. This works but it is obviously not efficient. What Mill target/task should be called before running, say, foo.test? Or which file/directory (under out) should be removed prior to running foo.test?

Again, this is for multi module builds. I get the impression that for single module builds things work correctly (with scoverage.consoleReport).

In a project with foo and bar modules, it seems that if foo.test must run then out/foo/scoverage should be removed. And only if either foo.test or bar.test were run then out/scoverage should be removed as well.

I found no obvious way to find the correct out/foo/scoverage directory when running the testTask task, is there? I would prefer not to have to hardcode it. Ditto for out/scoverage.

It was quite simple to add "__.testCached" to reportTask (called by scoverage.consoleReportAll), very similar to "__.allSources" and "__.scoverage.data" that are already there. But how would I know that at least 1 of them (i.e. foo.testCached or bar.testCached) actually caused tests to be run (indicating that I should remove out/scoverage prior to calling consoleReportAll?

The below build.mill illustrates what I have managed to create so far.

Main questions:

1. How do I determine the correct directory under out to delete, without hardcoding it? Or is there some sort of invalidate task?
2. How do I figure out whether any of the testCached targets actually ran any tests?

package build

import $ivy.`com.lihaoyi::mill-contrib-scoverage:`
import mill.contrib.scoverage.api.ScoverageReportWorkerApi2.ReportType
import mill.testrunner.TestResult
import mill.contrib.scoverage.ScoverageModule
import mill.contrib.scoverage.ScoverageReport
import mill.eval.Evaluator
import mill.resolve.Resolve
import mill.resolve.SelectMode
import mill._, scalalib._

trait Versions extends Module {
    final def scalaVersion: Target[String] = Task[String] { "3.3.4" }
    final def scoverageVersion: Target[String] = Task[String] { "2.2.1" }
}

trait BaseModule extends SbtModule with Versions with ScoverageModule {
    object test extends SbtTests with ScoverageTests {
        override def ivyDeps: Target[Agg[Dep]] = Target[Agg[Dep]] { Agg[Dep](ivy"com.lihaoyi::utest:0.8.4") }
        override def testFramework: Target[String] = Target[String] { "utest.runner.Framework" }

        override def testTask(
                args: Task[Seq[String]],
                globSelectors: Task[Seq[String]]
        ): Task[(String, Seq[TestResult])] = Task[(String, Seq[TestResult])] {
            //
            // Remove out/$module/scoverage
            //
            super.testTask(args, globSelectors)()
        }

    }
}

object scoverage extends ScoverageReport with Versions {

    override def consoleReportAll(
            evaluator: Evaluator,
            sources: String,
            dataTargets: String
    ): Command[PathRef] = Task.Command[PathRef] {
        //
        // Remove out/scoverage, if any tests ran
        //
        super.consoleReportAll(evaluator, sources, dataTargets)()
    }

    override def reportTask(
            evaluator: Evaluator,
            reportType: ReportType,
            sources: String,
            dataTargets: String
    ): Task[PathRef] = {
        // This nicely runs all 'testCached' targets before running the 'reportTask'
        val testTasks: Seq[Task[Unit]] = Resolve.Tasks.resolve(
            build,
            Seq("__.testCached"),
            SelectMode.Separated
        ) match {
            case Left(err) => throw new Exception(err)
            case Right(tasks) => tasks.asInstanceOf[Seq[Task[Unit]]]
        }
        Task[PathRef] {
            Target.sequence[Unit](testTasks)()
            super.reportTask(evaluator, reportType, sources, dataTargets)()
        }
    }

}

object foo
        extends BaseModule

object bar
        extends BaseModule

There are also these files:

bar/src/main/scala/org/example/Bar.scala
bar/src/test/scala/org/example/BarTest.scala
foo/src/main/scala/org/example/Foo.scala
foo/src/test/scala/org/example/FooTest.scala

that I will leave to your imagination.

[lefou]

The Scoverage design is to write some files to a hardcoded location. Depending on your use case, it can be desirable to run different tests with the same code, e.g. unit tests and integration tests (I use that to collect coverage data for Mill plugins from integration tests, example: https://app.codecov.io/gh/lefou/mill-osgi). But all coverage data for a specific class will be written to the same location. Hence, it's not possible or meaningful to directly couple it to a specific set of test runs by default.

Each module collects it's own coverage data. To clear all scoverage data, run:

> mill clean __.scoverage.data

To clear just the scoverage data of foo and bar, run:

> mill clean "{foo,bar}.scoverage.data"

Until #3898 is implemented, there is no easy way to run the cleanup task and the test and coverage task in one run. You could use a evaluator command though, but that's beyond a simple one-liner.