Improve Maven benchmarking and add profiling support, take 2

README.md

@@ -287,9 +287,26 @@ Here is an example:
 
 Values are optional and default to the values provided on the command-line or defined in the build.
 
-### Profiling incremental builds
+### Benchmark options
 
-A scenario can define changes that should be applied to the source before each build. You can use this to benchmark or profile an incremental build. The following mutations are available:
+- `iterations`: Number of builds to actually measure
+- `warm-ups`: Number of warmups to perform before measurement
+- `jvm-args`: Sets or overrides the jvm arguments set by `org.gradle.jvmargs` in gradle.properties.
+
+### Profiling change handling
+
+How a build tool handles changes to the source code can have a significant impact on the performance of the build.
+Gradle Profiler can simulate different kinds of changes to the source code to measure the impact of these changes on the build performance.
+These changes are applied by mutators at different points in the build benchmark process.
+Some mutators execute at a specific point, others can be configured to execute at a specific point, specified by the `schedule` parameter:
+
+- `SCENARIO`: before the scenario is executed,
+- `CLEANUP`: before cleaning preceeding each build invocation,
+- `BUILD`: before the build invocation (after cleanup).
+
+#### Source code mutators
+
+These mutations are applied before each build, and they introduce different kinds of change to the source code.
 
 - `apply-abi-change-to`: Add a public method to a Java or Kotlin source class. Each iteration adds a new method and removes the method added by the previous iteration.
 - `apply-android-layout-change-to`: Add a hidden view with id to an Android layout file. Supports traditional layouts as well as Databinding layouts with a ViewGroup as the root element.
@@ -303,23 +320,29 @@ A scenario can define changes that should be applied to the source before each b
 - `apply-non-abi-change-to`: Change the body of a public method in a Java or Kotlin source class.
 - `apply-project-dependency-change-to`: Add project dependencies to a Groovy or a Kotlin DSL build script. Each iteration adds a new combination of projects as dependencies and removes the projects added by the previous iteration.
 - `apply-property-resource-change-to`: Add an entry to a properties file. Each iteration adds a new entry and removes the entry added by the previous iteration.
-- `clear-android-studio-cache-before`: Invalidates the Android Studio caches before the scenario is executed (`SCENARIO`) or before the build is executed (`BUILD`). Due to Android Studio client specifics before cleanup (`CLEANUP`) is not supported. Note: cleaning the Android Studio caches is run only when Android Studio sync (`android-studio-sync`) is used.
-- `clear-build-cache-before`: Deletes the contents of the build cache before the scenario is executed (`SCENARIO`), before cleanup (`CLEANUP`) or before the build is executed (`BUILD`).
-- `clear-configuration-cache-state-before`: Deletes the contents of the `.gradle/configuration-cache-state` directory before the scenario is executed (`SCENARIO`), before cleanup (`CLEANUP`) or before the build is executed (`BUILD`).
-- `clear-gradle-user-home-before`: Deletes the contents of the Gradle user home directory before the scenario is executed (`SCENARIO`), before cleanup (`CLEANUP`) or before the build is executed (`BUILD`).
+
+#### Cache cleanup
+
+When simulating scenarios like ephemeral builds, it is important to make sure caches are not present.
+These mutators can be scheduled to execute at different points in the build benchmark process, specified by the `schedule` parameter.
+
+- `clear-android-studio-cache-before`: Invalidates the Android Studio caches. Due to Android Studio client specifics scheduling to run before cleanup (`CLEANUP`) is not supported. Note: cleaning the Android Studio caches is run only when Android Studio sync (`android-studio-sync`) is used.
+- `clear-build-cache-before`: Deletes the contents of the build cache at the given schedule.
+- `clear-configuration-cache-state-before`: Deletes the contents of the `.gradle/configuration-cache-state` directory.
+- `clear-gradle-user-home-before`: Deletes the contents of the Gradle user home directory.
   The mutator retains the `wrapper` cache in the Gradle user home, since the downloaded wrapper in that location is used to run Gradle.
-  Requires to use the `none` daemon option to use with `CLEANUP` or `BUILD`.
-- `clear-jars-cache-before`: Deletes the contents of the instrumented jars cache before the scenario is executed (`SCENARIO`), before cleanup (`CLEANUP`) or before the build is executed (`BUILD`).
-- `clear-project-cache-before`: Deletes the contents of the `.gradle` and `buildSrc/.gradle` project cache directories before the scenario is executed (`SCENARIO`), before cleanup (`CLEANUP`) or before the build is executed (`BUILD`).
-- `clear-transform-cache-before`: Deletes the contents of the transform cache before the scenario is executed (`SCENARIO`), before cleanup (`CLEANUP`) or before the build is executed (`BUILD`).
-- `copy-file`: Copies a file or a directory from one location to another. Has to specify a `source` and a `target` path; relative paths are resolved against the project directory. Can also specify a schedule (defaults `SCENARIO`). Can take an array of operations.
-- `delete-file`: Deletes a file or a directory. Has to specify a `target` path; when relative it is resolved against the project directory. Can also specify a schedule (defaults `SCENARIO`). Can take an array of operations.
+  Requires to use the `none` daemon option to use with `CLEANUP` or `BUILD` schedules.
+- `clear-jars-cache-before`: Deletes the contents of the instrumented jars cache.
+- `clear-project-cache-before`: Deletes the contents of the `.gradle` and `buildSrc/.gradle` project cache directories.
+- `clear-transform-cache-before`: Deletes the contents of the transform cache.
+- `show-build-cache-size`: Shows the number of files and their size in the build cache before scenario execution, and after each cleanup and build round.
+
+#### File operations
+
+- `copy-file`: Copies a file or a directory from one location to another. Has to specify a `source` and a `target` path; relative paths are resolved against the project directory. Can take an array of operations. Defaults to `SCNEARIO` schedule.
+- `delete-file`: Deletes a file or a directory. Has to specify a `target` path; when relative it is resolved against the project directory. Can take an array of operations. Defaults to `SCNEARIO` schedule.
 - `git-checkout`: Checks out a specific commit for the build step, and a different one for the cleanup step.
-- `git-revert`: Reverts a given set of commits before the build and resets it afterward.
-- `iterations`: Number of builds to actually measure
-- `jvm-args`: Sets or overrides the jvm arguments set by `org.gradle.jvmargs` in gradle.properties.
-- `show-build-cache-size`: Shows the number of files and their size in the build cache before scenario execution, and after each cleanup and build round..
-- `warm-ups`: Number of warmups to perform before measurement
+- `git-revert`: Reverts a given set of commits before the build and resets it afterward. 
 
 They can be added to a scenario file like this:
 

src/main/java/org/gradle/profiler/BuildToolCommandLineInvoker.java

@@ -1,31 +1,56 @@
 package org.gradle.profiler;
 
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
 import org.gradle.profiler.result.BuildActionResult;
 import org.gradle.profiler.result.BuildInvocationResult;
 
 import java.io.File;
 import java.time.Duration;
 import java.util.List;
+import java.util.Map;
 import java.util.function.Consumer;
 import java.util.function.Supplier;
 
 import static org.gradle.profiler.Phase.MEASURE;
 import static org.gradle.profiler.Phase.WARM_UP;
 
 public abstract class BuildToolCommandLineInvoker<T extends BuildToolCommandLineScenarioDefinition, R extends BuildInvocationResult> extends ScenarioInvoker<T, R> {
-    protected void doRun(T scenario, InvocationSettings settings, Consumer<BuildInvocationResult> resultConsumer, List<String> commandLine) {
+    protected void doRun(T scenario, InvocationSettings settings, Consumer<BuildInvocationResult> resultConsumer, List<String> commandLine, Map<String, String> envVars) {
+        doRun(scenario, settings, resultConsumer, commandLine, envVars, ImmutableList.of(), ImmutableMap.of());
+    }
+
+    protected void doRun(
+        T scenario,
+        InvocationSettings settings,
+        Consumer<BuildInvocationResult> resultConsumer,
+        List<String> commandLine,
+        Map<String, String> envVars,
+        List<String> profileCommandLine,
+        Map<String, String> profileEnvVars
+    ) {
         ScenarioContext scenarioContext = ScenarioContext.from(settings, scenario);
 
         BuildMutator mutator = CompositeBuildMutator.from(scenario.getBuildMutators());
         mutator.beforeScenario(scenarioContext);
         try {
             for (int iteration = 1; iteration <= scenario.getWarmUpCount(); iteration++) {
                 BuildContext buildContext = scenarioContext.withBuild(WARM_UP, iteration);
-                runMeasured(buildContext, mutator, measureCommandLineExecution(commandLine, settings.getProjectDir(), settings.getBuildLog()), resultConsumer);
+                BuildStepAction<R> action = measureCommandLineExecution(commandLine, envVars, settings.getProjectDir(), settings.getBuildLog());
+                runMeasured(buildContext, mutator, action, resultConsumer);
             }
             for (int iteration = 1; iteration <= scenario.getBuildCount(); iteration++) {
                 BuildContext buildContext = scenarioContext.withBuild(MEASURE, iteration);
-                runMeasured(buildContext, mutator, measureCommandLineExecution(commandLine, settings.getProjectDir(), settings.getBuildLog()), resultConsumer);
+                List<String> commandLineCombined = ImmutableList.<String>builder()
+                    .addAll(commandLine)
+                    .addAll(profileCommandLine)
+                    .build();
+                Map<String, String> envVarsCombined = ImmutableMap.<String, String>builder()
+                    .putAll(envVars)
+                    .putAll(profileEnvVars)
+                    .build();
+                BuildStepAction<R> action = measureCommandLineExecution(commandLineCombined, envVarsCombined, settings.getProjectDir(), settings.getBuildLog());
+                runMeasured(buildContext, mutator, action, resultConsumer);
             }
         } finally {
             mutator.afterScenario(scenarioContext);
@@ -35,7 +60,7 @@ protected void doRun(T scenario, InvocationSettings settings, Consumer<BuildInvo
     /**
      * Returns a {@link Supplier} that returns the result of the given command.
      */
-    private BuildStepAction<R> measureCommandLineExecution(List<String> commandLine, File workingDir, File buildLog) {
+    private BuildStepAction<R> measureCommandLineExecution(List<String> commandLine, Map<String, String> envVars, File workingDir, File buildLog) {
         return new BuildStepAction<R>() {
             @Override
             public boolean isDoesSomething() {
@@ -44,11 +69,16 @@ public boolean isDoesSomething() {
 
             @Override
             public R run(BuildContext buildContext, BuildStep buildStep) {
+                Logging.detailed().println("  Command: " + commandLine);
+                Logging.detailed().println("  Environment: " + envVars);
+                CommandExec commandExec = new CommandExec()
+                    .inDir(workingDir)
+                    .environmentVariables(envVars);
                 Timer timer = new Timer();
                 if (buildLog == null) {
-                    new CommandExec().inDir(workingDir).run(commandLine);
+                    commandExec.run(commandLine);
                 } else {
-                    new CommandExec().inDir(workingDir).runAndCollectOutput(buildLog, commandLine);
+                    commandExec.runAndCollectOutput(buildLog, commandLine);
                 }
                 Duration executionTime = timer.elapsed();
                 return (R) new BuildInvocationResult(buildContext, new BuildActionResult(executionTime));

src/main/java/org/gradle/profiler/BuildToolCommandLineScenarioDefinition.java

@@ -1,6 +1,7 @@
 package org.gradle.profiler;
 
 import javax.annotation.Nullable;
+import javax.annotation.OverridingMethodsMustInvokeSuper;
 import java.io.File;
 import java.io.PrintStream;
 import java.util.List;
@@ -33,9 +34,14 @@ protected void printDetail(PrintStream out) {
         out.println("  Targets: " + getTargets());
     }
 
-    public String getExecutablePath() {
+    public String getExecutablePath(File projectDir) {
         String toolHomePath = getToolHome() == null ? System.getenv(getToolHomeEnvName()) : getToolHome().getAbsolutePath();
-        return toolHomePath == null ? getExecutableName() : toolHomePath + "/bin/" + getExecutableName();
+        return toolHomePath == null ? getExecutablePathWithoutToolHome(projectDir) : toolHomePath + "/bin/" + getExecutableName();
+    }
+
+    @OverridingMethodsMustInvokeSuper
+    protected String getExecutablePathWithoutToolHome(File projectDir) {
+        return getExecutableName();
     }
 
     @Override
@@ -53,7 +59,7 @@ public File getToolHome() {
 
     @Override
     public boolean createsMultipleProcesses() {
-        return false;
+        return true;
     }
 
     @Override

src/main/java/org/gradle/profiler/CompositeProfiler.java

@@ -13,6 +13,11 @@ class CompositeProfiler extends Profiler {
         this.delegates = delegates;
     }
 
+    @Override
+    public boolean requiresGradle() {
+        return delegates.stream().anyMatch(Profiler::requiresGradle);
+    }
+
     @Override
     public String toString() {
         return delegates.stream().map(Object::toString).collect(Collectors.joining(", "));

src/main/java/org/gradle/profiler/InstrumentingProfiler.java

@@ -18,6 +18,12 @@
  * <p>The profiler may support starting recording multiple times for a given JVM. The implementation should indicate this by overriding {@link #canRestartRecording(ScenarioSettings)}.</p>
  */
 public abstract class InstrumentingProfiler extends Profiler {
+
+    @Override
+    public boolean requiresGradle() {
+        return false;
+    }
+
     /**
      * Calculates the JVM args for all builds, including warm-ups.
      *
@@ -67,7 +73,7 @@ public JvmArgsCalculator newInstrumentedBuildsJvmArgsCalculator(ScenarioSettings
      */
     @Override
     public ProfilerController newController(String pid, ScenarioSettings settings) {
-        SnapshotCapturingProfilerController controller = doNewController(settings);
+        SnapshotCapturingProfilerController controller = newSnapshottingController(settings);
         if (settings.getScenario().getInvoker().isDoesNotUseDaemon()) {
             return new SessionOnlyController(pid, controller);
         }
@@ -109,7 +115,7 @@ protected boolean canRestartRecording(ScenarioSettings settings) {
      */
     protected abstract JvmArgsCalculator jvmArgsWithInstrumentation(ScenarioSettings settings, boolean startRecordingOnProcessStart, boolean captureSnapshotOnProcessExit);
 
-    protected abstract SnapshotCapturingProfilerController doNewController(ScenarioSettings settings);
+    public abstract SnapshotCapturingProfilerController newSnapshottingController(ScenarioSettings settings);
 
     public interface SnapshotCapturingProfilerController {
         void startRecording(String pid) throws IOException, InterruptedException;

src/main/java/org/gradle/profiler/Main.java

@@ -88,6 +88,10 @@ public void run(String[] args) throws Exception {
             for (ScenarioDefinition scenario : scenarios) {
                 scenarioCount++;
                 Logging.startOperation("Running scenario " + scenario.getDisplayName() + " (scenario " + scenarioCount + "/" + totalScenarios + ")");
+                if (settings.isProfile() && scenario.getWarmUpCount() == 0) {
+                    throw new IllegalStateException("Using the --profile option requires at least one warm-up");
+                }
+
                 if (scenario instanceof BazelScenarioDefinition) {
                     invoke(bazelScenarioInvoker, (BazelScenarioDefinition) scenario, settings, benchmarkResults, failures);
                 } else if (scenario instanceof BuckScenarioDefinition) {

src/main/java/org/gradle/profiler/Profiler.java

@@ -18,15 +18,25 @@
 import java.io.File;
 import java.util.function.Consumer;
 
-public class Profiler {
+public abstract class Profiler {
 
     public static final Profiler NONE = new Profiler() {
+        @Override
+        public boolean requiresGradle() {
+            return false;
+        }
+
         @Override
         public String toString() {
             return "none";
         }
     };
 
+    /**
+     * Whether this profiler supports only Gradle builds.
+     */
+    public abstract boolean requiresGradle();
+
     public void validate(ScenarioSettings settings, Consumer<String> reporter) {
     }
 

src/main/java/org/gradle/profiler/ScenarioDefinition.java

@@ -1,8 +1,12 @@
 package org.gradle.profiler;
 
+import org.apache.commons.io.FileUtils;
+
 import javax.annotation.Nullable;
 import java.io.File;
+import java.io.IOException;
 import java.io.PrintStream;
+import java.io.UncheckedIOException;
 import java.util.List;
 import java.util.function.Consumer;
 
@@ -28,6 +32,11 @@ public ScenarioDefinition(
         this.warmUpCount = warmUpCount;
         this.buildCount = buildCount;
         this.outputDir = outputDir;
+        try {
+            FileUtils.forceMkdir(outputDir);
+        } catch (IOException e) {
+            throw new UncheckedIOException(e);
+        }
     }
 
     public void validate() {
@@ -98,6 +107,7 @@ public void printTo(PrintStream out) {
     }
 
     public void visitProblems(InvocationSettings settings, Consumer<String> reporter) {
+        settings.getProfiler().validate(new ScenarioSettings(settings, this), reporter);
     }
 
     protected void printDetail(PrintStream out) {
@@ -108,4 +118,8 @@ protected void printDetail(PrintStream out) {
     public abstract boolean doesCleanup();
 
     public abstract BuildConfiguration getBuildConfiguration();
+
+    public static String safeFileName(String name) {
+        return name.replace("/", "-");
+    }
 }

src/main/java/org/gradle/profiler/ScenarioLoader.java

@@ -238,7 +238,7 @@ static List<ScenarioDefinition> loadScenarios(File scenarioFile, InvocationSetti
             String title = scenario.hasPath(TITLE) ? scenario.getString(TITLE) : null;
 
             int buildCount = getBuildCount(settings, scenario);
-            File scenarioBaseDir = selectedScenarios.size() == 1 ? settings.getOutputDir() : new File(settings.getOutputDir(), GradleScenarioDefinition.safeFileName(scenarioName));
+            File scenarioBaseDir = selectedScenarios.size() == 1 ? settings.getOutputDir() : new File(settings.getOutputDir(), ScenarioDefinition.safeFileName(scenarioName));
 
             if (scenario.hasPath(BAZEL) && settings.isBazel()) {
                 Config executionInstructions = getConfig(scenarioFile, settings, scenarioName, scenario, BAZEL, BAZEL_KEYS);
@@ -331,8 +331,8 @@ private static List<BuildMutator> getMutators(Config scenario, String scenarioNa
     }
 
     private static Config getConfig(File scenarioFile, InvocationSettings settings, String scenarioName, Config scenario, String toolName, List<String> toolKeys) {
-        if (settings.isProfile()) {
-            throw new IllegalArgumentException("Can only profile scenario '" + scenarioName + "' when building using Gradle.");
+        if (settings.getProfiler().requiresGradle()) {
+            throw new IllegalArgumentException("Profiler " + settings.getProfiler() + " is not compatible with " + toolName + " scenarios.");
         }
         Config executionInstructions = scenario.getConfig(toolName);
         for (String key : scenario.getObject(toolName).keySet()) {

src/main/java/org/gradle/profiler/asyncprofiler/AsyncProfiler.java

@@ -24,7 +24,7 @@ protected JvmArgsCalculator jvmArgsWithInstrumentation(ScenarioSettings settings
     }
 
     @Override
-    protected SnapshotCapturingProfilerController doNewController(ScenarioSettings settings) {
+    public SnapshotCapturingProfilerController newSnapshottingController(ScenarioSettings settings) {
         return new AsyncProfilerController(config, settings);
     }