Add part 2 : CSV parser

EXERCISE_fr.adoc

@@ -34,12 +34,102 @@ Afficher le contenu d'un fichier dans la console
 
 * Créer une nouvelle classe `fr.lernejo.file.Cat`
 * Créer une fonction `main` de telle sorte que :
-** Le programme prenne un chemin (relatif ou absolu) comme unique argument
+** Le programme prent un chemin (relatif ou absolu) comme unique argument
 ** En cas d'absence d'argument, le programme sort avec le code 3 et le message "Missing argument" dans la sortie standard
 ** Dans le cas de deux arguments ou plus, le programme sort avec le code 4 et le message "Too many arguments" dans la sortie standard
 ** Dans le cas où le fichier indiqué n'existe pas, le programme sort avec le code 5 et le message "File not found"
 ** Dans le cas où le chemin indiqué est un répertoire, le programme sort avec le code 6 et le message "A file is required"
 ** Dans le cas où le fichier fait plus de 3KiB (soit 3*1024 = 3072 bytes), le programme sort avec le code 7 et le message "File too large"
 ** Dans le cas où le fichier est lisible et fait moins de 3KiB, afficher le contenu du fichier dans la console et sortir
 
-* ✔️ Running `java -jar target/maven-training.jar README.md` should display the content of the `README.md` file
+* ✔️ Running `java -cp target/maven-training.jar fr.lernejo.file.Cat README.md` should display the content of the `README.md` file
+
+== Partie 2
+
+Lire un fichier de données au format CSV, et réaliser un calcul simple.
+
+* Créer une nouvelle classe `fr.lernejo.file.CsvReader`
+* Créer une fonction `main` de telle sorte que :
+** Le programme prent ces paramètres :
+*** Un chemin vers un fichier (CSV)
+*** Une date de début au format `YYYY-MM-dd` (inclue)
+*** Une date de fin (exclue)
+*** Un nom de métrique parmi
+**** `temperature_2m` pour température (2ᵉ colonne)
+**** `pressure_msl` pour la pression (4ᵉ colonne)
+**** `wind_speed_10m` pour la vitesse du vent (6ᵉ colonne)
+**** `direct_normal_irradiance_instant` pour l'irradiance (9ᵉ colonne)
+*** Un sélecteur : `NIGHT` ou `DAY`
+*** Un type d'agrégation : `SUM`, `AVG`, `MIN`, `MAX`
+** Le programme affiche la valeur calculée suivie de son unité et sort
+
+Le fichier CSV en entrée sera toujours structuré de la même manière à savoir :
+
+* 3 lignes de préambule
+* 1 ligne d'entêtes (avec nom des métriques et unités)
+* N lignes de données
+* Les colonnes seront toujours les mêmes, et dans le même ordre.
+
+Exemple de fichier CSV :
+
+[source,csv]
+----
+latitude,longitude,elevation,utc_offset_seconds,timezone,timezone_abbreviation
+52.54833,13.407822,38.0,0,GMT,GMT
+
+time,temperature_2m (°C),rain (mm),pressure_msl (hPa),cloud_cover (%),wind_speed_10m (km/h),soil_temperature_0_to_7cm (°C),is_day (),direct_normal_irradiance_instant (W/m²)
+2010-01-01T00:00,-2.6,0.00,996.9,100,16.0,-0.6,0,0.0
+2010-01-01T01:00,-2.7,0.00,996.4,100,16.3,-0.6,0,0.0
+2010-01-01T02:00,-2.7,0.00,996.2,100,16.3,-0.6,0,0.0
+2010-01-01T03:00,-2.7,0.00,996.1,100,15.4,-0.6,0,0.0
+2010-01-01T04:00,-2.7,0.00,996.0,100,14.8,-0.6,0,0.0
+2010-01-01T05:00,-2.7,0.00,996.1,100,14.8,-0.6,0,0.0
+2010-01-01T06:00,-2.8,0.00,996.1,100,15.0,-0.6,0,0.0
+2010-01-01T07:00,-2.7,0.00,996.3,100,14.3,-0.6,0,0.0
+2010-01-01T08:00,-2.7,0.00,996.7,100,13.8,-0.6,1,0.0
+2010-01-01T09:00,-2.4,0.00,997.2,100,13.7,-0.6,1,8.5
+2010-01-01T10:00,-2.2,0.00,997.2,100,13.2,-0.6,1,20.4
+2010-01-01T11:00,-2.0,0.00,997.3,100,13.8,-0.6,1,50.0
+2010-01-01T12:00,-1.7,0.00,997.6,100,14.0,-0.6,1,44.7
+2010-01-01T13:00,-1.6,0.00,997.6,100,14.3,-0.6,1,27.9
+2010-01-01T14:00,-1.7,0.00,998.1,100,12.8,-0.6,1,33.9
+2010-01-01T15:00,-2.1,0.00,999.1,100,10.5,-0.6,1,0.0
+2010-01-01T16:00,-2.5,0.00,999.6,100,10.6,-0.6,0,0.0
+2010-01-01T17:00,-2.8,0.00,999.9,100,10.8,-0.6,0,0.0
+2010-01-01T18:00,-3.2,0.00,1000.4,100,10.3,-0.6,0,0.0
+2010-01-01T19:00,-2.9,0.00,1001.1,100,10.5,-0.6,0,0.0
+2010-01-01T20:00,-3.0,0.00,1001.8,100,10.0,-0.6,0,0.0
+2010-01-01T21:00,-2.9,0.00,1002.3,100,9.5,-0.6,0,0.0
+2010-01-01T22:00,-2.9,0.00,1002.9,100,11.0,-0.5,0,0.0
+2010-01-01T23:00,-2.9,0.00,1003.4,100,8.7,-0.5,0,0.0
+2010-01-02T00:00,-3.0,0.00,1004.0,100,8.3,-0.5,0,0.0
+2010-01-02T01:00,-3.2,0.00,1004.5,100,9.1,-0.5,0,0.0
+2010-01-02T02:00,-3.4,0.00,1005.2,100,10.1,-0.5,0,0.0
+2010-01-02T03:00,-3.5,0.00,1005.6,93,11.2,-0.5,0,0.0
+2010-01-02T04:00,-3.7,0.00,1006.2,94,11.2,-0.5,0,0.0
+2010-01-02T05:00,-3.8,0.00,1007.2,99,10.9,-0.5,0,0.0
+2010-01-02T06:00,-3.8,0.00,1008.0,97,10.0,-0.5,0,0.0
+2010-01-02T07:00,-3.6,0.00,1008.9,72,10.0,-0.5,0,0.0
+2010-01-02T08:00,-3.6,0.00,1009.8,62,10.5,-0.5,1,0.0
+2010-01-02T09:00,-3.2,0.00,1010.6,64,11.0,-0.5,1,16.9
+2010-01-02T10:00,-2.7,0.00,1011.4,92,11.6,-0.5,1,25.4
+2010-01-02T11:00,-1.9,0.00,1012.0,100,10.9,-0.5,1,82.9
+2010-01-02T12:00,-1.4,0.00,1012.7,100,11.2,-0.5,1,84.7
+2010-01-02T13:00,-1.2,0.00,1013.5,100,14.3,-0.5,1,69.3
+2010-01-02T14:00,-1.4,0.00,1014.1,100,15.8,-0.5,1,73.5
+2010-01-02T15:00,-1.9,0.00,1015.2,93,15.0,-0.5,1,0.0
+2010-01-02T16:00,-2.0,0.00,1016.0,100,15.3,-0.5,0,0.0
+2010-01-02T17:00,-2.4,0.00,1016.6,89,13.9,-0.5,0,0.0
+2010-01-02T18:00,-3.2,0.00,1017.4,88,12.4,-0.5,0,0.0
+2010-01-02T19:00,-4.4,0.00,1018.0,97,11.2,-0.5,0,0.0
+2010-01-02T20:00,-5.1,0.00,1018.6,85,10.8,-0.5,0,0.0
+2010-01-02T21:00,-5.7,0.00,1018.8,92,10.9,-0.5,0,0.0
+2010-01-02T22:00,-5.7,0.00,1019.1,100,9.2,-0.5,0,0.0
+2010-01-02T23:00,-5.6,0.00,1019.4,100,7.3,-0.5,0,0.0
+----
+
+Un tel fichier est disponible peut être construit sur le site https://open-meteo.com.
+Le fichier ci-dessus a été téléchargé à l'URL https://archive-api.open-meteo.com/v1/archive?latitude=52.52&longitude=13.41&start_date=2010-01-01&end_date=2010-01-02&hourly=temperature_2m,rain,pressure_msl,cloud_cover,wind_speed_10m,soil_temperature_0_to_7cm,is_day,direct_normal_irradiance_instant&format=csv
+Customisation possible avec https://open-meteo.com/en/docs/historical-weather-api#start_date=2010-01-01&end_date=2010-01-02&hourly=temperature_2m,rain,pressure_msl,cloud_cover,wind_speed_10m,soil_temperature_0_to_7cm,is_day,direct_normal_irradiance_instant
+
+10 ans de données =~ 5 MiB

src/main/java/com/github/lernejo/korekto/grader/load_file/LaunchingContext.java

@@ -2,16 +2,11 @@
 
 import com.github.lernejo.korekto.toolkit.GradingConfiguration;
 import com.github.lernejo.korekto.toolkit.GradingContext;
-import com.github.lernejo.korekto.toolkit.misc.OS;
-import com.github.lernejo.korekto.toolkit.misc.Processes;
 import com.github.lernejo.korekto.toolkit.partgrader.MavenContext;
 
-import java.io.BufferedInputStream;
 import java.io.IOException;
-import java.io.InputStream;
 import java.io.UncheckedIOException;
 import java.net.URISyntaxException;
-import java.nio.charset.StandardCharsets;
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.nio.file.Paths;
@@ -21,10 +16,14 @@
 import java.util.stream.IntStream;
 import java.util.stream.Stream;
 
+import static com.github.lernejo.korekto.grader.load_file.PathUtils.resourceToPath;
+
 public class LaunchingContext extends GradingContext implements MavenContext {
     private final List<String> dictionary;
+    private final List<WeatherComputationData> dataset = WeatherComputationData.Loader.load();
     private boolean compilationFailed;
     private boolean testFailed;
+    public static final Path DATA_FILE_PATH = resourceToPath("open-meteo-52.55N13.41E38m.csv").toAbsolutePath();
 
     public LaunchingContext(GradingConfiguration configuration) {
         super(configuration);
@@ -68,56 +67,6 @@ public Optional<Path> jarPath() {
         }
     }
 
-    public Processes.ProcessResult launchJava(Path workingDirectory, Path jarPath, String mainClass, String... arguments) {
-        Path binPath = Paths.get(System.getProperty("java.home")).resolve("bin");
-        final Path javaPath;
-        if (OS.WINDOWS.isCurrentOs()) {
-            javaPath = binPath.resolve("java.exe");
-        } else {
-            javaPath = binPath.resolve("java");
-        }
-        List<String> commandPrefix = List.of(
-            javaPath.toString(),
-            "-Duser.country=UK",
-            "-Duser.language=en",
-            "-cp",
-            jarPath.toString(),
-            mainClass
-        );
-        List<String> command = Stream.concat(commandPrefix.stream(), Arrays.stream(arguments)).toList();
-        ProcessBuilder processBuilder = new ProcessBuilder(command);
-        processBuilder.directory(workingDirectory.toFile());
-
-        try {
-            var process = processBuilder.start();
-            int exitCode = process.waitFor();
-            if (exitCode != 0) {
-                var stdout = readStream(process.getInputStream());
-                var stderr = readStream(process.getErrorStream());
-                return Processes.ProcessResult.Companion.error(exitCode, stderr + stdout);
-            }
-            var stdout = readStream(process.getInputStream());
-            return Processes.ProcessResult.Companion.success(stdout);
-        } catch (IOException | InterruptedException e) {
-            return Processes.ProcessResult.Companion.error(e);
-        }
-    }
-
-    public String readStream(InputStream inputStream) {
-        try (BufferedInputStream bis = new BufferedInputStream(inputStream)) {
-            bis.mark(1);
-            var firstByte = bis.read();
-            if (firstByte != -1) {
-                bis.reset();
-                return new Scanner(bis, StandardCharsets.UTF_8).useDelimiter("\\A").next();
-            } else {
-                return "";
-            }
-        } catch (IOException e) {
-            throw new UncheckedIOException(e);
-        }
-    }
-
     public String getRandomWord() {
         int index = getRandomSource().nextInt(dictionary.size());
         return dictionary.get(index);
@@ -140,4 +89,13 @@ public String writeRandomWords(Path path, int nbWords) {
             throw new UncheckedIOException(e);
         }
     }
+
+    public Set<WeatherComputationData> getDataset(int size) {
+        Set<WeatherComputationData> r = new HashSet<>();
+        while (r.size() != size) {
+            int index = getRandomSource().nextInt(dataset.size());
+            r.add(dataset.get(index));
+        }
+        return r;
+    }
 }

src/main/java/com/github/lernejo/korekto/grader/load_file/LoadFileGrader.java

@@ -1,6 +1,7 @@
 package com.github.lernejo.korekto.grader.load_file;
 
 import com.github.lernejo.korekto.grader.load_file.parts.Part1Grader;
+import com.github.lernejo.korekto.grader.load_file.parts.Part2Grader;
 import com.github.lernejo.korekto.toolkit.GradePart;
 import com.github.lernejo.korekto.toolkit.Grader;
 import com.github.lernejo.korekto.toolkit.GradingConfiguration;
@@ -59,12 +60,13 @@ private Collection<? extends PartGrader<LaunchingContext>> graders() {
                 "Compilation & Tests",
                 1.0D),
             new JacocoCoveragePartGrader<>("Code Coverage", 4.0D, 0.85D),
-            new Part1Grader("Part 1 - Cat program", 4.0D)
+            new Part1Grader("Part 1 - Cat program", 4.0D),
+            new Part2Grader("Part 2 - CSV reader", 6.0D)
         );
     }
 
     @Override
     public boolean needsWorkspaceReset() {
-        return false;
+        return true;
     }
 }

src/main/java/com/github/lernejo/korekto/grader/load_file/PathUtils.java

@@ -0,0 +1,16 @@
+package com.github.lernejo.korekto.grader.load_file;
+
+import java.net.URISyntaxException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+
+public class PathUtils {
+
+    public static Path resourceToPath(String resourceName) {
+        try {
+            return Paths.get(WeatherComputationData.class.getClassLoader().getResource(resourceName).toURI());
+        } catch (URISyntaxException e) {
+            throw new RuntimeException(e);
+        }
+    }
+}

src/main/java/com/github/lernejo/korekto/grader/load_file/StringUtils.java

@@ -0,0 +1,26 @@
+package com.github.lernejo.korekto.grader.load_file;
+
+import java.util.regex.Matcher;
+
+public class StringUtils {
+
+    public static String safeEscapeElide(String s) {
+        if (s == null) {
+            return "";
+        } else {
+            String s1 = s.trim().replaceAll("\n", Matcher.quoteReplacement("\\n"));
+            if (s1.length() > 80) {
+                return s1.substring(0, 77) + "...";
+            }
+            return s1;
+        }
+    }
+
+    public static String safeLowerTrim(String s) {
+        if (s == null) {
+            return "";
+        } else {
+            return s.trim().toLowerCase();
+        }
+    }
+}

src/main/java/com/github/lernejo/korekto/grader/load_file/WeatherComputationData.java

@@ -0,0 +1,25 @@
+package com.github.lernejo.korekto.grader.load_file;
+
+import com.github.lernejo.korekto.grader.load_file.vault.Vault;
+
+import java.util.List;
+
+import static com.github.lernejo.korekto.grader.load_file.PathUtils.resourceToPath;
+
+public record WeatherComputationData(String start, String end, String metric, String selector, String agg,
+                                     String result) {
+
+    public String asDescription() {
+        String period = Vault.isClear() ? start + '/' + end : "(period undisclosed)";
+        return period + '/' + metric + '/' + selector + '/' + agg;
+    }
+
+    public static final class Loader {
+        public static final Vault<List<WeatherComputationData>> vault = new Vault<>() {
+        };
+
+        public static List<WeatherComputationData> load() {
+            return vault.load(resourceToPath("vault/results.encrypted"), resourceToPath("vault/results.clear"));
+        }
+    }
+}

src/main/java/com/github/lernejo/korekto/grader/load_file/parts/Part1Grader.java

@@ -1,9 +1,10 @@
 package com.github.lernejo.korekto.grader.load_file.parts;
 
 import com.github.lernejo.korekto.grader.load_file.LaunchingContext;
+import com.github.lernejo.korekto.grader.load_file.process.JavaProcessLauncher;
+import com.github.lernejo.korekto.grader.load_file.process.ProcessResult;
 import com.github.lernejo.korekto.toolkit.GradePart;
 import com.github.lernejo.korekto.toolkit.PartGrader;
-import com.github.lernejo.korekto.toolkit.misc.Processes;
 import kotlin.Triple;
 
 import java.io.IOException;
@@ -13,33 +14,15 @@
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Optional;
-import java.util.regex.Matcher;
+
+import static com.github.lernejo.korekto.grader.load_file.StringUtils.safeEscapeElide;
+import static com.github.lernejo.korekto.grader.load_file.StringUtils.safeLowerTrim;
 
 public record Part1Grader(String name, Double maxGrade) implements PartGrader<LaunchingContext> {
 
     private static final String TEST_FILENAME = "test.txt";
     private static final String TEST_LONG_FILENAME = "test_long.txt";
 
-    private static String safeEscapeElide(String s) {
-        if (s == null) {
-            return "";
-        } else {
-            String s1 = s.trim().replaceAll("\n", Matcher.quoteReplacement("\\n"));
-            if (s1.length() > 80) {
-                return s1.substring(0, 77) + "...";
-            }
-            return s1;
-        }
-    }
-
-    private static String safeLowerTrim(String s) {
-        if (s == null) {
-            return "";
-        } else {
-            return s.trim().toLowerCase();
-        }
-    }
-
     private List<Feature> features(String content, String directoryName) {
         return List.of(
             new Feature("no argument", 3, "Missing argument"),
@@ -66,31 +49,36 @@ public GradePart grade(LaunchingContext context) {
 
         List<Feature> features = features(setup.component2(), setup.component3());
 
-        List<String> featureErrors = verifyFeatures(context, features, jarPath.get(), setup.component1());
+        List<String> featureErrors = verifyFeatures(features, jarPath.get(), setup.component1());
 
         return result(featureErrors, maxGrade - featureErrors.size() * (maxGrade / features.size()));
     }
 
-    private List<String> verifyFeatures(LaunchingContext context, List<Feature> features, Path jarPath, Path workingDirectory) {
-        List<String> featureErrors = new ArrayList<>();
+    private List<String> verifyFeatures(List<Feature> features, Path jarPath, Path workingDirectory) {
+        List<String> errors = new ArrayList<>();
         for (Feature feature : features) {
-            Processes.ProcessResult result = context.launchJava(workingDirectory, jarPath, "fr.lernejo.file.Cat", feature.arguments);
+            ProcessResult result = JavaProcessLauncher
+                .withClasspath(jarPath)
+                .withWorkingDirectory(workingDirectory)
+                .withMainClass("fr.lernejo.file.Cat")
+                .withParameters(feature.arguments)
+                .start();
             StringBuilder sb = new StringBuilder();
-            if (result.getExitCode() != feature.exitCode) {
-                sb.append("expecting exit code to be `").append(feature.exitCode).append("` but was `").append(result.getExitCode()).append('`');
+            if (result.exitCode() != feature.exitCode) {
+                sb.append("expecting exit code to be `").append(feature.exitCode).append("` but was `").append(result.exitCode()).append('`');
             }
 
-            if (!feature.content.trim().toLowerCase().equals(safeLowerTrim(result.getOutput()))) {
+            if (!feature.content.trim().toLowerCase().equals(safeLowerTrim(result.stdout()))) {
                 if (!sb.isEmpty()) {
                     sb.append(" and ");
                 }
                 sb.append("expecting output to be `").append(safeEscapeElide(feature.content)).append("` but was `").append(safeEscapeElide(result.getOutput())).append('`');
             }
             if (!sb.isEmpty()) {
-                featureErrors.add("In the case of " + feature.name + " " + sb);
+                errors.add("In the case of " + feature.name + " " + sb);
             }
         }
-        return featureErrors;
+        return errors;
     }
 
     private Triple<Path, String, String> setupWorkingDirectory(LaunchingContext context) {