[WIP] trying to tame the slow tests

[enricosada]

step 1: identify slow tests

set test timeouts

• tests: 2 sec
• integration tests: 20 sec

[matthid]

I more or less can confirm your list locally:

It seems some tests even run into a 15 minute timeout when they fail...
Note the screenshot is from running the .preview integration tests, while on AppVeyor the "regular" integration tests are run only.

[matthid]

I added https://github.com/fsprojects/Paket/compare/test_speed?expand=1 in order to download the test-results from AppVeyor. Now I got this:

Seems like half an hour is spent on 2 test (15 minutes each). I briefly looked at those tests and their log but nothing obvious:

Log output test #3166

\u001b[37;2mC:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp already exists.\u001b[0m
C:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp> dotnet build MyProj.Main -c Release
\u001b[32mdotnet build MyProj.Main -c Release\u001b[0m
{IsError = false;
 Message =
  "Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core";}
{IsError = false;
 Message = "Copyright (C) Microsoft Corporation. All rights reserved.";}
{IsError = false;
 Message = "";}
{IsError = false;
 Message = "  Paket version 5.209.1";}
{IsError = false;
 Message = "  Starting full restore process.";}
{IsError = false;
 Message = "  Downloading Argu 5.2";}
{IsError = false;
 Message = "  Downloading System.Memory 4.5.2";}
{IsError = false;
 Message = "  Download of Argu 5.2 done in 1 second. (2551 kbit/s, 0 MB)";}
{IsError = false;
 Message = "  Downloading FSharp.Core 4.3.2";}
{IsError = false;
 Message =
  "  Download of System.Memory 4.5.2 done in 1 second. (1184 kbit/s, 0 MB)";}
{IsError = false;
 Message =
  "  Download of FSharp.Core 4.3.2 done in 488 milliseconds. (40066 kbit/s, 2 MB)";}
{IsError = false;
 Message = "  Performance:";}
{IsError = false;
 Message = "   - Disk IO: 358 milliseconds";}
{IsError = false;
 Message = "   - Average Download Time: 202 milliseconds";}
{IsError = false;
 Message = "   - Number of downloads: 3";}
{IsError = false;
 Message = "   - Average Request Time: 96 milliseconds";}
{IsError = false;
 Message = "   - Number of Requests: 3";}
{IsError = false;
 Message = "   - Runtime: 2 seconds";}
{IsError = false;
 Message =
  "  Restoring packages for C:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp\MyProj.Main\MyProj.Main.csproj...";}
{IsError = false;
 Message =
  "  Restoring packages for C:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp\MyProj.Common\MyProj.Common.csproj...";}
{IsError = false;
 Message =
  "  Generating MSBuild file C:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp\MyProj.Common\obj\MyProj.Common.csproj.nuget.g.props.";}
{IsError = false;
 Message =
  "  Generating MSBuild file C:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp\MyProj.Common\obj\MyProj.Common.csproj.nuget.g.targets.";}
{IsError = false;
 Message =
  "  Generating MSBuild file C:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp\MyProj.Main\obj\MyProj.Main.csproj.nuget.g.props.";}
{IsError = false;
 Message =
  "  Generating MSBuild file C:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp\MyProj.Main\obj\MyProj.Main.csproj.nuget.g.targets.";}
{IsError = false;
 Message =
  "  Restore completed in 232.32 ms for C:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp\MyProj.Common\MyProj.Common.csproj.";}
{IsError = false;
 Message =
  "  Restore completed in 230.85 ms for C:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp\MyProj.Main\MyProj.Main.csproj.";}
{IsError = false;
 Message =
  "  MyProj.Common -> C:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp\MyProj.Common\bin\Release\net45\MyProj.Common.dll";}
{IsError = false;
 Message =
  "  MyProj.Main -> C:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp\MyProj.Main\bin\Release\net45\MyProj.Main.dll";}
{IsError = false;
 Message =
  "  MyProj.Common -> C:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp\MyProj.Common\bin\Release\netstandard2.0\MyProj.Common.dll";}
{IsError = false;
 Message =
  "  MyProj.Main -> C:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp\MyProj.Main\bin\Release\netstandard2.0\MyProj.Main.dll";}
{IsError = false;
 Message = "";}
{IsError = false;
 Message = "Build succeeded.";}
{IsError = false;
 Message = "    0 Warning(s)";}
{IsError = false;
 Message = "    0 Error(s)";}
{IsError = false;
 Message = "";}
{IsError = false;
 Message = "Time Elapsed 00:00:04.16";}
C:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp> paket C:\projects\paket\bin_netcore\paket.dll pack "C:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp\out" 
\u001b[32mdotnet C:\projects\paket\bin_netcore\paket.dll pack "C:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp\out" \u001b[0m
Performance:
 - Runtime: 1 second
Paket version 5.209.1
No description was provided for package "MyProj.Common". Generating from ID and project output type.
No description was provided for package "MyProj.Main". Generating from ID and project output type.
Packaging: C:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp\MyProj.Main\paket.template
Packaging: C:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp\MyProj.Common\paket.template
Wrote: C:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp\out\MyProj.Main.1.0.0.nupkg
Wrote: C:\projects\paket\integrationtests\scenarios\i003166-pack-multitarget-with-p2p-by-tfm\temp\out\MyProj.Common.1.0.0.nupkg
Performance:
 - Runtime: 1 second
{References = All;
 Dependencies = Value is not created.;
 OfficialName = "MyProj.Common";
 Version = "1.0.0";
 LicenseUrl = "";
 IsDevelopmentDependency = false;
 FrameworkAssemblyReferences = [];}
[(PackageName ("FSharp.Core","fsharp.core"),
  VersionRequirement (Minimum 3.1.2.5,No),
  ExplicitRestriction || (>= net45) (>= netstandard2.0));
 (PackageName ("Suave","suave"), VersionRequirement (Specific 1.1.3,No),
  ExplicitRestriction >= net45);
 (PackageName ("Argu","argu"), VersionRequirement (Specific 5.2.0,No),
  ExplicitRestriction && (< net45) (>= netstandard2.0))]
{References = All;
 Dependencies = Value is not created.;
 OfficialName = "MyProj.Main";
 Version = "1.0.0";
 LicenseUrl = "";
 IsDevelopmentDependency = false;
 FrameworkAssemblyReferences = [];}
[(PackageName ("FSharp.Core","fsharp.core"),
  VersionRequirement (Minimum 3.1.2.5,No),
  ExplicitRestriction || (>= net45) (>= netstandard2.0));
 (PackageName ("Suave","suave"), VersionRequirement (Specific 1.1.3,No),
  ExplicitRestriction >= net45);
 (PackageName ("MyProj.Common","myproj.common"),
  VersionRequirement (Minimum 1.0.0,No),
  ExplicitRestriction || (>= net45) (>= netstandard2.0));
 (PackageName ("Argu","argu"), VersionRequirement (Specific 5.2.0,No),
  ExplicitRestriction && (< net45) (>= netstandard2.0))]

Also even more interestingly this disappeared in #3593 For no real reason, now the test-suite (in the PR) runs in half an hour...