Problem: No obvious way for users to generate a racket environment with arbitary packages #158
Comments
clacke
added a commit
to clacke/fractalide
that referenced
this issue
Aug 14, 2018
Solution: Add 'time -p' to the fractalide-tests* commands. Result: pkgs$ nix-build -A fractalide-tests 2>/dev/null | xargs nix-store --read-log | tail -n 3 real 99.86 user 216.53 sys 155.04 $ nix-build -A fractalide-tests-flat 2>/dev/null | xargs nix-store --read-log | tail -n 3 real 16.67 user 48.75 sys 10.58 Each time a tested file requires a dependency, racket has to chase through 200 directories for it. This is made worse by the fact that we start racket once for each file we find. Without modifying 'raco test' itself, we could perhaps make our own wrapper around the test runner to keep everything in one racket process. Flattening fractalide is probably not an option. It takes so much longer to build, and the derivations for the dependencies cannot be reused. It would negate any speed advantage for the tests. The real improvement would probably be to solve fractalide/racket2nix#158 : Build separate derivations for all packages, then build a racket environment that links them all in one folder.
clacke
added a commit
to clacke/fractalide
that referenced
this issue
Aug 14, 2018
Solution: Add 'time -p' to the fractalide-tests* commands. Result: pkgs$ nix-build -A fractalide-tests 2>/dev/null | xargs nix-store --read-log | tail -n 3 real 99.86 user 216.53 sys 155.04 $ nix-build -A fractalide-tests-flat 2>/dev/null | xargs nix-store --read-log | tail -n 3 real 16.67 user 48.75 sys 10.58 Each time a tested file requires a dependency, racket has to chase through 200 directories for it. This is made worse by the fact that we start racket once for each file we find. Without modifying 'raco test' itself, we could perhaps make our own wrapper around the test runner to keep everything in one racket process. Flattening fractalide is probably not an option. It takes so much longer to build, and the derivations for the dependencies cannot be reused. It would negate any speed advantage for the tests. The real improvement would probably be to solve fractalide/racket2nix#158 : Build separate derivations for all packages, then build a racket environment that links them all in one folder.
clacke
added a commit
to clacke/fractalide
that referenced
this issue
Aug 15, 2018
Solution: Add 'time -p' to the fractalide-tests* commands. Result: pkgs$ nix-build -A fractalide-tests 2>/dev/null | xargs nix-store --read-log | tail -n 3 real 99.86 user 216.53 sys 155.04 $ nix-build -A fractalide-tests-flat 2>/dev/null | xargs nix-store --read-log | tail -n 3 real 16.67 user 48.75 sys 10.58 Each time a tested file requires a dependency, racket has to chase through 200 directories for it. This is made worse by the fact that we start racket once for each file we find. Without modifying 'raco test' itself, we could perhaps make our own wrapper around the test runner to keep everything in one racket process. Flattening fractalide is probably not an option. It takes so much longer to build, and the derivations for the dependencies cannot be reused. It would negate any speed advantage for the tests. The real improvement would probably be to solve fractalide/racket2nix#158 : Build separate derivations for all packages, then build a racket environment that links them all in one folder.
clacke
added a commit
to clacke/fractalide
that referenced
this issue
Aug 15, 2018
Solution: Add 'time -p' to the fractalide-tests* commands. Result: pkgs$ nix-build -A fractalide-tests 2>/dev/null | xargs nix-store --read-log | tail -n 3 real 99.86 user 216.53 sys 155.04 $ nix-build -A fractalide-tests-flat 2>/dev/null | xargs nix-store --read-log | tail -n 3 real 16.67 user 48.75 sys 10.58 Each time a tested file requires a dependency, racket has to chase through 200 directories for it. This is made worse by the fact that we start racket once for each file we find. Without modifying 'raco test' itself, we could perhaps make our own wrapper around the test runner to keep everything in one racket process. Flattening fractalide is probably not an option. It takes so much longer to build, and the derivations for the dependencies cannot be reused. It would negate any speed advantage for the tests. The real improvement would probably be to solve fractalide/racket2nix#158 : Build separate derivations for all packages, then build a racket environment that links them all in one folder.
clacke
added a commit
to clacke/fractalide
that referenced
this issue
Aug 15, 2018
Solution: Add 'time -p' to the fractalide-tests* commands. Result: pkgs$ nix-build -A fractalide-tests 2>/dev/null | xargs nix-store --read-log | tail -n 3 real 99.86 user 216.53 sys 155.04 $ nix-build -A fractalide-tests-flat 2>/dev/null | xargs nix-store --read-log | tail -n 3 real 16.67 user 48.75 sys 10.58 Each time a tested file requires a dependency, racket has to chase through 200 directories for it. This is made worse by the fact that we start racket once for each file we find. Without modifying 'raco test' itself, we could perhaps make our own wrapper around the test runner to keep everything in one racket process. Flattening fractalide is probably not an option. It takes so much longer to build, and the derivations for the dependencies cannot be reused. It would negate any speed advantage for the tests. The real improvement would probably be to solve fractalide/racket2nix#158 : Build separate derivations for all packages, then build a racket environment that links them all in one folder.
clacke
added a commit
to clacke/fractalide
that referenced
this issue
Aug 15, 2018
Solution: Add 'time -p' to the fractalide-tests* commands. Result: pkgs$ nix-build -A fractalide-tests 2>/dev/null | xargs nix-store --read-log | tail -n 3 real 99.86 user 216.53 sys 155.04 $ nix-build -A fractalide-tests-flat 2>/dev/null | xargs nix-store --read-log | tail -n 3 real 16.67 user 48.75 sys 10.58 Each time a tested file requires a dependency, racket has to chase through 200 directories for it. This is made worse by the fact that we start racket once for each file we find. Without modifying 'raco test' itself, we could perhaps make our own wrapper around the test runner to keep everything in one racket process. Flattening fractalide is probably not an option. It takes so much longer to build, and the derivations for the dependencies cannot be reused. It would negate any speed advantage for the tests. The real improvement would probably be to solve fractalide/racket2nix#158 : Build separate derivations for all packages, then build a racket environment that links them all in one folder.
clacke
added a commit
to clacke/fractalide
that referenced
this issue
Aug 15, 2018
Building fractalide-tests is 2-3x slower than non-nix raco test with just full racket and raco-linked fractalide. Is it because if the huge links.rktd? If so, flattening should help. Solution: Add a flattened version of fractalide-tests to find out. Add 'time -p' to the fractalide-tests* commands. Result: pkgs$ nix-build -A fractalide-tests 2>/dev/null | xargs nix-store --read-log | tail -n 3 real 99.86 user 216.53 sys 155.04 $ nix-build -A fractalide-tests-flat 2>/dev/null | xargs nix-store --read-log | tail -n 3 real 16.67 user 48.75 sys 10.58 Each time a tested file requires a dependency, racket has to chase through 200 directories for it. This is made worse by the fact that we start racket once for each file we find. Possible solutions: - Without modifying 'raco test' itself, we could perhaps make our own wrapper around the test runner to keep everything in one racket process. - The real improvement would probably be to solve fractalide/racket2nix#158 : Build separate derivations for all packages, then build a racket environment that links them all in one folder. Non-solution: - Flattening fractalide is probably not an option. It takes so much longer to build, and the derivations for the dependencies cannot be reused. It would negate any speed advantage for the tests.
clacke
added a commit
to clacke/fractalide
that referenced
this issue
Aug 15, 2018
Building fractalide-tests is 2-3x slower than non-nix raco test with just full racket and raco-linked fractalide. Is it because if the huge links.rktd? If so, flattening should help. Solution: Add a flattened version of fractalide-tests to find out. Add 'time -p' to the fractalide-tests* commands. Result: pkgs$ nix-build -A fractalide-tests 2>/dev/null | xargs nix-store --read-log | tail -n 3 real 99.86 user 216.53 sys 155.04 $ nix-build -A fractalide-tests-flat 2>/dev/null | xargs nix-store --read-log | tail -n 3 real 16.67 user 48.75 sys 10.58 Each time a tested file requires a dependency, racket has to chase through 200 directories for it. This is made worse by the fact that we start racket once for each file we find. Possible solutions: - Without modifying 'raco test' itself, we could perhaps make our own wrapper around the test runner to keep everything in one racket process. - The real improvement would probably be to solve fractalide/racket2nix#158 : Build separate derivations for all packages, then build a racket environment that links them all in one folder. Non-solution: - Flattening fractalide is probably not an option. It takes so much longer to build, and the derivations for the dependencies cannot be reused. It would negate any speed advantage for the tests.
clacke
added a commit
to clacke/fractalide
that referenced
this issue
Aug 15, 2018
Building fractalide-tests is 2-3x slower than non-nix raco test with just full racket and raco-linked fractalide. Is it because if the huge links.rktd? If so, flattening should help. Solution: Add a flattened version of fractalide-tests to find out. Add 'time -p' to the fractalide-tests* commands. Result: pkgs$ nix-build -A fractalide-tests 2>/dev/null | xargs nix-store --read-log | tail -n 3 real 99.86 user 216.53 sys 155.04 $ nix-build -A fractalide-tests-flat 2>/dev/null | xargs nix-store --read-log | tail -n 3 real 16.67 user 48.75 sys 10.58 Each time a tested file requires a dependency, racket has to chase through 200 directories for it. This is made worse by the fact that we start racket once for each file we find. Possible solutions: - Without modifying 'raco test' itself, we could perhaps make our own wrapper around the test runner to keep everything in one racket process. - The real improvement would probably be to solve fractalide/racket2nix#158 : Build separate derivations for all packages, then build a racket environment that links them all in one folder. Non-solution: - Flattening fractalide is probably not an option. It takes so much longer to build, and the derivations for the dependencies cannot be reused. It would negate any speed advantage for the tests.
clacke
added a commit
to clacke/fractalide
that referenced
this issue
Aug 15, 2018
Building fractalide-tests is 2-3x slower than non-nix raco test with
just full racket and raco-linked fractalide. Is it because if the huge
links.rktd?
If so, flattening should help.
Solution: Add a flattened version of fractalide-tests to find out.
Add 'time -p' to the fractalide-tests* commands.
Result:
```
pkgs$ nix-build -A fractalide-tests 2>/dev/null | xargs nix-store --read-log | tail -n 3
real 99.86
user 216.53
sys 155.04
```
```
pkgs$ nix-build -A fractalide-tests-flat 2>/dev/null | xargs nix-store --read-log | tail -n 3
real 16.67
user 48.75
sys 10.58
```
Each time a tested file requires a dependency, racket has to chase
through 200 directories for it. This is made worse by the fact that
we start racket once for each file we find.
Here's a run entirely nix-less:
```
modules/rkt/rkt-fbp/agents$ /usr/bin/time -p raco test *.rkt cardano-wallet compiled fvm hyperflow IO math mesg plumbing test
[ . . . ]
real 79.57
user 69.78
sys 9.97
```
That actually took longer (wall-clock time) than the flattened run
inside Nix! Apparently raco doesn't parallelize. Here's with parallel:
```
modules/rkt/rkt-fbp/agents$ nix-shell -p parallel --run "find . '(' -name gui -prune ')' -o '(' -name '*.rkt' -print ')' | /usr/bin/time -p parallel -j 4 bash /nix/store/4b1lh27jv1wvm0g7gpif0sxxihgdmi7n-raco-test.sh"
[ . . . ]
real 38.65
user 131.71
sys 15.29
```
I have no explanation for the user time being higher than flattened fractalide.
Possible solutions:
- Without modifying 'raco test' itself, we could perhaps make our own
wrapper around the test runner to keep everything in one racket
process.
- The real improvement would probably be to solve
fractalide/racket2nix#158 : Build separate derivations for all
packages, then build a racket environment that links them all in
one folder.
Non-solution:
- Sadly, flattening fractalide.
It takes so much longer to build, and the derivations for the
dependencies cannot be reused. It would negate any speed advantage
for the tests.
Building flat fractalide takes over an hour CPU time[0]. On our
Hydra it doesn't even build because it runs out of RAM.
[0] On an Intel Core i3-7100U 2.4 GHz
clacke
added a commit
to clacke/fractalide
that referenced
this issue
Aug 15, 2018
Building fractalide-tests is 2-3x slower than non-nix raco test with
just full racket and raco-linked fractalide. Is it because if the huge
links.rktd?
If so, flattening should help.
Solution: Add a flattened version of fractalide-tests to find out.
Add 'time -p' to the fractalide-tests* commands.
Result:
```
pkgs$ nix-build -A fractalide-tests 2>/dev/null | xargs nix-store --read-log | tail -n 3
real 99.86
user 216.53
sys 155.04
```
```
pkgs$ nix-build -A fractalide-tests-flat 2>/dev/null | xargs nix-store --read-log | tail -n 3
real 16.67
user 48.75
sys 10.58
```
Each time a tested file requires a dependency, racket has to chase
through 200 directories for it. This is made worse by the fact that
we start racket once for each file we find.
Here's a run entirely nix-less:
```
modules/rkt/rkt-fbp/agents$ /usr/bin/time -p raco test *.rkt cardano-wallet compiled fvm hyperflow IO math mesg plumbing test
[ . . . ]
real 79.57
user 69.78
sys 9.97
```
That actually took longer (wall-clock time) than the flattened run
inside Nix! Apparently raco doesn't parallelize. Here's with parallel:
```
modules/rkt/rkt-fbp/agents$ nix-shell -p parallel --run "find . '(' -name gui -prune ')' -o '(' -name '*.rkt' -print ')' | /usr/bin/time -p parallel -j 4 bash /nix/store/4b1lh27jv1wvm0g7gpif0sxxihgdmi7n-raco-test.sh"
[ . . . ]
real 38.65
user 131.71
sys 15.29
```
I have no explanation for the user time being higher than flattened fractalide.
Possible solutions:
- Without modifying 'raco test' itself, we could perhaps make our own
wrapper around the test runner to keep everything in one racket
process.
- The real improvement would probably be to solve
fractalide/racket2nix#158 : Build separate derivations for all
packages, then build a racket environment that links them all in
one folder.
Non-solution:
- Sadly, flattening fractalide.
It takes so much longer to build, and the derivations for the
dependencies cannot be reused. It would negate any speed advantage
for the tests.
Building flat fractalide takes over an hour CPU time[0]. On our
Hydra it doesn't even build because it runs out of RAM.
[0] On an Intel Core i3-7100U 2.4 GHz
|
Some of the reasoning above is outdated (as one might hope after half a year more of working with and on the tools). My current work in building racket package derivations in general is about skipping the big Preliminary results are promising. And the way I'm designing it is that the racket environment for each package is put together much like an environment for an arbitrary set of packages would be put together, so it should be easy to factor out and reuse to solve this |
|
Has there been any further progress on this? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Python on Nix has a nice separation between packages on the one hand, and python environments with packages installed on the other hand. racket2nix builds an
envoutput with every derivation.As a user, I cannot easily just say "hey, give me a racket that has fractalide and also compiler-lib, so I can run raco test on it". With python it's as easy as
nix-build -E 'with import <nixpkgs> {}; python.withPackages (pkgs: [pkgs.foo pkgs.bar]), super neat! Of course, that's cheating a bit because there's a python package catalog in there already. On the other hand, we kind of do too.With racket2nix I have to do
nix-build -E 'with import <racket2nix> {}; (buildRacket { package = "thepackage"; }).env. Not many more characters, but admittedly a bit clunky. And no way to create an environment with two unrelated packages present -- I'd have to synthesize a package that depends on them.If we generate the racket environment explicitly, we could make package derivations slightly thinner and less involved -- maybe we could even keep circular dependencies in their own derivations, maybe we wouldn't run setup on them, just copy them in -- and we could make the racket environment a bit thicker than today's implied env output. E.g. we could symlink all the dependencies into one place and skip the enormous
etc/racket/config.rktd.Most of all this relates to #78. A racket with all the deps explicit and in one place could generate
info-cache.rktdappropriately.This kind of thinking might also make it easier to generate the environment without having to explicitly list all transitive dependencies in
racketBuildInputs.Part of this is solvable without doing much new, like #157 solves part of it, but I think looking at it from a new angle could help clear the interfaces up.
The text was updated successfully, but these errors were encountered: