feat/venv convergence

[blast-hardcheese]

Why

.pythonlibs can be made into a proper virtualenv with a few minor edits, most of which are fairly fault tolerant.

What changed

Add to sitecustomize some logic to determine whether we need to heal the .pythonlibs directory. This, combined with two envvars (

PATH="$REPL_HOME/.pythonlibs/bin:$PATH"
VIRTUAL_ENV="$REPL_HOME/.pythonlibs"

) lets us get rid of all of the other python envvars (PYTHONHOME, PYTHONNOUSERSITE, etc), as well as making the following work:

$ pip install --upgrade pip && pip ...
$ uv pip ...

Test plan

$ pip install --upgrade pip && pip install --force-reinstall flask
$ flask --version

$ uv pip install flask
$ flask --version

Rollout

There are some considerations here.

First, I am explicitly not setting VIRTUAL_ENV in this PR. There are a handful of repls out in the wild that got VIRTUAL_ENV in their .replit during the period of time it was in the public template, which is regrettable, so I'll let Ask know to keep an eye out.

Second, users may have voluntarily set VIRTUAL_ENV to .pythonlibs and established their own venv there. If that's the case, we don't touch it.

Third, we might consider rolling this out to explorers first, though if they encounter a bug and want to revert, their .pythonlibs will not get un-converted, so they'll still need help getting working again in that case.

• This is fully backward and forward compatible

[blast-hardcheese]

Initial env exploration

~/2024-03-04-test-venvify$ echo $PYTHONPATH 
/nix/store/yydgqngkqwamgdk7fdxr8g7s78g2fcgm-sitecustomize/lib/python/site-packages:/nix/store/8w6mm5q1n7i7cs1933im5vkbgvjlglfn-python3-3.10.13/lib/python3.10:/home/runner/2024-03-04-test-venvify/.pythonlibs/lib/python3.10/site-packages:/nix/store/gj5phsvidzwvjbv7bgy5nhp1b62m4mn3-python3.10-pip-23.3.1/lib/python3.10/site-packages
~/2024-03-04-test-venvify$ echo $VIRTUAL_ENV

Installing packages

~/2024-03-04-test-venvify$ pip install requests
Collecting requests
  Downloading requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
... snip ...
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests
Successfully installed certifi charset-normalizer idna requests urllib3
~/2024-03-04-test-venvify$ ls .pythonlibs/
bin/ lib/ 

Running python

~/2024-03-04-test-venvify$ python
Python 3.10.13 (main, Aug 24 2023, 12:59:26) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
~/2024-03-04-test-venvify$ ls -Al .pythonlibs/bin/
total 4
-rwxr-xr-x 1 runner runner 291 Mar  4 20:27 normalizer

Enable the virtualenvifying codepath

~/2024-03-04-test-venvify$ VIRTUAL_ENV=$(pwd)/.pythonlibs python
Python 3.10.13 (main, Aug 24 2023, 12:59:26) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
~/2024-03-04-test-venvify$ ls .pythonlibs/
bin/        include/    lib/        lib64/      pyvenv.cfg  
~/2024-03-04-test-venvify$ ls -al .pythonlibs/*
lrwxrwxrwx 1 runner runner   3 Mar  4 20:28 .pythonlibs/lib64 -> lib
-rw-r--r-- 1 runner runner 125 Mar  4 20:28 .pythonlibs/pyvenv.cfg

.pythonlibs/bin:
total 16
drwxr-xr-x 1 runner runner  66 Mar  4 20:28 .
drwxr-xr-x 1 runner runner  56 Mar  4 20:28 ..
-rwxr-xr-x 1 runner runner 291 Mar  4 20:27 normalizer
lrwxrwxrwx 1 runner runner  74 Mar  4 20:28 python -> /nix/store/8w6mm5q1n7i7cs1933im5vkbgvjlglfn-python3-3.10.13/bin/python3.10
lrwxrwxrwx 1 runner runner  74 Mar  4 20:28 python3 -> /nix/store/8w6mm5q1n7i7cs1933im5vkbgvjlglfn-python3-3.10.13/bin/python3.10
lrwxrwxrwx 1 runner runner  74 Mar  4 20:28 python3.10 -> /nix/store/8w6mm5q1n7i7cs1933im5vkbgvjlglfn-python3-3.10.13/bin/python3.10

.pythonlibs/include:
total 0
drwxr-xr-x 1 runner runner  0 Mar  4 20:28 .
drwxr-xr-x 1 runner runner 56 Mar  4 20:28 ..

.pythonlibs/lib:
total 0
drwxr-xr-x 1 runner runner 20 Mar  4 20:27 .
drwxr-xr-x 1 runner runner 56 Mar  4 20:28 ..
drwxr-xr-x 1 runner runner 26 Mar  4 20:27 python3.10

Upgrading tracked python versions

~/2024-03-04-test-venvify$ ls -al .pythonlibs/bin/
total 12
drwxr-xr-x 1 runner runner  54 Mar  4 20:57 .
drwxr-xr-x 1 runner runner  56 Mar  4 20:28 ..
-rwxr-xr-x 1 runner runner 291 Mar  4 20:27 normalizer
lrwxrwxrwx 1 runner runner  74 Mar  4 20:28 python -> /nix/store/8w6mm5q1n7i7cs1933im5vkbgvjlglfn-python3-3.10.13/bin/python3.10
lrwxrwxrwx 1 runner runner  74 Mar  4 20:28 python3 -> /nix/store/8w6mm5q1n7i7cs1933im5vkbgvjlglfn-python3-3.10.13/bin/python3.10
lrwxrwxrwx 1 runner runner  74 Mar  4 20:28 python3.10 -> /nix/store/8w6mm5q1n7i7cs1933im5vkbgvjlglfn-python3-3.10.13/bin/python3.10

~/2024-03-04-test-venvify$ VIRTUAL_ENV=$(pwd)/.pythonlibs /nix/store/2w3zrgyqgiyr46xma8rv9fp1dm8wh6qs-python3-wrapper/bin/python
Python 3.11.7 (main, Dec  4 2023, 18:10:11) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> # This is a shell in a newer version of Python
>>>

~/2024-03-04-test-venvify$ ls -al .pythonlibs/bin/
total 20
drwxr-xr-x 1 runner runner  86 Mar  4 20:58 .
drwxr-xr-x 1 runner runner  56 Mar  4 20:28 ..
-rwxr-xr-x 1 runner runner 291 Mar  4 20:27 normalizer
lrwxrwxrwx 1 runner runner  73 Mar  4 20:58 python -> /nix/store/ip1pq4prmdrb0jkhv2iqv6rkrwv31gs3-python3-3.11.7/bin/python3.11
lrwxrwxrwx 1 runner runner  73 Mar  4 20:58 python3 -> /nix/store/ip1pq4prmdrb0jkhv2iqv6rkrwv31gs3-python3-3.11.7/bin/python3.11
lrwxrwxrwx 1 runner runner  74 Mar  4 20:28 python3.10 -> /nix/store/8w6mm5q1n7i7cs1933im5vkbgvjlglfn-python3-3.10.13/bin/python3.10
lrwxrwxrwx 1 runner runner  73 Mar  4 20:58 python3.11 -> /nix/store/ip1pq4prmdrb0jkhv2iqv6rkrwv31gs3-python3-3.11.7/bin/python3.11

[blast-hardcheese]

CC @airportyh

1. As Toby has elucidated previously, having a symlink into /nix in .pythonlibs effectively blocks automatically picking up new versions of Python from nixmodules
2. If we were to create /replit/python310/bin/python{,3,3.10} -> /nix/store... we would need a stable "root" of a module, to avoid conflicts (python310, node17, etc)
3. The nix-distributed stdlib is available via the PYTHONPATH, not implicitly resolved relative to the sys.executable as I had previously thought. This removes the only downside to creating a stable directory that I can think of.

[airportyh]

Another consideration is the Nix channel in .replit: a different channel will give you different version of the same major-minor version of Python. Currently the Pythons that come from nixmodules isn't dependent on the Nix channel in .replit, and we've just upgraded Python to unstable. Not sure if we want to change that in the future.

[airportyh]

We also have older Python templates which have a venv instead of .pythonlibs and those have VIRTUAL_ENV set as well.

[blast-hardcheese]

We also have older Python templates which have a venv instead of .pythonlibs and those have VIRTUAL_ENV set as well.

This will not be a problem:

    # Don't patch if VIRTUAL_ENV doesn't match our .pythonlibs
    # If the user is managing their own venv, don't touch it.
    if os.environ.get("VIRTUAL_ENV") != pythonlibs:
        return False

[blast-hardcheese]

Another consideration is the Nix channel in .replit: a different channel will give you different version of the same major-minor version of Python. Currently the Pythons that come from nixmodules isn't dependent on the Nix channel in .replit, and we've just upgraded Python to unstable. Not sure if we want to change that in the future.

This raises another problem. nix-shell -p python38 --command python will set python to /nix/store/...python3.8..., bypassing anything we tried to set up.

This is pushing me back towards upm fsck, and just having clear messaging around if/when we automatically run that command, otherwise we will never reconverge.

---

upm fsck...

1. ... iterates through all matching languages (optionally constrained by upm -l python3 fsck, etc)
2. ... removes all user-controlled $PATH entries
3. ... looks up the known interpreters (python/python3/python3.10, any others?) and ensures they're up-to-date

upm fsck is automatically executed...

1. If it is not explicitly disabled in .replit (? rfc)
2. When the repl is first booted(?)
3. Prior to packager3 operations, to ensure consistency before running uv and others
4. Prior to Run(?)