Follow-up python support

[jodersky]

This is a collection of ideas to improve Python support, following some of the things we came across while implementing #3928.

Treat this as a rough wishlist, the idea being that we don't forget any ideas we came across. We can create separate issues with more concrete tasks in the future.

• Use a standalone python build to bootstrap venvs, thus allowing users to specify a pythonVersion in the modules and also to cross build.

• Harmonize pyproject.toml support with mill. Right now, we only create a synthetic pyproject.toml file for packaging purposes. A lot of tools in python land however also use this file for configuration. For example, black can be configured in a section of this file, similar to how scalafmt has its own conf file. Mill should support reading this file and merging it with information it has for packaging purposes.

  A required step for this task is to introduce a TOML parser to mill.

• Editor support. Right now you have to manually tell your editor to use the venv that mill generates. Can we automate this? Related: can we add BSP support for python?

• Have a real-world multi-module Python project ported to Mill, similar to the maven/gradle/sbt projects we have in https://github.com/com-lihaoyi/mill/tree/main/example/thirdparty. That would go a long way in demonstrating and validating that the Mill Python toolchain is both useful and complete enough to handle real-world builds of real-world codebases

• Lock files (SBOM?) for python dependencies.

• runBackground support for PythonModule, so that development with -w becomes more convenient. Implemented in Generalize .runBackground to work with all kinds of subprocesses #4085

[lihaoyi]

One thing I missed in the original ticket is to have a real-world multi-module Python project ported to Mill, similar to the maven/gradle/sbt projects we have in https://github.com/com-lihaoyi/mill/tree/main/example/thirdparty. That would go a long way in demonstrating and validating that the Mill Python toolchain is both useful and complete enough to handle real-world builds of real-world codebases

EDIT: included in the PR description

[himanshumahajan138]

what about runBackground discussion i am working on web examples and ready with working python codes but stuck with runBackground task

@jodersky as you said me before about two options and considered second option as best approach and we need discussion about that so i think it's right time for this...

[jodersky]

Indeed, that's a good addition too. It so happens that Haoyi already has a fix for it in #4085 . I'll add it to the list here anyway.