Making Sconstruct a real Python script (aka playing with closing and opening issues)

[marcinn]

Here is a copy of closed issue, which still exists in Scons: #4661

The issue comes from old, ugly and hacky codebase. Needs some deep refactoring, but with preserving a high level API / SConstruct syntax.

Is it doable in one ticket and PR? I think so.
Will this break compatibility? Yes.But it is worth it (SCons2)

In fact there is nothing to talk about. It should be just done.

---

Nothing from scons is imported in SConstruct file. This makes SConstruct an invalid python script. In fact this is not a python script. Linters does not work. Also nobody knows which object / class comes from.

Also there is sys.path mangling and globals magic, which makes this package incompatible with all python tools (not mentioning good practices in python programming).

The point is to do it right. Refactoring can be difficult, but it's worth it if Scons is going to be a future-proof tool. The whole thing is about making SConstruct a real python script, as stated in the docs. This will allow using all python tools, not only rare IDEs (#3210, which is not a solution - think about linters, other IDEs).

Assumption: Adding explicite imports should remove globals magic, and importlib should solve importing other SConstruct files as modules (see import_from_path https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly). Both changes will make Scons user- and stack friendly.

There is no other way to make SConstruct a python script.

---

Enabling better interaction with IDE's however has been a goal for a long time.

This is not about IDE plugins. You are preventing from using all tools, incl. linters.

There's nothing preventing you from using explicit imports in your SConscripts.

Have you tried this? I tried. It does not work because of false negatives. This is not a solution. Even if I add imports, Scons is still changing sys.paths and not using importlib capabilities. This will never work, because there are ugly, old hacks under the hood, which causes a lot of troubles.

That said there's a large number of existing builds using SCons without them, so requiring them would break ALL existing builds.

I am aware of that. They must just die (read: be upgraded) in few years. That's why I said about the future. There is no other way. And we are talking just about imports, not changing the syntax.

[mwichmann]

The issue got closed before I had a change to suggest what you've done: move to a discussion. Despite your suggestion there's nothing to talk about, I think there's quite a lot.

Without going into the merits of a big change to SCons: build systems generally have some kind of language for describing the details of a build, filling in the details it can't deduce itself, and then you call the tool, and it does the work. SCons happens to use Python - mostly; there's also a language for describing token replacement, which is very heavily used in key places, so it's not just all Python. Not really aware of any build systems, beyond those that store the instructions in a "shell script" with minimal smarts, that think the description of the build is something that's considered a standalone program you could just run directly. Obviously, I know about only a small fraction of all the things the world has invented to solve build problems, but... what's the actual need here?

[marcinn]

what's the actual need here?

Make Scons readable, intuitive and compliant with tools and docs/design assumptions .

There is, however, an important difference between an SConstruct file and a Makefile: the SConstruct file is actually a Python script.
Scons User Guide, 2.5 The SConstruct File, 2.5.1 SConstruct Files Are Python Scripts

But

so it's not just all Python.

I understand, but it is a different statement than in the user guide. I see that already:

But the shebang states something else:

there's also a language for describing token replacement

I haven't seen it yet. I believe there could be a solution that doesn't involve hacking Python. However depending on what is currently being replaced, the compatibility may be broken.

---

Despite your suggestion there's nothing to talk about, I think there's quite a lot.

At the top level:

• Do not break compatibility (or as less as possible)*
• Remove hacks

The rest is tech details. Yes, maybe a lot, but it will not change the goal.
https://peps.python.org/pep-0020/#the-zen-of-python

* - camel case is against PEP8, but the compatibility of the high level interface is more important

---

Choosing Python gives flexibility, which is why Scons is often preferred over other tools. Let's make Scons fully Python-comptaible.

[marcinn]

There's nothing preventing you from using explicit imports in your SConscript

Another example why this just can't work:

from SCons.Script import (
    Variables,
    SConscript,
    Environment,
    ARGUMENTS,
    PathVariable,
    BoolVariable,
    Glob,
)

gives

$ pylint SConstruct 
[...]
SConstruct:4:0: E0611: No name 'Glob' in module 'SCons.Script' (no-name-in-module)
[...]

but

In [1]: from SCons.Script import Glob

In [2]: type(Glob)
Out[2]: SCons.Script.SConscript.DefaultEnvironmentCall

because of # Adding global functions to the SConscript name space. magic

You can expose Glob object, which will do what it should, using lazy initialization.

[mwichmann]

Certainly it's imperfect, in part due to the "tricks", no question. The design goal back then was to try to make things simple for the developer, which has to be taken in the context of the times. Consider to start:

• Python wasn't anything near as popular a language at the start of the 21st century, so there was an effort to let you do at least basic builds with SCons without being a Python programmeer - examples in the tutorial (User Guide) might be enough, and don't need to learn about imports and other Pythonic things.
• IDEs weren't really a major mainstream thing - the predecessors that ended up as Visual Studio were not very old, and Eclipse was just happening, so SCons wasn't built to work in that world, partcularly.

These are ancient decisions. Can SCons be "future-proofed" by evolving some of these choices? Maybe... but probably that's a new build tool project instead. This isn't intended to be negative, just realistic. If you have a proposal that can evolve SCons without breaking existing users (who largely work on the basis of "we got it to work for us, don't really want to change things") then of course it will be considered.